Ask Your Question
0

Hadoop FS HA How to configure ?

asked 2017-12-26 03:09:39 -0500

xiaoxu gravatar image

Hadoop FS configure Hadoop FS URI How to ?image description

edit retag flag offensive close merge delete

2 Answers

Sort by » oldest newest most voted
0

answered 2018-02-23 22:14:27 -0500

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

The problem is solved,Thanks very much

edit flag offensive delete link more
0

answered 2018-01-30 00:52:53 -0500

updated 2018-01-30 00:53:59 -0500

Configure the destination like this:

Hadoop FS tab:

Hadoop FS URI: you can leave this empty if SDC should use the default file system connection from the configuration files loaded below. Otherwise, for Hadoop FS specify a URI of the form hdfs://hostname/ and for MapR FS, maprfs:///mapr/my.cluster.com/.

HDFS User: an appropriate HDFS username

Kerberos Authentication: you should enable this if your Hadoop environment is secured

Hadoop FS Configuration Directory: you may need to change this to suit your environment. This directory must contain the core-site.xml and hdfs-site.xml configuration files.

Output Files tab:

Data Format: Avro

Directory in Header: Enabled

Max Records in File: 1

Use Roll Attribute: Enabled

Roll Attribute Name: roll

Note - the destination will continue writing to a file until the first of these five conditions is satisfied:

The number of records specified in ‘Max Records in File’ has been written (zero means there is no maximum)

The specified ‘Max File Size’ has been reached (again, zero means there is no maximum)

No records have been written for the specified ‘Idle Timeout’

A record with the specified roll header attribute is processed

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-12-26 03:09:39 -0500

Seen: 88 times

Last updated: Feb 23