Ask Your Question

Hadoop FS HA How to configure ?

asked 2017-12-26 03:09:39 -0500

xiaoxu gravatar image

Hadoop FS configure Hadoop FS URI How to ?image description

edit retag flag offensive close merge delete

1 Answer

Sort by » oldest newest most voted

answered 2018-01-30 00:52:53 -0500

updated 2018-01-30 00:53:59 -0500

Configure the destination like this:

Hadoop FS tab:

Hadoop FS URI: you can leave this empty if SDC should use the default file system connection from the configuration files loaded below. Otherwise, for Hadoop FS specify a URI of the form hdfs://hostname/ and for MapR FS, maprfs:///mapr/

HDFS User: an appropriate HDFS username

Kerberos Authentication: you should enable this if your Hadoop environment is secured

Hadoop FS Configuration Directory: you may need to change this to suit your environment. This directory must contain the core-site.xml and hdfs-site.xml configuration files.

Output Files tab:

Data Format: Avro

Directory in Header: Enabled

Max Records in File: 1

Use Roll Attribute: Enabled

Roll Attribute Name: roll

Note - the destination will continue writing to a file until the first of these five conditions is satisfied:

The number of records specified in ‘Max Records in File’ has been written (zero means there is no maximum)

The specified ‘Max File Size’ has been reached (again, zero means there is no maximum)

No records have been written for the specified ‘Idle Timeout’

A record with the specified roll header attribute is processed

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-12-26 03:09:39 -0500

Seen: 675 times

Last updated: Jan 30 '18