Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

How do I get sdc to write to HDFS?

I am running a data collector (3.10.0), connected to Control Hub (3.8?). While trying to run a pipeline, I get the following error:

"Pipeline status: RUNNING_ERROR: HADOOPFS_13 - Error while writing to HDFS: java.io.IOException: org,apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to fina any Kerberos tgt)"

I have followed the online documentation to enable Kerberos via CDH 6.2 ("Enabling Kerberos with Cloudera Manager" - https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Configuration/DCConfig.html#concept_hnm_n4l_xs).

I can knit as the sdc user using the sdc keytab (from root) and can list HDFS, touchz a file etc. Running the pipeline results in the error above. Kerberos authentication is checked in the pipeline config. I am also using a custom Configuration Files Directory value, as it had issues finding the hive-site.xml (so I have links in that directoty).

How do I get sdc to write to HDFS?

I am running a data collector (3.10.0), connected to Control Hub (3.8?). While trying to run a pipeline, I get the following error:

"Pipeline status: RUNNING_ERROR: HADOOPFS_13 - Error while writing to HDFS: java.io.IOException: org,apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to fina find any Kerberos tgt)"

I have followed the online documentation to enable Kerberos via CDH 6.2 ("Enabling Kerberos with Cloudera Manager" - https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Configuration/DCConfig.html#concept_hnm_n4l_xs).

I can knit as the sdc user using the sdc keytab (from root) and can list HDFS, touchz a file etc. Running the pipeline results in the error above. Kerberos authentication is checked in the pipeline config. I am also using a custom Configuration Files Directory value, as it had issues finding the hive-site.xml (so I have links in that directoty).