How do I get sdc to write to HDFS?

asked 2019-09-15 20:11:44 -0500

chompers gravatar image

updated 2019-09-16 18:21:24 -0500

I am running a data collector (3.10.0), connected to Control Hub (3.8?). While trying to run a pipeline, I get the following error:

"Pipeline status: RUNNING_ERROR: HADOOPFS_13 - Error while writing to HDFS: java.io.IOException: org,apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)"

I have followed the online documentation to enable Kerberos via CDH 6.2 ("Enabling Kerberos with Cloudera Manager" - https://streamsets.com/documentation/...).

I can knit as the sdc user using the sdc keytab (from root) and can list HDFS, touchz a file etc. Running the pipeline results in the error above. Kerberos authentication is checked in the pipeline config. I am also using a custom Configuration Files Directory value, as it had issues finding the hive-site.xml (so I have links in that directoty).

edit retag flag offensive close merge delete

Comments

You say "from root". Are you able to do the same command line validation running as the sdc user? Or are you running SDC itself as root?

jeff gravatar imagejeff ( 2019-09-17 21:39:44 -0500 )edit

The sdc user is a system user and does not have a log in shell.

chompers gravatar imagechompers ( 2019-09-17 21:40:57 -0500 )edit