HADOOPFS_44 - Could not verify the base directory

asked 2018-06-25 06:59:35 -0500

Edward gravatar image

updated 2018-06-25 10:01:06 -0500

metadaddy gravatar image

I've provided the following to Hadoop FS Destination.

Hadoop FS URI (under Hadoop FS tab): hdfs://technocrat:8020/ as fs.defaultFS is hdfs://technocrat:8020 in core-site.xml where technocrat is my host name in the cluster.

Directory Template (under Output Files tab): /user/streamsets/streamer where streamer is the directory that needs to be created under which I want to save my output files.

Error is: HADOOPFS_44 - Could not verify the base directory: 'java.io.IOException: Failed on local exception: java.nio.channels.ClosedByInterruptException; Host Details : local host is: "technocrat/xxx.xxx.xx.xxx"; destination host is: "technocrat":8020; '

I don't understand where does it go wrong! Any help would be greatly appreciated.

Regards, Edward

edit retag flag offensive close merge delete

Comments

Can you please provide the entire Java exception stack trace? Have you also tried using defaults to isolate the issue?

iamontheinet gravatar imageiamontheinet ( 2018-06-26 11:52:46 -0500 )edit

The thing is it is giving validation error HADOOPFS_44 - Could not verify the base directory: 'java.io.IOException: java.lang.InterruptedException' when I tried to preview it,else it is working fine when I run the pipeline or validate it.

Edward gravatar imageEdward ( 2018-06-27 04:01:43 -0500 )edit

To clarify, you only see this error when you preview but *not* when you validate or start/run the pipeline? If so, do you see the desired output when you start/run the pipeline? And, have you tried using default values to isolate the issue?

iamontheinet gravatar imageiamontheinet ( 2018-06-27 10:27:35 -0500 )edit

yes @iamontheinet ! I could see the output when I run the pipeline but I want to check how the record is being processed where I stuck.

Edward gravatar imageEdward ( 2018-06-28 02:20:23 -0500 )edit
1

To see how the records are being processed at any stage you can take Snapshots as described here --https://streamsets.com/documentation/datacollector/3.3.0/help/#datacollector/UserGuide/Pipeline_Monitoring/PipelineMonitoring_title.html

iamontheinet gravatar imageiamontheinet ( 2018-06-28 11:37:43 -0500 )edit