Ask Your Question
1

HadoopFS as destination throws error while running pipeline

asked 2019-03-27 16:30:24 -0500

renukakallepalli gravatar image

Hi,

I created a pipeline to push data from oracle to hive

JDBC Query consumer -> hive metadata -> hadoop FS & Hive metastore

I am getting the below error while running the pipeline but it passes the validation

Pipeline Status: RUNNING_ERROR: com.streamsets.pipeline.api.StageException: HADOOPFS_13 - Error while writing to HDFS: com.streamsets.pipeline.api.StageException: HADOOPFS_14 - Cannot write record: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:88) at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1835) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1483) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:4310) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:901) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getFileInfo(AuthorizationProviderProxyClientProtocol.java:528) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:839) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211)

Did anyone face this issue , can anyone throw some light on this issue?

Thanks,
Renuka

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2019-04-04 04:50:36 -0500

rishi gravatar image

updated 2019-05-24 01:46:17 -0500

metadaddy gravatar image

Do you have HDFS HA enabled? Seems like request reached to Standby Namenode instead Active one.

get dfs.nameservices and try to connect to hdfs using namespace as follows, it may help you i.e hdfs://<ClusterName>-ns/<hdfs_path>

Also make sure to store hdfs-site.xml, core-site.xml in SDC. -Store the files or a symlink to the files in the Data Collector resources directory. -In the Hadoop FS destination, specify the location of the files.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-03-27 16:30:24 -0500

Seen: 286 times

Last updated: May 24