Ask Your Question

HDFS ORC data format to Kinesis; getting no records

asked 2019-04-10 15:34:04 -0600

Ashokvaka gravatar image

updated 2019-04-10 22:15:55 -0600

metadaddy gravatar image

Trying to push HDFS ORC files to Kinesis with SDC Record as destination data type. StreamSets logs don't show any errors but when I try get the records from the stream, there are none.

log :
[2019-04-10 18:11:42.798569] [0x000163dc][0x00007ff99dd58700] [info] [] Updating shard map for stream "tds.dmr.busmap_event"
[2019-04-10 18:11:42.831412] [0x000163dc][0x00007ff98b7fe700] [info] [] Successfully updated shard map for stream "tds.dmr.busmap_event" found 1 shards

sending no-more-data event. records 1 errors 0 files 1

I'm using standalone Hadoop FS with whole file data file type; Kinesis destination with SDC Record data file type.

edit retag flag offensive close merge delete


Which origin are you using? The standalone or regular Hadoop FS?

metadaddy gravatar imagemetadaddy ( 2019-04-10 15:42:23 -0600 )edit

I'm using standalone Hadoop FS and wholefile data file type and at kinesis SDC data file type .

Ashokvaka gravatar imageAshokvaka ( 2019-04-10 18:15:22 -0600 )edit

1 Answer

Sort by » oldest newest most voted

answered 2019-04-10 22:13:50 -0600

metadaddy gravatar image

If you are using whole file data format in the origin, you also have to use whole file in the destination - see the docs on Whole File Data Format.

Unfortunately, ORC is not supported by either of the Hadoop FS origins, so I think you’ll have to find another way round this. Please vote/watch/comment on SDC-6344.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2019-04-10 15:34:04 -0600

Seen: 152 times

Last updated: Apr 10 '19