HDFS Json.BZ2 to Kinesis JSON ,getting no records

asked 2019-04-15 13:26:10 -0500

Ashokvaka gravatar image

updated 2019-04-15 13:53:25 -0500

Trying to push HDFS json.bz2 files to Kinesis with json as destination data type. StreamSets logs don't show any errors but when I try get the records from the stream, there are none.

[2019-04-15 18:09:36.769905] [0x000145ac][0x00007f63e2ba9700] [info] [shard_map.cc:144] Successfully updated shard map for stream "tds.dmr.busmap_event" found 1 shards LogInputStreamReader *admin 0 kpl-daemon-0003 2019-04-15 18:09:41,913 HDFS_To_HW3/HDFSToHW34021e68c-a9d6-497b-bba1-26766f646659 INFO sending no-more-data event. records 10 errors 0 files 1

-bash-4.2$ aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name tds.dmr.busmap_event { "ShardIterator": "AAAAAAAAAAG4+Q9gSTSqCh5wHDcH6l3976R/hhJS6dJY3vbG84FtZKkbwbb7Cv9J/+MCplljmCYyAwxF3cEDZS/qjlt6Yse8sJR9s3jncGCHN2bRRkkQmwp9An5n3NCbFXMtnIv1Jr/RRhx9qZTJynBuAjqPt3IJ3RLllx5lB79T7dq/WVBHZ4jWUC9nTrqrJFzVnJ2NCNUO9MFFBIEsgVghk7R6H3DdC47lG5l2lPARCvEJvi48Kw==" } -bash-4.2$ aws kinesis get-records --shard-iterator AAAAAAAAAAG4+Q9gSTSqCh5wHDcH6l3976R/hhJS6dJY3vbG84FtZKkbwbb7Cv9J/+MCplljmCYyAwxF3cEDZS/qjlt6Yse8sJR9s3jncGCHN2bRRkkQmwp9An5n3NCbFXMtnIv1Jr/RRhx9qZTJynBuAjqPt3IJ3RLllx5lB79T7dq/WVBHZ4jWUC9nTrqrJFzVnJ2NCNUO9MFFBIEsgVghk7R6H3DdC47lG5l2lPARCvEJvi48Kw== { "Records": [], "NextShardIterator": "AAAAAAAAAAHiU+PTxX5yJ2Fas9j8PM+Kz/rlXaxZaF7k4BLtRGY2aY0CSYwJC8rJz22CoRT3rDwWPbpqJez1YDFUiyI3tAfDCn5XbAuKW9GV91v1e5ucYJTwqCH2i61voJksi7uDCrCRtHo6HT6hkKWcdmfL2EScWgyWtyWVuJdnnL2dIn5pKUYngXZYSGesRB3tYurIB19fm1+nOZNS9kFZGicCZAEE5MthAOMnf1PQ5kx1K+ea3Q==", "MillisBehindLatest": 85847000 }

edit retag flag offensive close merge delete

Comments

Can you give some more detail? That's not much to go on!

metadaddy gravatar imagemetadaddy ( 2019-04-15 13:29:32 -0500 )edit

i have added details

Ashokvaka gravatar imageAshokvaka ( 2019-04-15 14:01:43 -0500 )edit