Ask Your Question

Revision history [back]

HDFS Json.BZ2 to Kinesis JSON ,getting null records

Standaalone HDFS json.bz2 to kinesis json ,getting null records

HDFS Json.BZ2 to Kinesis JSON ,getting null records

Standaalone Stand alone HDFS json.bz2 to kinesis json ,getting null records

HDFS Json.BZ2 to Kinesis JSON ,getting null records

Stand alone HDFS json.bz2 to kinesis json ,getting null records

[2019-04-15 18:09:36.769905] [0x000145ac][0x00007f63e2ba9700] [info] [shard_map.cc:144] Successfully updated shard map for stream "tds.dmr.busmap_event" found 1 shards LogInputStreamReader *admin 0 kpl-daemon-0003 2019-04-15 18:09:41,913 HDFS_To_HW3/HDFSToHW34021e68c-a9d6-497b-bba1-26766f646659 INFO sending no-more-data event. records 10 errors 0 files 1

HDFS Json.BZ2 to Kinesis JSON ,getting null records

Stand alone HDFS json.bz2 to kinesis json ,getting null records

[2019-04-15 18:09:36.769905] [0x000145ac][0x00007f63e2ba9700] [info] [shard_map.cc:144] Successfully updated shard map for stream "tds.dmr.busmap_event" found 1 shards LogInputStreamReader *admin 0 kpl-daemon-0003 2019-04-15 18:09:41,913 HDFS_To_HW3/HDFSToHW34021e68c-a9d6-497b-bba1-26766f646659 INFO sending no-more-data event. records 10 errors 0 files 1

HDFS Json.BZ2 to Kinesis JSON ,getting null no records

Stand alone HDFS json.bz2 to kinesis json ,getting null records

[2019-04-15 18:09:36.769905] [0x000145ac][0x00007f63e2ba9700] [info] [shard_map.cc:144] Successfully updated shard map for stream "tds.dmr.busmap_event" found 1 shards LogInputStreamReader *admin 0 kpl-daemon-0003 2019-04-15 18:09:41,913 HDFS_To_HW3/HDFSToHW34021e68c-a9d6-497b-bba1-26766f646659 INFO sending no-more-data event. records 10 errors 0 files 1

-bash-4.2$ aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name tds.dmr.busmap_event { "ShardIterator": "AAAAAAAAAAG4+Q9gSTSqCh5wHDcH6l3976R/hhJS6dJY3vbG84FtZKkbwbb7Cv9J/+MCplljmCYyAwxF3cEDZS/qjlt6Yse8sJR9s3jncGCHN2bRRkkQmwp9An5n3NCbFXMtnIv1Jr/RRhx9qZTJynBuAjqPt3IJ3RLllx5lB79T7dq/WVBHZ4jWUC9nTrqrJFzVnJ2NCNUO9MFFBIEsgVghk7R6H3DdC47lG5l2lPARCvEJvi48Kw==" } -bash-4.2$ aws kinesis get-records --shard-iterator AAAAAAAAAAG4+Q9gSTSqCh5wHDcH6l3976R/hhJS6dJY3vbG84FtZKkbwbb7Cv9J/+MCplljmCYyAwxF3cEDZS/qjlt6Yse8sJR9s3jncGCHN2bRRkkQmwp9An5n3NCbFXMtnIv1Jr/RRhx9qZTJynBuAjqPt3IJ3RLllx5lB79T7dq/WVBHZ4jWUC9nTrqrJFzVnJ2NCNUO9MFFBIEsgVghk7R6H3DdC47lG5l2lPARCvEJvi48Kw== { "Records": [], "NextShardIterator": "AAAAAAAAAAHiU+PTxX5yJ2Fas9j8PM+Kz/rlXaxZaF7k4BLtRGY2aY0CSYwJC8rJz22CoRT3rDwWPbpqJez1YDFUiyI3tAfDCn5XbAuKW9GV91v1e5ucYJTwqCH2i61voJksi7uDCrCRtHo6HT6hkKWcdmfL2EScWgyWtyWVuJdnnL2dIn5pKUYngXZYSGesRB3tYurIB19fm1+nOZNS9kFZGicCZAEE5MthAOMnf1PQ5kx1K+ea3Q==", "MillisBehindLatest": 85847000 }

HDFS Json.BZ2 to Kinesis JSON ,getting no records

Stand alone Trying to push HDFS json.bz2 files to kinesis Kinesis with json ,getting null recordsas destination data type. StreamSets logs don't show any errors but when I try get the records from the stream, there are none.

[2019-04-15 18:09:36.769905] [0x000145ac][0x00007f63e2ba9700] [info] [shard_map.cc:144] Successfully updated shard map for stream "tds.dmr.busmap_event" found 1 shards LogInputStreamReader *admin 0 kpl-daemon-0003 2019-04-15 18:09:41,913 HDFS_To_HW3/HDFSToHW34021e68c-a9d6-497b-bba1-26766f646659 INFO sending no-more-data event. records 10 errors 0 files 1

-bash-4.2$ aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name tds.dmr.busmap_event { "ShardIterator": "AAAAAAAAAAG4+Q9gSTSqCh5wHDcH6l3976R/hhJS6dJY3vbG84FtZKkbwbb7Cv9J/+MCplljmCYyAwxF3cEDZS/qjlt6Yse8sJR9s3jncGCHN2bRRkkQmwp9An5n3NCbFXMtnIv1Jr/RRhx9qZTJynBuAjqPt3IJ3RLllx5lB79T7dq/WVBHZ4jWUC9nTrqrJFzVnJ2NCNUO9MFFBIEsgVghk7R6H3DdC47lG5l2lPARCvEJvi48Kw==" } -bash-4.2$ aws kinesis get-records --shard-iterator AAAAAAAAAAG4+Q9gSTSqCh5wHDcH6l3976R/hhJS6dJY3vbG84FtZKkbwbb7Cv9J/+MCplljmCYyAwxF3cEDZS/qjlt6Yse8sJR9s3jncGCHN2bRRkkQmwp9An5n3NCbFXMtnIv1Jr/RRhx9qZTJynBuAjqPt3IJ3RLllx5lB79T7dq/WVBHZ4jWUC9nTrqrJFzVnJ2NCNUO9MFFBIEsgVghk7R6H3DdC47lG5l2lPARCvEJvi48Kw== { "Records": [], "NextShardIterator": "AAAAAAAAAAHiU+PTxX5yJ2Fas9j8PM+Kz/rlXaxZaF7k4BLtRGY2aY0CSYwJC8rJz22CoRT3rDwWPbpqJez1YDFUiyI3tAfDCn5XbAuKW9GV91v1e5ucYJTwqCH2i61voJksi7uDCrCRtHo6HT6hkKWcdmfL2EScWgyWtyWVuJdnnL2dIn5pKUYngXZYSGesRB3tYurIB19fm1+nOZNS9kFZGicCZAEE5MthAOMnf1PQ5kx1K+ea3Q==", "MillisBehindLatest": 85847000 }

HDFS Json.BZ2 to Kinesis JSON ,getting no records

Trying to push HDFS json.bz2 files to Kinesis with json as destination data type. StreamSets logs don't show any errors but when I try get the records from the stream, there are none.

[2019-04-15 18:09:36.769905] [0x000145ac][0x00007f63e2ba9700] [info] [shard_map.cc:144] Successfully updated shard map for stream "tds.dmr.busmap_event" found 1 shards LogInputStreamReader *admin 0 kpl-daemon-0003 2019-04-15 18:09:41,913 HDFS_To_HW3/HDFSToHW34021e68c-a9d6-497b-bba1-26766f646659 INFO sending no-more-data event. records 10 errors 0 files 1

-bash-4.2$ aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name tds.dmr.busmap_event { "ShardIterator": "AAAAAAAAAAG4+Q9gSTSqCh5wHDcH6l3976R/hhJS6dJY3vbG84FtZKkbwbb7Cv9J/+MCplljmCYyAwxF3cEDZS/qjlt6Yse8sJR9s3jncGCHN2bRRkkQmwp9An5n3NCbFXMtnIv1Jr/RRhx9qZTJynBuAjqPt3IJ3RLllx5lB79T7dq/WVBHZ4jWUC9nTrqrJFzVnJ2NCNUO9MFFBIEsgVghk7R6H3DdC47lG5l2lPARCvEJvi48Kw==" } -bash-4.2$ aws kinesis get-records --shard-iterator AAAAAAAAAAG4+Q9gSTSqCh5wHDcH6l3976R/hhJS6dJY3vbG84FtZKkbwbb7Cv9J/+MCplljmCYyAwxF3cEDZS/qjlt6Yse8sJR9s3jncGCHN2bRRkkQmwp9An5n3NCbFXMtnIv1Jr/RRhx9qZTJynBuAjqPt3IJ3RLllx5lB79T7dq/WVBHZ4jWUC9nTrqrJFzVnJ2NCNUO9MFFBIEsgVghk7R6H3DdC47lG5l2lPARCvEJvi48Kw== { "Records": [], "NextShardIterator": "AAAAAAAAAAHiU+PTxX5yJ2Fas9j8PM+Kz/rlXaxZaF7k4BLtRGY2aY0CSYwJC8rJz22CoRT3rDwWPbpqJez1YDFUiyI3tAfDCn5XbAuKW9GV91v1e5ucYJTwqCH2i61voJksi7uDCrCRtHo6HT6hkKWcdmfL2EScWgyWtyWVuJdnnL2dIn5pKUYngXZYSGesRB3tYurIB19fm1+nOZNS9kFZGicCZAEE5MthAOMnf1PQ5kx1K+ea3Q==", "MillisBehindLatest": 85847000 }