Kafka to MongoDB destination [closed]
I am trying to load the data from Kafka to MongoDB destination. The data in kafka is in below format : { "xid": 1948, "nextlsn": "0/286FB420", "timestamp": "2018-11-29 16:41:21.217215-05", "change": [ { "kind": "insert", "schema": "public", "table": "wal_test3", "columnnames": ["col1", "description"], "columntypes": ["integer", "character varying"], "columnvalues": [5, "wal_test5"] } ] } I am able to read data from Kafka Consumer if data format is Text (but throwing an error if I set JSON format). I am unable to read this in JSON format. I am using JSON generator initially to convert to json and able to see in data preview as: Record1-Output Record1 : STRING {"xid":1948,"nextlsn":"0/286FB420","timestamp":"2018-11-29 16:41:21.217215-05","change":[{"kind":"insert","schema":"public","table":"wal_test3","columnnames":["col1","description"],"columntypes":["integer","character varying"],"columnvalues":[5,"wal_test5"]}]}
I need to parse the data and load in to have MongoDB destination with database name and Collection name (did not do any further configurations). I have Kafka source, JSON Generator and MongoDB destination in my pipeline.
When tried to start the pipeline, getting the below error: Pipeline Status: RUNNING_ERROR: org.bson.BsonInvalidOperationException: readStartDocument can only be called when CurrentBSONType is DOCUMENT, not when CurrentBSONType is STRING.
Please suggest me how to parse the incoming data and load in to MongODB destination.
What exception does it throw when you set Data Format to JSON on Kafka Consumer origin?
Below exception if I set Data Format as JSON for Kafka Consumer: KAFKA_37 - Cannot parse record from message 'postgres_wal::0::1770': com.fasterxml.jackson.core.JsonParseException: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
Is this your example message { "timestamp": "2018-11-29 16:41:21.217215-05", "change": [ { "kind": "insert", "table": "wal_test3", } ] }? If so, there's an extra comma in the end, but if I remove it, it works for me in SDC 3.5. What exactly does you message look like and what's your SDC version?