How do I convert delimited data to Avro?
I am looking for some help in using the JSON converter or advice if this is the right converter to do what I am trying to do. I have data coming in from a Kafka topic that looks like the following:
XYZ,1,10132977,-121.935583,37.505102,9,12,0,0.0,7/17/18 21:17:24,278.875,-1.160,0.360,9.740,-9.375,5.062,278.875,0.188,-0.750,-0.375,106,16,5,,IDLE;
The data does not have headers since we usually match these into a database schema where the fields match the exact order that the data comes over as.
So, the problem I am trying to solve is what is the best way to convert this text into JSON so that I can convert and compress it to Avro before stuffing it into S3? If I send a proper JSON message as a test from Kafka my current model works as intended but it seems apparent that I have to have JSON into to convert to the Avro schema. Can I use the JSON Generator for this? Is there a way to add the headers into the JSON message to field match the text data coming in?
I appreciate any guidance on this topic.
Is the data in the message actually surrounded by quotes?
No. The data is just comma separated text with no quotes. The quotes just show up in the Kafka consumer display. Sorry for the confusion.