Ask Your Question

How does Kafka producer treats data received from SDC origin

asked 2017-10-10 12:17:17 -0500

Roh gravatar image

updated 2017-10-10 16:58:25 -0500

metadaddy gravatar image

I have a follow-up question related to the below two questions :

I have to feed data into Kafka producer (Topic Name - X) from an SDC origin which gets fed from JDBC MultiTable consumer and write it to Hive/HDFS. So I was wondering if I can treat the further process after Kafka origin as if I'm reading from the JDBC Multitable consumer? Does the Kafka producer change anything after reading from the SDC Origin ??

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2017-10-10 16:57:56 -0500

metadaddy gravatar image

Yes - you can read records from JDBC, feed them into Kafka in one pipeline, read them out in another, then write them to Hive/HDFS. In fact, this is pretty much exactly what Cox Automotive does, ingesting data from their subsidiary companies into a central Hive data lake - see their use case and architecture in this webinar.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-10-10 12:17:17 -0500

Seen: 316 times

Last updated: Oct 10 '17