Ask Your Question

Connect Kafka to Google BigQuery

asked 2017-09-08 21:44:36 -0500

etler gravatar image

updated 2017-09-12 17:17:35 -0500

metadaddy gravatar image

I need to connect with kafka in my pipeline to Google BigQuery/Google Cloud Storage as target and source db.

  • What is the best way in StreamSets to do that?
  • How to configure Google BigQuery to do that?
edit retag flag offensive close merge delete


Can you please clarify somewhat? Are you saying that Kafka is the source, or that you are wanting to use Kafka as an intermediate step? In general, each SDC pipeline has a single source (origin) and destination.

jeff gravatar imagejeff ( 2017-09-11 13:09:06 -0500 )edit

My original sources are SQL server and SFTP. From these data is being pushed into Kafka. Now since the data is in Kafka, I should be able to send this data to Google BigQuery as the destination.

etler gravatar imageetler ( 2017-09-11 13:16:01 -0500 )edit

1 Answer

Sort by » oldest newest most voted

answered 2017-09-11 16:35:33 -0500

jeff gravatar image

updated 2017-09-12 17:16:18 -0500

metadaddy gravatar image

You should be able to create a pipeline with a Kafka origin and Google BigQuery target. Are you running into any issues trying to set that up?

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-09-08 21:44:36 -0500

Seen: 323 times

Last updated: Sep 12 '17