Ask Your Question
1

Connect Kafka to Google BigQuery

asked 2017-09-08 21:44:36 -0600

etler gravatar image

updated 2017-09-12 17:17:35 -0600

metadaddy gravatar image

I need to connect with kafka in my pipeline to Google BigQuery/Google Cloud Storage as target and source db.

  • What is the best way in StreamSets to do that?
  • How to configure Google BigQuery to do that?
edit retag flag offensive close merge delete

Comments

Can you please clarify somewhat? Are you saying that Kafka is the source, or that you are wanting to use Kafka as an intermediate step? In general, each SDC pipeline has a single source (origin) and destination.

jeff gravatar imagejeff ( 2017-09-11 13:09:06 -0600 )edit

My original sources are SQL server and SFTP. From these data is being pushed into Kafka. Now since the data is in Kafka, I should be able to send this data to Google BigQuery as the destination.

etler gravatar imageetler ( 2017-09-11 13:16:01 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted
1

answered 2017-09-11 16:35:33 -0600

jeff gravatar image

updated 2017-09-12 17:16:18 -0600

metadaddy gravatar image

You should be able to create a pipeline with a Kafka origin and Google BigQuery target. Are you running into any issues trying to set that up?

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-09-08 21:44:36 -0600

Seen: 45 times

Last updated: Sep 12 '17