Ask Your Question

Pipeline: from csv to PostgreSQL

asked 2018-11-26 09:21:38 -0600

Daniel gravatar image


I'd like to load data every night from csv to PostgreSQL. I'd like to insert all the csv data into the target table. My questions are the following: - What is the best way to implement it in StreamSets? - Is there any PostgreSQL specific operator to handle this? - I think I can't use the PostgreSQL CDC operator for this batch job. Is it right?

Thanks in advance!

Regards, Daniel

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-11-26 10:42:26 -0600

Maithri gravatar image

updated 2018-11-28 03:13:45 -0600

You can schedule cron jobs for your pipeline to run every night.You can use Directory origin and JDBC Producer destination to load data from a file to PostgreSQL database.Configure the origin to select data from the CSV file with latest timestamp.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-11-26 09:21:38 -0600

Seen: 157 times

Last updated: Nov 28 '18