How to run multiple pipelines to load delta in postgres
My scenario is that I would like to load delta from one schema in postgres to another schema in the same postgres db. I don't want to do a full load. Instead, an incremental load with update or delete.
The solution I found is:
- Origin : JDBC Query Consumer :
select * from postgreg_schema01.view_a - Destination : JDBC Query Executor :
insert into postgres_schema02.table_a on conflict do update set...(upsert)
But I have to create 20 different pipeline for 20 different views. Is there any way to create one pipeline instead of 20 different pipeline to solve this scenario?
Can you post some details of what you tried? What did your pipeline look like? What query did you use? What was the result - did you see an error?
In my scenario , after I posted question in here , I applied a solution. It is ; Origin : JDBC Query Consuemer : select * from postgreg_schema01.view_a Destination : JDBC Query : insert into pgostgre_schema02.table_a on conflict do update set... (upsert) But nearly I have to create 20 different pi
Can you edit your question to include that information? it looks like you ran out of space.
@metadaddy does streamsets supports multiple running instances of the same pipeline? Example: If I have a streaming pipeline, can I run 2 concurrent instances of it using different parameters?
I tidied up your question and posted an answer. Thanks for giving some clarification!