Ask Your Question

How do I improve performance reading from Oracle, writing to PostgreSQL?

asked 2019-05-10 02:02:16 -0600

anonymous user


updated 2019-05-10 20:49:07 -0600

metadaddy gravatar image


I am ingesting data from Oracle to PostgreSQL; my flow is JDBCQueryConsumer->FieldRenamer->EvalutionExpression->JDBCProducer but while importing the data it's running very slow. It's taking more than an hour for 100,000 records, fetching records in batches of 1000. How can I Increase performance?


edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2019-05-10 20:47:22 -0600

metadaddy gravatar image

You can increase the batch size above 1000 - note that you will need to edit the SDC configuration file - see this answer. Note also that there is a trade-off here between speed and memory usage. Bigger batches require more memory!

You can also get an idea of where it is spending the time by replacing the FieldRenamer->EvalutionExpression->JDBCProducer chain with a Trash destination, so it is just JDBCQueryConsumer->Trash. If this is still slow, then you should look at your query - are you using an unindexed column in a condition?

edit flag offensive delete link more


JDBCQueryConsumer->Trash the flow is working very fast but using FieldRenamer it's slow how will I increase that? I have an index column in origin

Dep gravatar imageDep ( 2019-05-12 00:24:20 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2019-05-10 02:02:16 -0600

Seen: 238 times

Last updated: May 10 '19