Ask Your Question
1

How do I improve performance reading from Oracle, writing to PostgreSQL?

asked 2019-05-10 02:02:16 -0500

anonymous user

Anonymous

updated 2019-05-10 20:49:07 -0500

metadaddy gravatar image

Team,

I am ingesting data from Oracle to PostgreSQL; my flow is JDBCQueryConsumer->FieldRenamer->EvalutionExpression->JDBCProducer but while importing the data it's running very slow. It's taking more than an hour for 100,000 records, fetching records in batches of 1000. How can I Increase performance?

Thanks.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2019-05-10 20:47:22 -0500

metadaddy gravatar image

You can increase the batch size above 1000 - note that you will need to edit the SDC configuration file - see this answer. Note also that there is a trade-off here between speed and memory usage. Bigger batches require more memory!

You can also get an idea of where it is spending the time by replacing the FieldRenamer->EvalutionExpression->JDBCProducer chain with a Trash destination, so it is just JDBCQueryConsumer->Trash. If this is still slow, then you should look at your query - are you using an unindexed column in a condition?

edit flag offensive delete link more

Comments

JDBCQueryConsumer->Trash the flow is working very fast but using FieldRenamer it's slow how will I increase that? I have an index column in origin

Dep gravatar imageDep ( 2019-05-12 00:24:20 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-05-10 02:02:16 -0500

Seen: 87 times

Last updated: May 10