Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

ingest very large database tables over jdbc

I've been playing with StreamSets and love it! Let's say I have 10 billion row table in Oracle DB which is not partitioned and has a single column primary key. Is there a way to load that table using multiple threads? I use sqoop today with 32 mappers which takes about 5 hours. Once initial load is done, I can load data incrementally using only one thread.

Is it possible to use StreamSets for that initial load or sqoop still the best option? I have a few dozens very large tables like that and using single threaded jdbc origin is not really an option..

ingest very large database tables over jdbc

I've been playing with StreamSets and love it! Let's say I have 10 billion row table in Oracle DB which is not partitioned and has a single column primary key. Is there a way to load that table using multiple threads? I use sqoop today with 32 mappers which takes about 5 hours. Once initial load is done, I can load data incrementally using only one thread.

Is it possible to use StreamSets for that initial load or sqoop still the best option? I have a few dozens very large tables like that and using single threaded jdbc origin is not really an option..

ingest very large database tables over jdbc

I've been playing with StreamSets and love it! Let's say I have 10 billion row table in Oracle DB which is not partitioned and has a single column primary key. Is there a way to load that table using multiple threads? I use sqoop today with 32 mappers which takes about 5 hours. Once initial load is done, I can load data incrementally using only one thread.

Is it possible to use StreamSets for that initial load or sqoop still the best option? I have a few dozens very large tables like that and using single threaded jdbc origin is not really an option..