Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Batch Processing

Hi Guys,

How can I fetch the data from any database (mysql, oracle etc) in batches use case is to fetch data for every 5 minutes? Also I want to run my processing in parallel/distributed means read data in one sort and process in parallel e.g. read 1000 record in 5 minute but divide 200 and process (cleaning, apply business rules etc) in parallel and then load into another db. Can anyone help how can I do this.

Batch Processing

Hi Guys,

How can I fetch the data from any database (mysql, oracle etc) in batches use case is to fetch data for every 5 minutes? Also I want to run my processing in parallel/distributed means read data in one sort and process in parallel e.g. read 1000 record in 5 minute but divide 200 and process (cleaning, apply business rules etc) in parallel and then load into another db. db. Why I want to do this because want to complete my whole pipeline within 5 minute so that while running next batch there will not be any discrepancy. So reading is fine but processing is the task

Can anyone help how can I do this.this?

Batch Processing

Hi Guys,

How can I fetch the data from any database (mysql, oracle etc) in batches use case is to fetch data for every 5 minutes? Also I want to run my processing in parallel/distributed means read data in one sort and process in parallel e.g. read 1000 record in 5 minute but divide 200 and process (cleaning, apply business rules etc) in parallel and then load into another db. Why I want to do this because want to complete my whole pipeline within 5 minute so that while running next batch there will not be any discrepancy. discrepancy also if any point load is high like weekends or in festival season then again I can run pipeline seamlessly. So reading is fine but processing is the task

Can anyone help how can I do this?