Ask Your Question

How to limit the number of records to read from a flat file?

asked 2018-10-17 01:47:25 -0500

Vijay Shekhawat gravatar image

updated 2018-10-17 01:58:29 -0500

I want to limit the number of records I read from a flat file, We can change the batch size but can I put a limit on the number of batches itself?

Basically, I want to read only 1 record from a file in a directory and generate an event.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-10-18 10:14:10 -0500

ak47 gravatar image

Can you explain what your usecase / what are you trying to do? Why do you need to pull only one record from the file? Why do you need to generate an event ?

You could simply put a batchsize of 1 and in parallel write to both a destination and a pipeline finisher and will accomplish the same thing without generating an event per se.

edit flag offensive delete link more


How about testing a pipe with limited number of record or a small subset of record instead of running for a large volume?

Prasanna gravatar imagePrasanna ( 2019-05-28 02:07:20 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-10-17 01:47:25 -0500

Seen: 432 times

Last updated: Oct 17 '18