Ask Your Question

How to get row count as part of filename?

asked 2020-07-21 16:02:37 -0500

amnopro gravatar image

I am using JDBC Query consumer as my origin and Local FS as my destination. I need the number of rows fetched to be part of filename before closing(writing) the file. Similar pipeline is setup for Oracle CDC origin. I need rewcount to be written as filename, So that I can use that for validation in my downstream systems. Please help me if there is a way for this?! Thanks in advance.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2020-07-21 18:49:04 -0500

iamontheinet gravatar image


The number of records that get written to a file depends on many attributes like Max Records in File, Max File Size (MB), Idle Timeout, Use Roll Attribute, Late Record Time Limit (secs) and possibly combinations of those attributes. (Also note that based on the configuration several output files could get created.) So, AFAIK, there's no easy way to get a count of records written to a specific file.

The two options you might want to explore are:

  1. Enable JDBC Query Consumer origin to produce events and then note record-count available via no-more-data event. For details, refer to the documentation.
  2. Use Notifications >> Webhook to get notified of attributes like PIPELINE_INPUT_RECORDS_COUNT and PIPELINE_OUPUT_RECORDS_COUNT among other things. For details, refer to the documentation.

Hope this helps.

Cheers, Dash

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2020-07-21 16:02:37 -0500

Seen: 719 times

Last updated: Jul 21 '20