Ask Your Question

How to set up a metric rules to trigger after every 10000 records are passed through a pipeline?

asked 2019-01-17 13:55:48 -0500

Vijay Shekhawat gravatar image

updated 2019-01-29 04:02:26 -0500

I can set up a metric rule to trigger an email or webhook event after a certain threshold like such:

${value() > 10000}

but is something like this possible?

${value() % 10000==0}

I have tried this in the condition block but it doesn't seem to be working. I am trying to trigger a webhook event after every 10000 records are passed through the pipeline.

In case I am not clear - Webhook should be triggered at 10000,20000,30000 and so on input records.

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted

answered 2019-01-17 16:00:29 -0500

iamontheinet gravatar image

updated 2019-01-17 16:02:33 -0500


A couple of things: 1) For clarity, you might want to add parenthesis in your expression, like so ${(value() % 10000) == 0} and, more importantly, 2) The record count as evaluated by function value() is capped at pipeline batch size. So, if you increase production.maxBatchSize in file first and then restart data collector, it should work.

image description

Hope this helps.

Cheers, Dash

edit flag offensive delete link more


Hi @iamontheinet, I checked in as of now its production .maxBatchSize=1000. As per Documentation, this is- its Maximum number of records included in a batch when the pipeline runs. How does this ensure my webhook is called after every 1000 or 10000 records are passed through pipeline

Vijay Shekhawat gravatar imageVijay Shekhawat ( 2019-01-29 03:59:41 -0500 )edit

As per my answer above, whatever value you set for production.maxBatchSize you should be able to apply mod operation for a <= value. So if you leave it at 1000, ${(value() % 1000) == 0} should work. Note that value() count is per batch; not the total number of records passing through pipeline.

iamontheinet gravatar imageiamontheinet ( 2019-01-29 22:08:24 -0500 )edit

answered 2021-03-12 06:58:33 -0500

Mahender gravatar image

I have a requirement of sending the email notification with input and output records (not batch count), overall pipeline count.

my source is kafka consumer and target is http client. I have tried with data rules but it is not working? Could you please help regarding this?

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2019-01-17 13:55:48 -0500

Seen: 508 times

Last updated: Mar 12