Ask Your Question
1

How to set up a metric rules to trigger after every 10000 records are passed through a pipeline?

asked 2019-01-17 13:55:48 -0600

Vijay Shekhawat gravatar image

updated 2019-01-29 04:02:26 -0600

I can set up a metric rule to trigger an email or webhook event after a certain threshold like such:

${value() > 10000}

but is something like this possible?

${value() % 10000==0}

I have tried this in the condition block but it doesn't seem to be working. I am trying to trigger a webhook event after every 10000 records are passed through the pipeline.

In case I am not clear - Webhook should be triggered at 10000,20000,30000 and so on input records.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2019-01-17 16:00:29 -0600

iamontheinet gravatar image

updated 2019-01-17 16:02:33 -0600

Hi!

A couple of things: 1) For clarity, you might want to add parenthesis in your expression, like so ${(value() % 10000) == 0} and, more importantly, 2) The record count as evaluated by function value() is capped at pipeline batch size. So, if you increase production.maxBatchSize in sdc.properties file first and then restart data collector, it should work.

image description

Hope this helps.

Cheers, Dash

edit flag offensive delete link more

Comments

Hi @iamontheinet, I checked in sdc.properties as of now its production .maxBatchSize=1000. As per Documentation, this is- its Maximum number of records included in a batch when the pipeline runs. How does this ensure my webhook is called after every 1000 or 10000 records are passed through pipeline

Vijay Shekhawat gravatar imageVijay Shekhawat ( 2019-01-29 03:59:41 -0600 )edit

As per my answer above, whatever value you set for production.maxBatchSize you should be able to apply mod operation for a <= value. So if you leave it at 1000, ${(value() % 1000) == 0} should work. Note that value() count is per batch; not the total number of records passing through pipeline.

iamontheinet gravatar imageiamontheinet ( 2019-01-29 22:08:24 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-01-17 13:55:48 -0600

Seen: 48 times

Last updated: Jan 29