Ask Your Question
1

Pipeline Finisher with no data

asked 2018-08-29 09:59:30 -0600

anna.szabo-simon gravatar image

I tried Pipeline Finisher executor with Amazon S3 origin. It works fine, except when there is no data to begin with - then it never stops. Do you have any idea how to trigger it?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2019-01-17 11:25:36 -0600

metadaddy gravatar image

By design, origins do not produce a no-more-data event if there is no data when the pipeline starts. You can, however, use a script evaluator to detect this condition and produce an event. I used the Jython evaluator, but you could use Groovy or JavaScript if you prefer. Add the evaluator after the origin, and enable 'Produce Events' on its 'General' tab. Connect the event output to your pipeline finisher:

image description

Set the evaluator's init script to initialize state:

state['done'] = False

Now set the evaluator's script to send an event if the first batch it sees is empty

if len(records) == 0 and not state['done']:
  event_record = sdcFunctions.createEvent('no-more-data', 1)
  event_record.value = {'record-count' : 0, 'error-count' : 0, 'file-count' : 0}
  sdcFunctions.toEvent(event_record)

state['done'] = True

Now the pipeline will immediately stop if there is no data at startup.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

2 followers

Stats

Asked: 2018-08-29 09:59:30 -0600

Seen: 87 times

Last updated: Jan 17