Ask Your Question
0

Jython evaluator sends an event on pipeline start

asked 2020-05-06 17:04:11 -0500

wilson shamim gravatar image

updated 2020-05-07 12:30:56 -0500

jeff gravatar image

hi Team,

I am creating a pipeline with jython evaluator in it. I noticed that it sends an event constantly when the pipeline is started tough there are no message in kafka consumer.

I am trying to generate a signle json payload for all the message from kafka in a batch.here is my jython code:

import json
lst=[]
sourceId = ''

for record in sdc.records:
  try:
    lst.append(record.value)
    sourceId = record.sourceId

  except Exception as e:
    # Send record to error
    sdc.error.write(record, str(e))
newRecord1 = sdc.createRecord(sourceId)
newRecord1.value = {'lst1':lst}
if newRecord1 != '':
  sdc.output.write(newRecord1)

Is this an expected behavior?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2020-05-10 12:18:22 -0500

wilson shamim gravatar image

updated 2020-05-10 14:19:55 -0500

Its resolved now. I added the below line in order to fix it

if len(lst) != 0: sdc.output.write(newRecord1) I believe this is what is happening, kafka consumer polls on a topic every few seconds, it completes the pipeline causing empty record sent to http client and invoking it.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-05-06 17:04:11 -0500

Seen: 43 times

Last updated: May 10