Passing batchMaker to custom class makes it unusable by that class.

asked 2018-07-20 15:00:03 -0600

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

It might be a dumb question and I might not understand Streamsets well enough but I wanted to get a second opinion.

I'm trying to build a custom processor. It will accept one record and output multiple ( not determined number of records ).

"Custom processing" was already implemented as a class ( several other classes help out ). It doesn't return anything but it holds the business logic and eventually it outputs a number of records. What I wanted to do is pass record and batchMaker to my custom class and create new records as they come and place them in a batch.

I've figured out how to make a new instance of a record through context, I'm populating them and adding them to the batchMaker (passed to this custom class, so not in process method) but batch of records is never passed down the pipeline after processing is done.

I'm sure I'm missing something. I've followed the example on github and it works when I add records to batchMaker in process method but if I pass it into another class/method it doesn't seem to work.

edit retag flag offensive close merge delete


It's hard to figure out why this might be without seeing your code. Would it be possible to create a minimal example that reproduces the problem, and edit it into your question?

metadaddy gravatar imagemetadaddy ( 2018-07-20 15:08:39 -0600 )edit

I can second @metadaddy - it's hard to help without seeing the actual source code. Do you happen to have it available somewhere?

Jarcec gravatar imageJarcec ( 2018-07-24 14:31:49 -0600 )edit