Ask Your Question

How to control the data in pipeline suggested procceser?

asked 2020-02-20 10:14:08 -0500

anonymous user


Hi guys, I'm using Jython origin to read the API. I have a question.

  • I can't control the API data we don't have any parameters from API to get the data from date A to date B. we are getting user response for each app. So data from API is very random but we need from 2018 to current ( How to control the data in a pipeline before storing the result in output I want to trash the data before 2018) can you let me know is there any processer that can trash the data if it is before X date?

if any processer is available can you please let me know how we can use it in my case (dates are converted to string from unix timestamp).


edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2020-02-20 10:33:27 -0500

jeff gravatar image

If you're already setting up a Jython origin to read the data in the first place, you can just add some additional Python code there to do the filtering right there. Simply don't add those records to the current batch. Alternately, you can create a Stream Selector processor after the origin to send the undesired records to a Trash destination, based on evaluating the appropriate condition within the record fields. Have a look at the documentation here and here for more information on creating the expressions.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2020-02-20 10:14:08 -0500

Seen: 38 times

Last updated: Feb 20