Ask Your Question
1

Saving State In-Memory

asked 2020-05-10 17:29:04 -0600

Peter Delaney gravatar image

I am looking for ideas on how to save state in a pipeline without using an external persistence from the SDC Pipeline. I need to keep track of the number of articles processed for a particular jobId/batchId and at some point send this state to stdout or rsyslog. I am open to other peoples ideas on how to achieve this using the basic SDC stages.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2020-05-10 23:37:55 -0600

iamontheinet gravatar image

updated 2020-05-11 11:26:32 -0600

Hi Peter,

You can use state object in one of the three scripting processors. For example, check out Groovy Evaluator. Also note that if you add one of the scripting processors in the pipeline, you'll see sample code in its Init Script section.

Here's an example of using stateobject -- https://ask.streamsets.com/question/4...

Hope this helps.

Cheers, Dash

edit flag offensive delete link more

Comments

1

Thank you I will play with this.

Peter Delaney gravatar imagePeter Delaney ( 2020-05-11 11:25:30 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-05-10 17:29:04 -0600

Seen: 99 times

Last updated: May 11