Ask Your Question

How to create a snapshot for one "stage" only and not for entire pipeline

asked 2019-05-01 08:09:32 -0500

rleyba gravatar image

updated 2019-05-01 09:19:36 -0500

metadaddy gravatar image

Hi team,

I have a scenario where my pipeline has multiple branches and I need to debug the stage(s) show in the pipeline screenshot below (field flattener3 ---> field type converter 3). My problem is the records going to this branch of the pipeline is so small (maybe one record out of 2000) such that every time I do a snapshot to capture the data traversing that branch of the pipeline, the snapshot always shows zero records. I have to take several snapshots just to raise the chance that I will eventually catch something.

I realize that there is an option to change the snapshot size (the default is 10) but if I were to do that, it would be a system wide setting and I don't want to capture that much data for each snapshot

Is there a way for streamsets to do a 10 record snapshot of data going through one particular stage only instead of for the whole pipeline?

Many thanks


edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2019-05-01 09:23:50 -0500

metadaddy gravatar image

You can't restrict a snapshot to a single stage, but there are a couple of tricks in Preview that you can use. You can either:

  • Preview the pipeline and edit the data so that it meets the criteria for that branch
  • Configure a Test Origin in the pipeline with the appropriate data, then preview the pipeline
edit flag offensive delete link more



Thanks Pat, I will take a look at the references you mentioned. Regards, rleyba

rleyba gravatar imagerleyba ( 2019-05-02 05:44:29 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2019-05-01 08:09:32 -0500

Seen: 104 times

Last updated: May 01 '19