Ask Your Question
1

How do I recover existing pipelines?

asked 2018-01-23 23:25:49 -0500

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

Our existing StreamSets instance was not accessible so we restarted it by firing "sudo streamsets dc", however when the instance was up it is not showing existing pipelines (comes up as if it is launched for the first time). Has anyone faced such issue before?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2018-01-24 11:18:26 -0500

metadaddy gravatar image

Look in the $SDC_DATA directory - by default, this is the data directory wherever you installed StreamSets Data Collector, but it might have been set in sdc-env.sh or sdcd-env.sh to be somewhere else. See the documentation on configuring the SDC environment for more details. If the files are not there, then look for files named pipeline.json on your disk.

There's no reason SDC would delete pipelines at restart, so my guess is that the environment is somehow different from when you started it before and when you're starting it now. For example, if you previously started SDC as a service and you're not starting it manually, it could be reading the $SDC_DATA value from a different file - sdc-env.sh rather than sdcd-env.sh.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-01-23 23:25:49 -0500

Seen: 157 times

Last updated: Jan 24