How to upgrade a data pipeline running on production by OPS?

edit

The short answer to all of this is to use StreamSets Control Hub. Control Hub supports team development with

• a pipeline repository that supports versioning
• jobs that decouple the pipeline from the Data Collector instance
• labels that allow you to specify the Data Collector instance(s) that should run a job
• a central store for job offsets
• the ability to edit pipeline parameters on a job, rather than keeping them with the pipeline itself.

This is tricky to do manually in Data Collector. Importing a pipeline will assign it a new pipeline ID, so it will no longer be associated with the old offset. It would probably be better to stop the old pipeline, drop the new pipeline.json file into the data/pipelines/<pipelineId> directory, edit the pipeline ID to match the old one, then check and start the new pipeline. To be honest, you're on your own here.

As far as upgrading Data Collector goes, you can open older pipelines in newer versions of Data Collector, but not vice versa. You have to be careful that your pipeline developers aren't using a newer version of Data Collector than ops.