Ask Your Question

How can I query or report on my pipelines?

asked 2018-05-14 14:52:34 -0600

EyeOpener gravatar image

I'm trying to gather some documentation on our StreamSets pipelines. Is it possible to query the Data Collector repository? For example, I'd like to identify all our pipelines that use a JDBC query origin and have a certain set of tags. For those, I'd like to pull pieces of information like the pipeline title, JDBC connection string, and SQL query, and retrieve those as a CSV or insert them into a database table (this will be combined with other documentation from other sources). Is that all possible? Thanks!

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-05-15 09:31:22 -0600

jeff gravatar image

The Data Collector REST API provides a lot of what you're asking about, in JSON format. It is accessible once Data Collector is running and you provide the proper authentication with your request. Please see documentation here or use the Help menu to access the API information from the UI.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-05-14 14:52:34 -0600

Seen: 155 times

Last updated: May 15 '18