Ask Your Question

Is it possible to access contents(ex:Timestamp value) of pipeline History from the UI?

asked 2018-02-04 23:36:01 -0500

Maithri gravatar image

updated 2018-02-14 22:26:18 -0500

I want to make use of pipeline's last runtime to fetch table data,Is it possible that we can access and include streamsets pipeline log information in a pipeline?....Suppose I want to fetch table data after comparing the last updated time of table and the last run time of a pipeline??

Is it possible to access and use the Timestamp data which is available in the following?

edit retag flag offensive close merge delete


I also need to know this! Either this solution or have a variable attached to the pipeline that can be read and updated.

Fayan gravatar imageFayan ( 2018-03-12 11:58:31 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2018-03-12 13:55:34 -0500

jeff gravatar image

I'm not sure if this is exactly what you're after, but Data Collector provides a REST API that may provide the data you're interested in. One method is GET /v1/pipeline/{pipelineId}/history - which will basically show the data that is shown in the UI screenshot you linked. Another is GET /v1/pipeline/{pipelineId}/committedOffsets - which will return the committed offsets from the pipeline (assuming the origin tracks them). For more information on the API, use the Help menu in Data Collector or view the documentation here.

edit flag offensive delete link more


thanks for the response @jeff

Maithri gravatar imageMaithri ( 2018-03-14 01:09:18 -0500 )edit
Login/Signup to Answer

Question Tools



Asked: 2018-02-04 23:36:01 -0500

Seen: 3,095 times

Last updated: Mar 12 '18