Ask Your Question

How to export the pipeline history summary

asked 2018-02-06 02:46:52 -0600

davidha gravatar image

Hello, Every time we run a pipeline, after the pipeline stopped, there is a nice GUI summary given about how many rows are loaded for each stage how many rows are error etc. However, is there a way to export that in file format so that we can keep track of the data load history in the past?

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted

answered 2018-02-12 00:50:25 -0600

Jisun gravatar image

You can try rest API, you can find the list of rest api that provided from SDC via http://<sdc hostname="">/collector/restapi

this is respond with the list of history of the pipeline => "/v1/pipeline/{pipelineId}/history"

edit flag offensive delete link more

answered 2018-02-06 05:08:36 -0600

Maithri gravatar image

updated 2018-02-06 05:30:04 -0600

Yow have option to download the log file,Which has the basic pipeline summary.

In case if you are speaking about preview data,You have an option called snapshot data, refer this link

edit flag offensive delete link more


For the option of downloading log files, do you mean garbing the log file with SCP directly? If so, the problem is that the log is keep appending to the same file, can we split it by time or by job level?

davidha gravatar imagedavidha ( 2018-02-06 22:52:09 -0600 )edit

For your requirement ,I would suggest snapshot in preview is a better option. I am not sure if you could split the log file as it keeps updating the same log file in the path specified during configuration.

Maithri gravatar imageMaithri ( 2018-02-07 00:24:32 -0600 )edit To download the latest log file, click Download. To download a specific log file, click Download > <file name="">.

Maithri gravatar imageMaithri ( 2018-02-07 00:43:06 -0600 )edit

The GUI is well designed, however, when we are trying to deploy in large scale (dynamic data pipelines), it is always a good idea to be able to gather the load detail history in some directory or even pipelines. I am looking for something like Rest-API to grab the history.

davidha gravatar imagedavidha ( 2018-02-07 03:20:53 -0600 )edit

Anyway, thank you for the info

davidha gravatar imagedavidha ( 2018-02-07 03:21:13 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-02-06 02:46:52 -0600

Seen: 73 times

Last updated: Feb 12 '18