Ask Your Question
0

How to export the pipeline history summary

asked 2018-02-06 02:46:52 -0500

davidha gravatar image

Hello, Every time we run a pipeline, after the pipeline stopped, there is a nice GUI summary given about how many rows are loaded for each stage how many rows are error etc. However, is there a way to export that in file format so that we can keep track of the data load history in the past?

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted
0

answered 2018-02-12 00:50:25 -0500

Jisun gravatar image

You can try rest API, you can find the list of rest api that provided from SDC via http://<sdc hostname="">/collector/restapi

this is respond with the list of history of the pipeline => "/v1/pipeline/{pipelineId}/history"

edit flag offensive delete link more
0

answered 2018-02-06 05:08:36 -0500

Maithri gravatar image

updated 2018-02-06 05:30:04 -0500

Yow have option to download the log file,Which has the basic pipeline summary.

In case if you are speaking about preview data,You have an option called snapshot data, refer this link https://streamsets.com/documentation/...

edit flag offensive delete link more

Comments

For the option of downloading log files, do you mean garbing the log file with SCP directly? If so, the problem is that the log is keep appending to the same file, can we split it by time or by job level?

davidha gravatar imagedavidha ( 2018-02-06 22:52:09 -0500 )edit

For your requirement ,I would suggest snapshot in preview is a better option. I am not sure if you could split the log file as it keeps updating the same log file in the path specified during configuration.

Maithri gravatar imageMaithri ( 2018-02-07 00:24:32 -0500 )edit

https://streamsets.com/documentation/datacollector/latest/help/#Administration/Administration_title.html#task_gbm_s3k_br To download the latest log file, click Download. To download a specific log file, click Download > <file name="">.

Maithri gravatar imageMaithri ( 2018-02-07 00:43:06 -0500 )edit

The GUI is well designed, however, when we are trying to deploy in large scale (dynamic data pipelines), it is always a good idea to be able to gather the load detail history in some directory or even pipelines. I am looking for something like Rest-API to grab the history.

davidha gravatar imagedavidha ( 2018-02-07 03:20:53 -0500 )edit

Anyway, thank you for the info

davidha gravatar imagedavidha ( 2018-02-07 03:21:13 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-02-06 02:46:52 -0500

Seen: 69 times

Last updated: Feb 12