Ask Your Question

How to get the timestamp value from the logs?

asked 2018-02-01 00:11:57 -0500

Shruthi gravatar image

updated 2018-02-05 07:18:04 -0500

How can i get the Timestamp from the logs of stremasets. I want to implement CDC. For that i want to compare the pipeline's last run time with the updated time of data.The Timestamp in streamset logs can give the last run time. So how can i get it?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-02-05 14:57:03 -0500

Jisun gravatar image

I am not sure if I understood your goal correctly. One approach that I can think of is store the current timestamp on the CDC offset so that you can track down the last timestamp via last source offset. This is if you mean you want to build customer stage library for CDC. Another approach is to get the timestamp is via the pipeline statues file located in $SDC_DATA/runInfo/<pipeline id="">/0/pipelineStateHistory.json

If those two approaches won't work in your case. Can you elaborate a bit more about your use case?

edit flag offensive delete link more
Login/Signup to Answer

Question Tools



Asked: 2018-02-01 00:11:57 -0500

Seen: 201 times

Last updated: Feb 05 '18