Ask Your Question
1

Use current EPOCH time (in seconds) as a parameter in pipeline

asked 2019-04-15 09:41:26 -0500

srinath_222 gravatar image

updated 2019-04-15 09:41:47 -0500

Hi,

I need to extract the current EPOCH time (in seconds) and use it as a parameter in Streamsets pipeline and use it for pulling incremental data from a database origin.

I am trying to use the below expression to get epoch time but it is giving in milliseconds ${time:dateTimeToMilliseconds(time:now())}. Trying to use below but its not working: ${time:dateTimeToMilliseconds(time:now())/1000} but not working.

Please suggest how to extract current EPOCH time in seconds and how to use that in an expression in parameter

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted
2

answered 2019-04-19 16:14:52 -0500

iamontheinet gravatar image

updated 2019-04-22 09:22:17 -0500

Hi!

This expression ${math:round(time:dateTimeToMilliseconds(time:now()) / 1000)} will give you current time converted to Epoch in seconds.

Cheers, Dash

edit flag offensive delete link more

Comments

Thanks Dash...It is giving epoch time in seconds. Can I get this expression evaluated and used as a run time parameter ?? I need to use this parameter in configuration of one of the origin.

srinath_222 gravatar imagesrinath_222 ( 2019-04-22 09:24:35 -0500 )edit

No, time:dateTimeToMilliseconds(time:now()) function is not available to use when configuring pipeline parameter. Which origin specifically?

iamontheinet gravatar imageiamontheinet ( 2019-04-22 09:41:31 -0500 )edit

I want to use in MongoDB oplog origin for Initial Timestamp property which captures changes in MongoDB from current timestamp. I want to get the current epoch time in seconds and use it as run time parameter for this origin.

srinath_222 gravatar imagesrinath_222 ( 2019-04-22 09:47:01 -0500 )edit
iamontheinet gravatar imageiamontheinet ( 2019-04-22 09:55:35 -0500 )edit

Thanks Dash for suggesting runtime resources. Can I use the expression math:round(time:dateTimeToMilliseconds(time:now()) / 1000) in a text file for runtime resources (saved in $SDC_RESOURCES directory) ???

srinath_222 gravatar imagesrinath_222 ( 2019-04-22 10:11:08 -0500 )edit
0

answered 2019-04-26 16:46:06 -0500

srinath_222 gravatar image

I tried using runtime resources. Used param1 as ${runtime:loadResource("epoch.txt", false)} and called in Initial Timestamp property of MongoDB Oplog Origin.

It is giving error as below:

CREATION_005 - Could not resolve implicit EL expression '${param1}': com.streamsets.pipeline.api.el.ELEvalException: CTRCMN_0100 - Error evaluating expression ${param1}: javax.servlet.jsp.el.ELException: An exception occured trying to convert String "math:round(time:dateTimeToMilliseconds(time:now()) / 1000)" to type "int"

Looks like it runtime resources is supporting only string types and in the origin it is expecting int data type. I was unable to convert to Int type.

For the second option of starting pipeline using REST API, I am not sure of how to proceed. We do not have Control Hub setup, all the documentation refers to Streamsets Control Hub.

edit flag offensive delete link more

Comments

@iamontheinet any suggestions ??

srinath_222 gravatar imagesrinath_222 ( 2019-05-14 08:31:51 -0500 )edit

What do you have in epoch.txt? You can't have math:round(time:dateTimeToMilliseconds(time:now()) / 1000) in there. You need to specify the actual epoch value in the resource file -- which you'll have to do using some script before you start the pipeline.

iamontheinet gravatar imageiamontheinet ( 2019-05-14 18:34:13 -0500 )edit

BTW, SDC has REST APIs as well -- in your SDC go to /collector/restapi and you'll see the endpoints.

iamontheinet gravatar imageiamontheinet ( 2019-05-14 18:36:29 -0500 )edit

Will try it Dash. I am trying to get Initial Timestamp dynamically so that I do not miss any data from MongoDB Oplog origin. Does Mongodb oplog origin store the offset such that next time it picks from last load time. Posted as https://ask.streamsets.com/question/5120/offset-in-mongodb-oplog/

srinath_222 gravatar imagesrinath_222 ( 2019-05-15 09:18:10 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-04-15 09:41:26 -0500

Seen: 96 times

Last updated: Apr 26