Ask Your Question

Time functions as Pipeline Parameter

asked 2018-11-06 07:59:02 -0600

LuisFe Melo gravatar image

Dear all,

I would like to know if there are possible using time functions as a Pipeline Parameter.

I will try to get a daily file from the internet. This file has a name convention related to yesterday's date. For example, if a make the request today, the name of the file is “global-feed20181101.csv” If I make a request tomorrow, the name file would be “global-feed20181102.csv” And so on…

I tried to use the mentioned functions in the “Resource URL” parameter (HTTP Client) directly and as a "Pipeline Parameter". But, I got some errors.

Do you have any suggestion to solve that?

edit retag flag offensive close merge delete

1 Answer

Sort by » oldest newest most voted

answered 2018-11-07 10:59:33 -0600

iamontheinet gravatar image


Currently Time functions are not supported as part of the expression language when setting pipeline parameter values and Resource URL for HTTP Client origin. Please open an enhancement ticket by visiting

In the meantime, here are your options:

1) For Resource URL in Http Client origin, load it as a resource.

image description

2) Pass it in directly as a pipeline parameter or load it as a resource.

image description

For more details on runtime values, refer to the documentation.

Hope this helps.

Cheers, Dash

edit flag offensive delete link more


The problem appears when the filename is related to yesterday's date. The idea is to not hard-code the date but use something automatic (like time:now()) because we made an extraction daily. Do you have any suggestion, other than hard-code the name of the file?

LuisFe Melo gravatar imageLuisFe Melo ( 2018-11-07 13:40:20 -0600 )edit

I was not suggesting you hard-code it in the resource file. I was presenting an idea where you can programmatically (using script, for example) write/update filename in the resource file and then have the pipeline read it using resource:loadResource.

iamontheinet gravatar imageiamontheinet ( 2018-11-07 13:48:45 -0600 )edit

Sorry, I had not seen the function in the first picture. Thanks so much!!!

LuisFe Melo gravatar imageLuisFe Melo ( 2018-11-08 02:34:13 -0600 )edit

Because I don't have permissions in the (StreamSets) server, I decided to use an ExpressionEvaluator to add Time variables and an HTTP Client to use it. It works now! However, I only get one record from the CSV file. Do you have any idea to solve that?

LuisFe Melo gravatar imageLuisFe Melo ( 2018-11-16 02:57:21 -0600 )edit

Please ask a new question with all the relevant details including logs etc.

iamontheinet gravatar imageiamontheinet ( 2018-11-16 10:21:40 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-11-06 07:59:02 -0600

Seen: 4,796 times

Last updated: Nov 07 '18