Ask Your Question
0

Time functions as Pipeline Parameter

asked 2018-11-06 07:59:02 -0600

LuisFe Melo gravatar image

Dear all,

I would like to know if there are possible using time functions as a Pipeline Parameter.

I will try to get a daily file from the internet. This file has a name convention related to yesterday's date. For example, if a make the request today, the name of the file is “global-feed20181101.csv” If I make a request tomorrow, the name file would be “global-feed20181102.csv” And so on…

I tried to use the mentioned functions in the “Resource URL” parameter (HTTP Client) directly and as a "Pipeline Parameter". But, I got some errors.

Do you have any suggestion to solve that?

edit retag flag offensive close merge delete

1 Answer

Sort by » oldest newest most voted
0

answered 2018-11-07 10:59:33 -0600

iamontheinet gravatar image

Hi!

Currently Time functions are not supported as part of the expression language when setting pipeline parameter values and Resource URL for HTTP Client origin. Please open an enhancement ticket by visiting https://issues.streamsets.com.

In the meantime, here are your options:

1) For Resource URL in Http Client origin, load it as a resource.

image description

2) Pass it in directly as a pipeline parameter or load it as a resource.

image description

For more details on runtime values, refer to the documentation.

Hope this helps.

Cheers, Dash

edit flag offensive delete link more

Comments

The problem appears when the filename is related to yesterday's date. The idea is to not hard-code the date but use something automatic (like time:now()) because we made an extraction daily. Do you have any suggestion, other than hard-code the name of the file?

LuisFe Melo gravatar imageLuisFe Melo ( 2018-11-07 13:40:20 -0600 )edit

I was not suggesting you hard-code it in the resource file. I was presenting an idea where you can programmatically (using script, for example) write/update filename in the resource file and then have the pipeline read it using resource:loadResource.

iamontheinet gravatar imageiamontheinet ( 2018-11-07 13:48:45 -0600 )edit

Sorry, I had not seen the function in the first picture. Thanks so much!!!

LuisFe Melo gravatar imageLuisFe Melo ( 2018-11-08 02:34:13 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-11-06 07:59:02 -0600

Seen: 66 times

Last updated: Nov 07