Ask Your Question

HTTP Client , dynamically replace the text in URL

asked 2020-04-19 16:24:27 -0500

hishailesh77 gravatar image

updated 2020-04-21 12:20:51 -0500

I would like to transfer the data from HTTP Client to LocalFS , the data is located at<YEAR>-<MONTH>.csv. With fixed URL e.g., I am successfully able to transfer the file , but I would like to replace the <year> and <month> part dynamically .

I have tried to use "FileTail", but not able to run the Pipeline successfully .

Could you please let me know the step by step instruction to configure this pipeline , so that i could transfer the files from above URL to Local FS .

Sample URLs are given below.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2020-04-23 21:19:10 -0500

metadaddy gravatar image

The way to do this is to configure the Amazon S3 origin to process all files under a given path:

image description

edit flag offensive delete link more


I think the question has been misinterpreted , this is just a coincidence that NYC trip data is hosted on AWS and we can guess the region and bucket name from URL pattern. But i would like to treat the endpoint as any other HTTP endpoint (which could be hosted anywhere on internet other than AWS).

hishailesh77 gravatar imagehishailesh77 ( 2020-04-23 22:07:31 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2020-04-19 16:24:27 -0500

Seen: 42 times

Last updated: Apr 23