Ask Your Question
1

HTTP Client , dynamically replace the text in URL

asked 2020-04-19 16:24:27 -0500

hishailesh77 gravatar image

updated 2020-04-21 12:20:51 -0500

I would like to transfer the data from HTTP Client to LocalFS , the data is located at https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_<YEAR>-<MONTH>.csv. With fixed URL e.g. https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2009-01.csv, I am successfully able to transfer the file , but I would like to replace the <year> and <month> part dynamically .

I have tried to use "FileTail", but not able to run the Pipeline successfully .

Could you please let me know the step by step instruction to configure this pipeline , so that i could transfer the files from above URL to Local FS .

Sample URLs are given below.

  • https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2009-01.csv
  • https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2009-02.csv
  • https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2010-01.csv
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2020-04-23 21:19:10 -0500

metadaddy gravatar image

The way to do this is to configure the Amazon S3 origin to process all files under a given path:

image description

edit flag offensive delete link more

Comments

I think the question has been misinterpreted , this is just a coincidence that NYC trip data is hosted on AWS and we can guess the region and bucket name from URL pattern. But i would like to treat the endpoint as any other HTTP endpoint (which could be hosted anywhere on internet other than AWS).

hishailesh77 gravatar imagehishailesh77 ( 2020-04-23 22:07:31 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-04-19 16:24:27 -0500

Seen: 42 times

Last updated: Apr 23