Ask Your Question

Runtime resources do not load dynamically in S3 origin

asked 2018-06-21 12:22:19 -0500

mp gravatar image

updated 2018-06-21 13:30:16 -0500

metadaddy gravatar image

I have a runtime resource defined per the instructions here. It is a formatted date that changes daily per a cronjob. I thought runtime resources were meant to load dynamically from the file and do not require a restart of the pipeline, per the answer in this post. I still have to restart in order for the value in the file to be recognized. Is there another solution?

edit retag flag offensive close merge delete


Where are you using the runtime resource - which stage/configuration property? It's possible that that stage is caching the value. There may be a workaround, though!

metadaddy gravatar imagemetadaddy ( 2018-06-21 12:32:31 -0500 )edit

Awesome. I am using it in the Amazon S3 Origin in the Amazon S3/Prefix Pattern configuration field: ${runtime:loadResource("yesterday.txt", false)}/*.log.gz

mp gravatar imagemp ( 2018-06-21 12:39:45 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2018-06-21 13:29:59 -0500

metadaddy gravatar image

I was unable to get this to work at all by putting the parameter in the Prefix Pattern configuration; I used Common Prefix.

You are correct that the S3 origin will not pick up changes to the resource file when you use it this way. This is because the origin is listening to S3 for changes that match that pattern, and has no mechanism for knowing that the file changed. Even if it did, you would likely run into issues because the offset (the pipeline's record of the last data it read) would be in one path, and you would be reading from another.

A more robust way to do this and achieve the automation you're looking for would be to use a pipeline parameter. You would reference the pipeline parameter in the configuration as ${awsprefix}, and set it when you start the pipeline, either in the UI (using Start With Parameters on the More menu) or via the API or CLI. You should reset the origin between one pipeline run and the next, otherwise you might run into the kind of problems I mention above. Again, you can do this in the UI, API or CLI.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-06-21 12:22:19 -0500

Seen: 1,975 times

Last updated: Jun 21 '18