Ask Your Question
0

Add date and time to file prefix

asked 2018-01-15 12:33:48 -0500

Invigor gravatar image

updated 2018-01-18 15:24:20 -0500

Roh gravatar image

We are looking to use StreamSets to replace an in-house logging system which includes the date and time in the file name.

Is there any way to configure the File Prefix or File Suffix to include the date and/or time?

I only seem to be able to do this with the directory template.

The pipeline is as follows:

ORIGIN An HTTP Client origin calls an API every 60 seconds and receives a JSON array of event data

  1. A Jython Evaluator processor parses the JSON in order to create individual records for each element of the JSON array
  2. An Expression Evaluator processor converts the records to List-Map format
  3. A 2nd Expression Evaluator processor adds some static fields and does some format conversions (timestamp, etc)
  4. A Field Order processor puts the fields in the correct order

DESTINATION A Local File destination saves the records to local disk in TSV format

I'm happy to look at other Destinations e.g. S3 if that will help with getting the file names in the correct format.

Thanks

edit retag flag offensive close merge delete

Comments

Can you add small description of your pipeline like origin and destination? if we know the destination you are using we can help you with adding the timestamp in the files name.

Roh gravatar imageRoh ( 2018-01-16 14:42:51 -0500 )edit

Thanks for updating my question with the details. I’ve deleted my answer.

Invigor gravatar imageInvigor ( 2018-01-18 15:41:07 -0500 )edit

1 Answer

Sort by » oldest newest most voted
0

answered 2020-06-11 18:02:34 -0500

iamontheinet gravatar image

Hi!

What you could is rename the output files using Events framework and Shell Executor. Here's a quick 2m24sec"how-to" video that will show you how to rename files after they're written out by/from Local FS destination -- https://youtu.be/VRqnHmHuZFw.

Cheers, Dash

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-01-15 12:33:48 -0500

Seen: 663 times

Last updated: Jun 11