Add date and time to file prefix

asked 2018-01-15 12:33:48 -0600

Invigor gravatar image

updated 2018-01-18 15:24:20 -0600

Roh gravatar image

We are looking to use StreamSets to replace an in-house logging system which includes the date and time in the file name.

Is there any way to configure the File Prefix or File Suffix to include the date and/or time?

I only seem to be able to do this with the directory template.

The pipeline is as follows:

ORIGIN An HTTP Client origin calls an API every 60 seconds and receives a JSON array of event data

  1. A Jython Evaluator processor parses the JSON in order to create individual records for each element of the JSON array
  2. An Expression Evaluator processor converts the records to List-Map format
  3. A 2nd Expression Evaluator processor adds some static fields and does some format conversions (timestamp, etc)
  4. A Field Order processor puts the fields in the correct order

DESTINATION A Local File destination saves the records to local disk in TSV format

I'm happy to look at other Destinations e.g. S3 if that will help with getting the file names in the correct format.

Thanks

edit retag flag offensive close merge delete

Comments

Can you add small description of your pipeline like origin and destination? if we know the destination you are using we can help you with adding the timestamp in the files name.

Roh gravatar imageRoh ( 2018-01-16 14:42:51 -0600 )edit

Thanks for updating my question with the details. I’ve deleted my answer.

Invigor gravatar imageInvigor ( 2018-01-18 15:41:07 -0600 )edit