Ask Your Question

How to eliminate sdc default prefix and suffix from filename while writing data to a file in local FS?

asked 2018-10-04 05:30:57 -0600

Maithri gravatar image

updated 2018-10-09 01:24:29 -0600

My origin is JDBC Query Consumer and destination is local FS. I am writing my query data to a file in json format.I want my filename to exclude the SDC prefix and suffix and literally match some filename pattern.As every time first pipeline creates a new file and writes the data,that file is used as input to the second pipeline.Say employee as a file name.How to achieve this?

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted

answered 2018-10-09 00:32:15 -0600

Maithri gravatar image

updated 2018-10-09 00:50:06 -0600

I could solve this by specifying regular expression pattern and select files based on Last Modified Timestamp.

edit flag offensive delete link more

answered 2018-10-04 10:12:59 -0600

iamontheinet gravatar image

Hi! It appears that you can get rid of everything but the unique data collector ID (i.e. sdc:id()) from filenames. However, note that you can set directory (i.e. Directory Template) to be anything you'd like including record field values such as employee Id, name etc.

Cheers, Dash

edit flag offensive delete link more


My problem is, the output of this pipeline is the input to another pipeline.If i want to schedule that pipeline i need to know the filename.With these sdc prefix and suffixes the filename keeps changing.How to set a fixed filename?

Maithri gravatar imageMaithri ( 2018-10-04 23:33:41 -0600 )edit

can you show with an example

Maithri gravatar imageMaithri ( 2018-10-04 23:33:59 -0600 )edit

Like I mentioned, you can't access record attributes (such as, employee id, name, etc.) in filenames and there's no way to chop off sdc:id(). On the other hand, to customize directory name you can set Directory Template to something like, /filepath/${record:value('/field1')}.

iamontheinet gravatar imageiamontheinet ( 2018-10-05 10:17:04 -0600 )edit

Thanks for letting me know that we cannot eliminate SDC id from filename.

Maithri gravatar imageMaithri ( 2018-10-09 01:21:27 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-10-04 05:30:57 -0600

Seen: 683 times

Last updated: Oct 09 '18