Ask Your Question
1

How can I set the filename when writing binary data format from HTTP Client

asked 2020-02-13 02:52:22 -0600

Larry gravatar image

updated 2020-02-13 10:11:56 -0600

metadaddy gravatar image

I am fetching multiple base64 encoded files via HTTP Client. I am decoding the files into a binary field (/fileRef). I have a defined name for each file (/filename).

After that the files shall be written to a directory (local fs or HDFS).

The files are all written successfully, but the filenames are random. Is there a possibility to set the filename?

The Pipeline looks like this: ImgUrl

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2020-02-13 10:20:40 -0600

metadaddy gravatar image

The Hadoop FS destination uses a UUID in the filenames to avoid inadvertent overwrites. You can enable the destination to produce events (checkbox on the General tab) and then use the HDFS File Metadata Executor to rename the file.

You'll probably want to set some part of the filename in the Hadoop FS Destination (you can do this via Files Prefix on the Output Files tab) to your desired final filename and extract that from the event record using the string EL functions.

edit flag offensive delete link more

Comments

Thank you! If I want to write the files to a Local FS (the HDFS destination is not optimal for such small files): is there a way to achieve that (for example by setting headers in an expression or in JS evaluator)?

Larry gravatar imageLarry ( 2020-02-13 13:30:04 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-02-13 02:50:51 -0600

Seen: 23 times

Last updated: Feb 13