Ask Your Question

How can I set the filename when writing binary data format from HTTP Client

asked 2020-02-13 02:52:22 -0500

Larry gravatar image

updated 2020-02-13 10:11:56 -0500

metadaddy gravatar image

I am fetching multiple base64 encoded files via HTTP Client. I am decoding the files into a binary field (/fileRef). I have a defined name for each file (/filename).

After that the files shall be written to a directory (local fs or HDFS).

The files are all written successfully, but the filenames are random. Is there a possibility to set the filename?

The Pipeline looks like this: ImgUrl

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2020-02-13 10:20:40 -0500

metadaddy gravatar image

The Hadoop FS destination uses a UUID in the filenames to avoid inadvertent overwrites. You can enable the destination to produce events (checkbox on the General tab) and then use the HDFS File Metadata Executor to rename the file.

You'll probably want to set some part of the filename in the Hadoop FS Destination (you can do this via Files Prefix on the Output Files tab) to your desired final filename and extract that from the event record using the string EL functions.

edit flag offensive delete link more


Thank you! If I want to write the files to a Local FS (the HDFS destination is not optimal for such small files): is there a way to achieve that (for example by setting headers in an expression or in JS evaluator)?

Larry gravatar imageLarry ( 2020-02-13 13:30:04 -0500 )edit

You can use the Local FS destination with a Shell Executor to rename files. See

metadaddy gravatar imagemetadaddy ( 2020-05-09 12:35:24 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2020-02-13 02:50:51 -0500

Seen: 75 times

Last updated: Feb 13