Ask Your Question

Adding processing steps after destination in Data Collector

asked 2018-02-08 00:40:19 -0500

aakansha gravatar image

updated 2018-02-09 15:14:44 -0500

jeff gravatar image

Is there a way to add processor after destination is added, in the same pipeline? I want to run shell scripts after the data is loaded in HDFS (destination).

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-02-09 15:19:12 -0500

jeff gravatar image

For certain destinations, this is possible. Specifically for the HDFS destination, you can enable events on the destination, and an event will be generated when each file is closed. You can then attach an executor (including shell executor, for your case) to that event lane. See the specific page for HDFS destination for more information on that.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-02-08 00:40:19 -0500

Seen: 333 times

Last updated: Feb 09 '18