Ask Your Question
0

Adding processing steps after destination in Data Collector

asked 2018-02-08 00:40:19 -0600

updated 2018-02-09 15:14:44 -0600

jeff gravatar image

Is there a way to add processor after destination is added, in the same pipeline? I want to run shell scripts after the data is loaded in HDFS (destination).

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-02-09 15:19:12 -0600

jeff gravatar image

For certain destinations, this is possible. Specifically for the HDFS destination, you can enable events on the destination, and an event will be generated when each file is closed. You can then attach an executor (including shell executor, for your case) to that event lane. See the specific page for HDFS destination for more information on that.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-02-08 00:40:19 -0600

Seen: 14 times

Last updated: Feb 09