Ask Your Question

Can one pipeline trigger another?

asked 2018-06-14 10:03:42 -0500

SUSER gravatar image

updated 2018-06-15 10:26:57 -0500

metadaddy gravatar image

Is there a way to trigger another pipeline from the current pipeline when the HDFS file is completely done and closed.

Current pipeline runs in cluster streaming mode.

edit retag flag offensive close merge delete


Cluster batch or cluster streaming?

metadaddy gravatar imagemetadaddy ( 2018-06-14 10:22:33 -0500 )edit

Yarn cluster streaming

SUSER gravatar imageSUSER ( 2018-06-15 10:23:33 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2018-06-15 10:34:44 -0500

metadaddy gravatar image

You can configure event generation for the Hadoop FS destination - there are lots of examples in the docs - then use the HTTP Client destination with the Data Collector REST API to start the second pipeline.

For details on the REST API, click 'Help' from the Data Collector UI, then 'RESTful API'

image description

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-06-14 10:03:42 -0500

Seen: 426 times

Last updated: Jun 15 '18