START_PIPELINE_04 Error starting another pipeline based on an event

asked 2020-12-14 15:03:20 -0600

KateF gravatar image

updated 2020-12-22 17:16:55 -0600

I have a Data Collector pipeline that is essentially a JDBC multitable consumer origin and a JDBC destination - transferring all data from one destination to another.

I have the Multitable origin generating events, and a stream selector which is isolating the no-more-data event. I had the stream selector pointing straight at a pipeline finisher, and it was all working fine.

Now, I really want to log row counts. So in between the stream selector and the pipeline finisher I added a Start Pipelines processor to invoke an HTTP pipeline that calls the metrics API and logs the rows processed by the original pipeline before finishing it.

So it's like this: Origin Event > Stream Selector (no-more-data event) > Start Pipelines > Pipeline Finisher

So now, the Start Pipelines processor gets invoked, the second pipeline runs successfully (I can see the correct row counts in the mongo instance I'm using to log), but the first pipeline gets a stage error on the Start Pipelines processor and never gets to the Pipeline Finisher event. The stage error is:

START_PIPELINE_04 - Failed to start pipeline: java.lang.ClassCastException: java.util.HashMap cannot be cast to java.util.LinkedHashMap

I'm thinking this has to do with mixing event and orchestration processing but should it really error out like that? Is there some way around it?


edit retag flag offensive close merge delete


I tried to work around the error like this: Origin Event > Stream Selector 1 (schema-finished event) > Start Pipelines > Trash. Stream Selector 2 (no-more-data event with 30 seconds delay) > Pipeline Finisher. This seems like it should work but didn't.

KateF gravatar imageKateF ( 2020-12-22 16:51:09 -0600 )edit