Pipeline failed to get the messages out Event Hub after restart

asked 2019-10-24 15:32:42 -0600

CindyRu gravatar image

updated 2019-10-24 17:25:52 -0600

metadaddy gravatar image

We are running SDC 3.11.0 on an Azure Linux VM, V1 generation.

We were able to subscribe to an Azure Event Hub and then write the messages to a gen2 data lake file system. We noticed though that the pipeline would fail to get the messages out after the pipeline was stopped/restarted a few times.

Here is the sequence of events for a particular test run –

  1. Start pipeline 1 which was configured to publish data from a Dev Data Generator connector to an Azure Event Hub Producer. It was configured to publish 10 messages per minute.
  2. Start pipeline 2 which was configured to subscribe from the same event hub via an Azure IoT/Event Hub Consumer a connector and then write to a gen2 data lake via an Azure Data Lake Storage Gen2 connector. In the Record Count area, we would see the number for Input and Output change from 10 to 20 to 30, etc. Pipeline 2 was started at 18:54 for this run.
  3. Stop pipeline 2 at 18:55.
  4. Re-start pipeline 2 at 19:08. In the Record Count area, we would see the number for Input and Output remain 0 forever while pipeline 1 kept on publishing messages to the event hub.

What did we do wrong??

The log file entries for the test run described above:

2019-10-23 18:54:02,823 [user:cynthia] [pipeline:115622 Ingest from Event Hub to Data Lake/115622test313f5c70e-41fe-432d-9f6d-9c9b30d88aae] [runner:] [thread:webserver-22] [stage:] INFO  StandaloneRunner - Preparing to start pipeline '115622test313f5c70e-41fe-432d-9f6d-9c9b30d88aae::0
2019-10-23 18:54:02,826 [user:*cynthia] [pipeline:115622 Ingest from Event Hub to Data Lake/115622test313f5c70e-41fe-432d-9f6d-9c9b30d88aae] [runner:] [thread:runner-pool-2-thread-33] [stage:] INFO  StandaloneRunner - Starting pipeline 115622test313f5c70e-41fe-432d-9f6d-9c9b30d88aae 0
2019-10-23 18:54:02,875 [user:*cynthia] [pipeline:115622 Ingest from Event Hub to Data Lake/115622test313f5c70e-41fe-432d-9f6d-9c9b30d88aae] [runner:] [thread:runner-pool-2-thread-33] [stage:] INFO  ProductionPipelineRunner - Adding error listeners0
2019-10-23 18:54:02,921 [user:*cynthia] [pipeline:115622 Ingest from Event Hub to Data Lake/115622test313f5c70e-41fe-432d-9f6d-9c9b30d88aae] [runner:] [thread:ProductionPipelineRunnable-115622test313f5c70e-41fe-432d-9f6d-9c9b30d88aae-115622 Ingest from Event Hub to Data Lake] [stage:] INFO  Pipeline - Processing lifecycle start event with stage
2019-10-23 18:54:02,924 [user:*cynthia] [pipeline:115622 Ingest from Event Hub to Data Lake/115622test313f5c70e-41fe-432d-9f6d-9c9b30d88aae] [runner:] [thread:event-hub-consumer-pool-0] [stage:] INFO  MessagingFactory - messagingFactory[MessagingFactoryaa0046], hostName[xxxxx.servicebus.windows.net], info[starting reactor instance.]
2019-10-23 18:54:02,924 [user:*cynthia] [pipeline:115622 Ingest from Event Hub to Data Lake/115622test313f5c70e-41fe-432d-9f6d-9c9b30d88aae] [runner:] [thread:event-hub-consumer-pool-0] [stage:] INFO  ReactorHandler - reactor.onReactorInit
2019-10-23 18:54:02,924 [user:*cynthia] [pipeline:115622 Ingest from Event Hub to Data Lake/115622test313f5c70e-41fe-432d-9f6d-9c9b30d88aae] [runner:] [thread:event-hub-consumer-pool-0] [stage:] INFO  ConnectionHandler - onConnectionInit hostname[xxxxx.servicebus.windows.net]
2019-10-23 18:54:02,924 [user:*cynthia] [pipeline:115622 Ingest from Event Hub to Data Lake/115622test313f5c70e-41fe-432d-9f6d-9c9b30d88aae] [runner:] [thread:event-hub-consumer-pool-0] [stage:] INFO  ConnectionHandler - onConnectionLocalOpen: hostname[xxxxx.servicebus.windows.net:5671], errorCondition[null], errorDescription[null]
2019-10-23 18:54:02,925 [user:*cynthia] [pipeline:115622 Ingest from Event Hub to Data Lake/115622test313f5c70e-41fe-432d-9f6d-9c9b30d88aae] [runner:] [thread:event-hub-consumer-pool-0] [stage:] INFO  ConnectionHandler - onConnectionBound hostname[xxxxx.servicebus.windows.net]
2019-10-23 18:54:02,985 [user:*cynthia ...
(more)
edit retag flag offensive close merge delete

Comments

We're not aware of anything that might cause this. Could you file an issue at issues.streamsets.com and we can take a look. Thanks!

metadaddy gravatar imagemetadaddy ( 2019-10-24 19:00:16 -0600 )edit