Kafka consumer creation/interruption during pipeline validation

asked 2018-08-02 07:16:15 -0500

Crash gravatar image

Prerequisite conditions:

SDC 3.1.0.0 used

Scenario:

1) Start SDC

2) Import pipeline with 3 stages: Kafka multi source origin -> Groovy Evaluator -> ToError

3) Perform any update of Groove Script in Groovy Evaluator.

Observable:

In logs I can see exception related to KafkaMultiSource thread being interrupted. Also I can see that Kafka related threads are being created and deleted shortly. But some threads (kafka-coordinator-heartbeat-thread-* in my case) stay alive which is wierd since I didn't even start my pipeline.

For me it looks like after Groovy script change SDC performs pipeline validations which leads to Kafka consumer creation and deleteion. Since it happens pretty fast some threads do not stop properly. Could you provide some input on such behavior?

LOG:

2018-08-02 11:20:44,941 [thread:preview-pool-1-thread-1] INFO Pipeline - Processing lifecycle start event with stage

2018-08-02 11:20:45,443 [thread:preview-pool-1-thread-1] WARN ConsumerConfig - The configuration 'schema.registry.url' was supplied but isn't a known config.

2018-08-02 11:20:45,459 [thread:preview-pool-1-thread-1] WARN ConsumerConfig - The configuration 'schema.registry.url' was supplied but isn't a known config.

2018-08-02 11:20:54,837 [thread:preview-pool-1-thread-1] INFO MultiKafkaSource - Total messages consumed by all threads: 0

2018-08-02 11:20:54,838 [thread:preview-pool-1-thread-2] INFO Pipeline - Destroying pipeline with reason=FAILURE

2018-08-02 11:20:54,839 [thread:preview-pool-1-thread-1] INFO Pipeline - Destroying pipeline with reason=FINISHED

2018-08-02 11:20:54,838 [thread:kafkaConsumerThread-0] ERROR MultiKafkaSource - Encountered error in multi kafka thread 0 during read org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException

2018-08-02 11:20:54,841 [thread:preview-pool-1-thread-1] INFO Pipeline - Processing lifecycle stop event

2018-08-02 11:20:54,842 [thread:preview-pool-1-thread-2] INFO Pipeline - Processing lifecycle stop event

2018-08-02 11:20:54,842 [thread:kafkaConsumerThread-1] ERROR MultiKafkaSource - Encountered error in multi kafka thread 1 during read org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException

2018-08-02 11:20:54,842 [thread:preview-pool-1-thread-2] INFO Pipeline - Pipeline finished destroying with final reason=FAILURE

2018-08-02 11:20:54,843 [thread:preview-pool-1-thread-1] INFO Pipeline - Pipeline finished destroying with final reason=FINISHED

2018-08-02 11:20:55,316 [thread:webserver-16] WARN StandaloneAndClusterPipelineManager - Evicting idle previewer 'bbb' in status 'FINISHED'

edit retag flag offensive close merge delete