Kafka consumer creation/interruption during pipeline validation
Prerequisite conditions:
SDC 3.1.0.0 used
Scenario:
1) Start SDC
2) Import pipeline with 3 stages: Kafka multi source origin -> Groovy Evaluator -> ToError
3) Perform any update of Groove Script in Groovy Evaluator.
Observable:
In logs I can see exception related to KafkaMultiSource thread being interrupted. Also I can see that Kafka related threads are being created and deleted shortly. But some threads (kafka-coordinator-heartbeat-thread-* in my case) stay alive which is wierd since I didn't even start my pipeline.
For me it looks like after Groovy script change SDC performs pipeline validations which leads to Kafka consumer creation and deleteion. Since it happens pretty fast some threads do not stop properly. Could you provide some input on such behavior?
LOG:
2018-08-02 11:20:44,941 [thread:preview-pool-1-thread-1] INFO Pipeline - Processing lifecycle start event with stage
2018-08-02 11:20:45,443 [thread:preview-pool-1-thread-1] WARN ConsumerConfig - The configuration 'schema.registry.url' was supplied but isn't a known config.
2018-08-02 11:20:45,459 [thread:preview-pool-1-thread-1] WARN ConsumerConfig - The configuration 'schema.registry.url' was supplied but isn't a known config.
2018-08-02 11:20:54,837 [thread:preview-pool-1-thread-1] INFO MultiKafkaSource - Total messages consumed by all threads: 0
2018-08-02 11:20:54,838 [thread:preview-pool-1-thread-2] INFO Pipeline - Destroying pipeline with reason=FAILURE
2018-08-02 11:20:54,839 [thread:preview-pool-1-thread-1] INFO Pipeline - Destroying pipeline with reason=FINISHED
2018-08-02 11:20:54,838 [thread:kafkaConsumerThread-0] ERROR MultiKafkaSource - Encountered error in multi kafka thread 0 during read org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException
2018-08-02 11:20:54,841 [thread:preview-pool-1-thread-1] INFO Pipeline - Processing lifecycle stop event
2018-08-02 11:20:54,842 [thread:preview-pool-1-thread-2] INFO Pipeline - Processing lifecycle stop event
2018-08-02 11:20:54,842 [thread:kafkaConsumerThread-1] ERROR MultiKafkaSource - Encountered error in multi kafka thread 1 during read org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException
2018-08-02 11:20:54,842 [thread:preview-pool-1-thread-2] INFO Pipeline - Pipeline finished destroying with final reason=FAILURE
2018-08-02 11:20:54,843 [thread:preview-pool-1-thread-1] INFO Pipeline - Pipeline finished destroying with final reason=FINISHED
2018-08-02 11:20:55,316 [thread:webserver-16] WARN StandaloneAndClusterPipelineManager - Evicting idle previewer 'bbb' in status 'FINISHED'
Stack trace example:
org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeThrowInterruptException(ConsumerNetworkClient.java:447)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:254)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:214)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:205)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitPendingRequests(ConsumerNetworkClient.java:289)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.close(AbstractCoordinator.java:706)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.close(ConsumerCoordinator.java:498)
at org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1729)
at org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1697)
at org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1672)
at com.streamsets.pipeline.stage.origin.multikafka.v0_10.loader.Kafka0_10ConsumerLoader$WrapperKafkaConsumer.close(Kafka0_10ConsumerLoader.java:61)
at com ...
What effect do you see after this? Does the pipeline still run?
I observe only plenty of errors in logs and alive Kafka consumer threads even if there are no running pipelines. From execution perspective everything works as usually. Maybe one thing is that sometimes when I do start/stop/start pipeline it takes much time before message consumption starts again.