Kafka consumer creation/interruption during pipeline validation

edit

Prerequisite conditions:

SDC 3.1.0.0 used

Scenario:

1) Start SDC

2) Import pipeline with 3 stages: Kafka multi source origin -> Groovy Evaluator -> ToError

3) Perform any update of Groove Script in Groovy Evaluator.

Observable:

In logs I can see exception related to KafkaMultiSource thread being interrupted. Also I can see that Kafka related threads are being created and deleted shortly. But some threads (kafka-coordinator-heartbeat-thread-* in my case) stay alive which is wierd since I didn't even start my pipeline.

For me it looks like after Groovy script change SDC performs pipeline validations which leads to Kafka consumer creation and deleteion. Since it happens pretty fast some threads do not stop properly. Could you provide some input on such behavior?

LOG:

2018-08-02 11:20:44,941 [thread:preview-pool-1-thread-1] INFO  Pipeline - Processing lifecycle start event with stage

2018-08-02 11:20:45,443 [thread:preview-pool-1-thread-1] WARN  ConsumerConfig - The configuration 'schema.registry.url' was supplied but isn't a known config.

2018-08-02 11:20:45,459 [thread:preview-pool-1-thread-1] WARN  ConsumerConfig - The configuration 'schema.registry.url' was supplied but isn't a known config.

2018-08-02 11:20:54,837 [thread:preview-pool-1-thread-1] INFO  MultiKafkaSource - Total messages consumed by all threads: 0

2018-08-02 11:20:54,838 [thread:preview-pool-1-thread-2] INFO  Pipeline - Destroying pipeline with reason=FAILURE

2018-08-02 11:20:54,839 [thread:preview-pool-1-thread-1] INFO  Pipeline - Destroying pipeline with reason=FINISHED

2018-08-02 11:20:54,838 [thread:kafkaConsumerThread-0] ERROR MultiKafkaSource - Encountered error in multi kafka thread 0 during read org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException

2018-08-02 11:20:54,841 [thread:preview-pool-1-thread-1] INFO  Pipeline - Processing lifecycle stop event

2018-08-02 11:20:54,842 [thread:preview-pool-1-thread-2] INFO  Pipeline - Processing lifecycle stop event

2018-08-02 11:20:54,842 [thread:kafkaConsumerThread-1] ERROR MultiKafkaSource - Encountered error in multi kafka thread 1 during read org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException

2018-08-02 11:20:54,842 [thread:preview-pool-1-thread-2] INFO  Pipeline - Pipeline finished destroying with final reason=FAILURE

2018-08-02 11:20:54,843 [thread:preview-pool-1-thread-1] INFO  Pipeline - Pipeline finished destroying with final reason=FINISHED

2018-08-02 11:20:55,316 [thread:webserver-16] WARN  StandaloneAndClusterPipelineManager - Evicting idle previewer 'bbb' in status 'FINISHED'

Stack trace example:

org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeThrowInterruptException(ConsumerNetworkClient.java:447)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:254)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:214)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:205)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitPendingRequests(ConsumerNetworkClient.java:289)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.close(AbstractCoordinator.java:706)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.close(ConsumerCoordinator.java:498)
    at org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1729)
    at org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1697)
    at org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1672)
    at com.streamsets.pipeline.stage.origin.multikafka.v0_10.loader.Kafka0_10ConsumerLoader$WrapperKafkaConsumer.close(Kafka0_10ConsumerLoader.java:61)
    at com ...