Ask Your Question
1

Kafka consumer creation/interruption during pipeline validation

asked 2018-08-02 07:16:15 -0600

Crash gravatar image

updated 2018-10-01 11:29:48 -0600

metadaddy gravatar image

Prerequisite conditions:

SDC 3.1.0.0 used

Scenario:

1) Start SDC

2) Import pipeline with 3 stages: Kafka multi source origin -> Groovy Evaluator -> ToError

3) Perform any update of Groove Script in Groovy Evaluator.

Observable:

In logs I can see exception related to KafkaMultiSource thread being interrupted. Also I can see that Kafka related threads are being created and deleted shortly. But some threads (kafka-coordinator-heartbeat-thread-* in my case) stay alive which is wierd since I didn't even start my pipeline.

For me it looks like after Groovy script change SDC performs pipeline validations which leads to Kafka consumer creation and deleteion. Since it happens pretty fast some threads do not stop properly. Could you provide some input on such behavior?

LOG:

2018-08-02 11:20:44,941 [thread:preview-pool-1-thread-1] INFO  Pipeline - Processing lifecycle start event with stage

2018-08-02 11:20:45,443 [thread:preview-pool-1-thread-1] WARN  ConsumerConfig - The configuration 'schema.registry.url' was supplied but isn't a known config.

2018-08-02 11:20:45,459 [thread:preview-pool-1-thread-1] WARN  ConsumerConfig - The configuration 'schema.registry.url' was supplied but isn't a known config.

2018-08-02 11:20:54,837 [thread:preview-pool-1-thread-1] INFO  MultiKafkaSource - Total messages consumed by all threads: 0

2018-08-02 11:20:54,838 [thread:preview-pool-1-thread-2] INFO  Pipeline - Destroying pipeline with reason=FAILURE

2018-08-02 11:20:54,839 [thread:preview-pool-1-thread-1] INFO  Pipeline - Destroying pipeline with reason=FINISHED

2018-08-02 11:20:54,838 [thread:kafkaConsumerThread-0] ERROR MultiKafkaSource - Encountered error in multi kafka thread 0 during read org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException

2018-08-02 11:20:54,841 [thread:preview-pool-1-thread-1] INFO  Pipeline - Processing lifecycle stop event

2018-08-02 11:20:54,842 [thread:preview-pool-1-thread-2] INFO  Pipeline - Processing lifecycle stop event

2018-08-02 11:20:54,842 [thread:kafkaConsumerThread-1] ERROR MultiKafkaSource - Encountered error in multi kafka thread 1 during read org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException

2018-08-02 11:20:54,842 [thread:preview-pool-1-thread-2] INFO  Pipeline - Pipeline finished destroying with final reason=FAILURE

2018-08-02 11:20:54,843 [thread:preview-pool-1-thread-1] INFO  Pipeline - Pipeline finished destroying with final reason=FINISHED

2018-08-02 11:20:55,316 [thread:webserver-16] WARN  StandaloneAndClusterPipelineManager - Evicting idle previewer 'bbb' in status 'FINISHED'

Stack trace example:

org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeThrowInterruptException(ConsumerNetworkClient.java:447)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:254)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:214)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:205)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitPendingRequests(ConsumerNetworkClient.java:289)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.close(AbstractCoordinator.java:706)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.close(ConsumerCoordinator.java:498)
    at org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1729)
    at org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1697)
    at org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1672)
    at com.streamsets.pipeline.stage.origin.multikafka.v0_10.loader.Kafka0_10ConsumerLoader$WrapperKafkaConsumer.close(Kafka0_10ConsumerLoader.java:61)
    at com ...
(more)
edit retag flag offensive close merge delete

Comments

What effect do you see after this? Does the pipeline still run?

metadaddy gravatar imagemetadaddy ( 2018-10-01 13:40:46 -0600 )edit

I observe only plenty of errors in logs and alive Kafka consumer threads even if there are no running pipelines. From execution perspective everything works as usually. Maybe one thing is that sometimes when I do start/stop/start pipeline it takes much time before message consumption starts again.

seolse gravatar imageseolse ( 2018-10-02 10:42:39 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-10-02 12:10:57 -0600

metadaddy gravatar image

Not sure why it's doing that when you edit the Groovy script. Please file an issue at http://issues.streamsets.com with all of the above details.

edit flag offensive delete link more

Comments

Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-08-02 07:12:39 -0600

Seen: 104 times

Last updated: Oct 02