Kafka Consumer in Cluster Mode increase processing time

asked 2018-02-17 14:59:48 -0600

germaneduardo gravatar image

updated 2018-02-20 20:06:51 -0600

I am seeing a particular behavior when we use the Kafka consumer in cluster mode.

When the topic read by the consumer have 2 partition we have 215 ms on average processing time (running time 10 minutes)

Avg Processing time with 2 part

but when we start to test several number of partitions the processing time begin to show a strange behavior:

3 partitions: avg time: 217 ms Avg Processing time with 3 part

4 partitions: avg time: 274 ms Avg Processing time with 5 part

5 partitions: avg time: 4221 ms: Avg Processing time with 5 part

6 partitions: avg time: 413 ms: Avg Processing time with 6 part

7 partitions: avg time: 4209ms ms: Avg Processing time with 7 part

10 partitions: avg time: 390 ms: Avg Processing time with 10 part

the pipeline tested have other stages: pipeline

and the configuration of kafka consumer: kafka

We are testing with a small input rate (1 records/s). Are we missing something on the configuration? why with 5 and 7 partitions on the topic the processing time is around 4 s if is the same configuration?

thanks in advance

EDIT 1: Also I am seeing idle time on the tasks: kafka

edit retag flag offensive close merge delete