Slow DNS lookup!

asked 2018-09-19 11:42:46 -0500

supahcraig gravatar image

I occasionally see this warning in /var/log/sdc.log and whenever I do see it, it seems to halt logging for several hours although the pipeline itself still shows progress in the UI.

2018-09-18 14:57:13,122 pipeline kafka to kudu/pipelinekafkatokudu0df21797-cc3f-4b15-b698-b76dec022881 WARN Slow DNS lookup! Resolved IP of 'cdhdatanode3.mydomain.com' to cdhdatanode3.mydomain.com/10.4.16.13 in 3188834ns    NetUtil *admin  0   New I/O worker #63
2018-09-18 17:07:34,691 pipeline kafka to kudu/pipelinekafkatokudu0df21797-cc3f-4b15-b698-b76dec022881  WARN    Slow DNS lookup! Resolved IP of 'cdhdatanode5.mydomain.com' to cdhdatanode5.mydomain.com/10.4.16.15 in 3966369ns    NetUtil *admin  0   New I/O worker #43
2018-09-19 10:35:08,722 pipeline kafka to kudu/pipelinekafkatokudu0df21797-cc3f-4b15-b698-b76dec022881  INFO    Marking the coordinator 10.4.3.62:9092 (id: 2147483646 rack: null) dead for group streamsetsDataCollector   AbstractCoordinator *admin      ProductionPipelineRunnable-pipelinekafkatokudu0df21797-cc3f-4b15-b698-b76dec022881-pipeline kafka to kudu

For one, 31888834ns is only about 3msec which doesn't seem all that slow. I had 3 reasonably chatty pipelines running during this window, but only one was doing any logging (at least as far as I know where to look). Once the first slow DNS lookup was reported, the next log entry was a little over 2 hrs later, and the next log entry after that was over 17 hours later. Again, the UI shows records being moved through all 3 pipelines I had running, and I can see records showing up in my destinations, but not having any logging for extended windows is quite troubling.

I realize streamsets is probably not responsible for the slow DNS lookup, but why does that slow lookup appear to have such a negative impact on the logging?

edit retag flag offensive close merge delete