Exponentially slow transfer speeds after update.

asked 2019-09-12 10:17:02 -0500

CaptinCCat gravatar image

updated 2019-09-12 14:09:46 -0500

Hi All, So in the past, we were running sdc 3.8.0 Recently we upgraded to sdc 3.10.1

Now we followed the instruction to perform an upgrade, made sure all our configs were saved backed up and re-implemented into the new version.

However, with the upgrade, we have been noticing much slower transfer speeds then what we had with 3.8. I'm not sure if it's with trying to write to an HDFS cluster, or if the pipeline reading from the DB.

If anyone could help me, or give me some advice on where to look/what to do that could possibly bring performance to where it was before that would be greatly appreciated. I can provide most of whatever would be needed to assist me.

Thank you ~Cole

EDIT: image description

This is the Pipeline (We have been running for 13 days, however, did restarts and whatnot before, we just need it running)

image description

I think this is what you needed also

edit retag flag offensive close merge delete


What kinds of pipelines are you dealing with? As in, what stages are in play? Can you capture the performance pie chart of a running one that is slower?

jeff gravatar imagejeff ( 2019-09-12 11:52:12 -0500 )edit

Not sure how to tag people or if you can, I posted the information you asked for. If you need more let me konw. I had to blackout a few things just for privacy reasons

CaptinCCat gravatar imageCaptinCCat ( 2019-09-12 14:06:59 -0500 )edit

Which stage is the large chunk of the pie graph, in blue (mouse over)?

jeff gravatar imagejeff ( 2019-09-12 14:10:34 -0500 )edit

FN ConsumptionReading, should the Origin at 4.33

CaptinCCat gravatar imageCaptinCCat ( 2019-09-12 14:12:41 -0500 )edit

Click on the origin stage as it's running to examine the metrics. Look at the `Table Metrics for Thread` gauges. Does anything stand out? What is the status associated with the longest time period?

jeff gravatar imagejeff ( 2019-09-12 14:17:55 -0500 )edit