Could anyone share how is the approach to do load testing?

asked 2019-11-18 07:08:39 -0500

Meihong gravatar image

There are some open source testing tools for performance testing. I'm trying to use Jmeter which is quite popular. Any good practice on this?

edit retag flag offensive close merge delete


What kind of load, specifically, are you interested in testing? Mentioning JMeter makes me think you're talking about the SDC web server component itself. But pipeline runners are completely orthogonal to that.

jeff gravatar imagejeff ( 2019-11-18 13:02:07 -0500 )edit

Ya true... Actually I want to test how much data flow Streamsets pipeline can handle and how fast it is. While I'm using JDBC Query Consumer to ingest data, it's in sequential incremental ingesting order, I don't really see the need for data load testing. What is your idea about it?

Meihong gravatar imageMeihong ( 2019-11-19 00:02:36 -0500 )edit

Your best bet is to simply set up your test data and pipelines, and run them. Check the pipeline metrics as it runs (ex: stage level, overall batch count, times, etc.). If you need to automate it, then the same data the UI shows is available through the REST API.

jeff gravatar imagejeff ( 2019-11-19 10:47:14 -0500 )edit