Streamsets Architecture

asked 2018-01-08 19:10:24 -0600

krishnaM gravatar image

I am not very clear about the architecture of streamsets How do we scale streamset in a distributed environment? Let's say, our input data velocity increases from origin(say database table) then how to ensure that SDC doesn't give performance issues? How many daemons will be running? Will it be Master worker architecture or peer to peer architecture?

If there are multiple daemons running on multiple machines (e.g. one sdc along with one NodeManager in YARN) then how it will show centralized view of data i.e. total record count etc.?

edit retag flag offensive close merge delete