Ask Your Question

How Does Security Work within StreamSets?

asked 2018-04-17 23:23:26 -0600

Monty gravatar image

I am trying to find the security best practices for StreamSets and how security should be setup. For instance, if my current origins(database servers, file folders, etc) are not setup to use SSL and I added a SSL Certificate to my StreamSets Server, would my origin and destination connections be encrypted while the data is being transported? Also, how is data encrypted when I am previewing within the GUI? Is data ever pushed to the StreamSets local disks?


edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-04-18 13:37:23 -0600

jeff gravatar image

If you configure the servers to use SSL, and then configure the clients (from the StreamSets perspective, i.e. origins) to use that secure channel (ex: by having https as the URL prefix for HTTP and setting up the trust store, using encrypted ports/options for JDBC URLs, etc.), then the communication will be encrypted as you would expect. You can verify by turning off the insecure ports for your servers and ensure everything is still running.

As data flows through Data Collector, it is not encrypted, but is only stored in memory. It is not written to disk unless you specifically configure a stage to do so. Beware that certain stages can end up logging various things that can turn into data (ex: if you enable request logging for HTTP client origin, the response payload could be logged depending on your configuration). For preview, data is sent to the browser in plaintext, so if you are concerned with snooping on that traffic then you need to ensure you have configured Data Collector to use HTTPS via the https.port property (see here).

edit flag offensive delete link more


jeff - what about when I am using Edge to push data to my pipeline? Is the data encrypted by default or does SSL have to be setup on my StreamSets server?

Monty gravatar imageMonty ( 2018-04-18 14:02:21 -0600 )edit

You're talking about Data Collector Edge? It's not encrypted by default, you will need to explicitly configure the connection to use encryption.

jeff gravatar imagejeff ( 2018-04-20 15:01:12 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-04-17 23:23:26 -0600

Seen: 680 times

Last updated: Apr 18 '18