Ask Your Question

Encrypting data with HashiCorp Vault

asked 2017-11-30 04:09:44 -0500

HARSH TG gravatar image

updated 2017-11-30 11:40:05 -0500

metadaddy gravatar image

I have configured Vault with StreamSets successfully. Now I need to call Vault API before the destination in the pipeline. The actual task is to encrypt personal details using the Vault API before it reaches the destination.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2017-11-30 17:40:31 -0500

metadaddy gravatar image

This is not built-in to StreamSets Data Collector, but you can probably engineer something using the HTTP Client processor. You would need to obtain a Vault token, then use the HTTP API to encrypt the data. One suggestion is to store a permanent token as a generic secret, retrieve that with the existing Vault EL functions, and use it in the HTTP Client processor. This is not going to be super fast since you'd be making a lot of API calls to encrypt lots of data.

Note that, for security reasons, there is no way to extract SDC's token for use in your stage configs such as HTTP Client processor.

Feel free to file an issue to request this as an enhancement, or even have a go at implementing this - we welcome pull requests!

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-11-30 04:09:44 -0500

Seen: 400 times

Last updated: Nov 30 '17

Related questions