Ask Your Question

Testing Custom stages/transformers

asked 2017-10-19 13:40:49 -0500

amandra gravatar image

StreamSets really needs a test library that at the very least exposes the ParserFactorys and ideally a means to emulate a spark job (or any other framework aspects). I spent days building complicated transformers/processors and dealing with the endless deploy/restart/pray-for-a-log cycle. I finally pulled the source and stubbed out a simple test library and was done in hours. The project seems to treat customization as a complete oversight.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2017-10-20 12:49:34 -0500

jeff gravatar image

Sorry for the difficulty. It's true that there are a lot of areas that need improvement to make custom development easier. We are working on some of these already. You should pay attention to the SDC-7362 epic, which will eventually make the data format parsing functionality available to custom stages in a straightforward manner.

Regarding your debugging case, are you talking about a custom SDC stage (i.e. your own processor)? Or a custom Spark job in the context of the existing SDC Spark Evaluator stage?

edit flag offensive delete link more


Both. I've designed a custom processor (leveraging a jni parser) and now a `SparkTransformer`. Both were fairly brutal processes. It's just way to hard to create a record and test locally in anything resembling the StreamSets life cycle.

amandra gravatar imageamandra ( 2017-10-20 12:54:56 -0500 )edit

After the dozenth time deploying to the server, restarting, re-running, checking logs for that path typo you would have detected in 5s debugging you start to question the value of the framework.

amandra gravatar imageamandra ( 2017-10-20 12:56:20 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-10-19 13:40:49 -0500

Seen: 215 times

Last updated: Oct 20 '17