Ask Your Question

How to add/inject a new field into a pipeline

asked 2018-07-10 13:32:01 -0500

supahcraig gravatar image

I've got a JDBC query consumer executing some SQL and pushing to Kafka. I need to add or inject an additional field (a literal) into the pipeline. It's easy with the query consumer; I just add a literal into the SELECT statement. But with a multi-table consumer I can't define SQL (which is really frustrating, TBH). I can easily imagine needing to do this with other origins as well.

So, what are some ways/best practices to do this sort of thing within streamsets?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-07-10 13:39:39 -0500

jeff gravatar image

The easiest way to do this is to attach an Expression Evaluator processor after your origin. You can add any output field you like, and the value can be an expression (including a constant one).

edit flag offensive delete link more


Thanks so much. I'm not 100% sure I understand all that I can do in there (or what each section actually does), but I added a field expression (not a field *attribute* expression) and got exactly what I wanted. I would upvote but I'm a n00b so I can't yet.

supahcraig gravatar imagesupahcraig ( 2018-07-10 14:17:21 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-07-10 13:32:01 -0500

Seen: 22 times

Last updated: Jul 10