Ask Your Question

How to add/inject a new field into a pipeline

asked 2018-07-10 13:32:01 -0500

supahcraig gravatar image

I've got a JDBC query consumer executing some SQL and pushing to Kafka. I need to add or inject an additional field (a literal) into the pipeline. It's easy with the query consumer; I just add a literal into the SELECT statement. But with a multi-table consumer I can't define SQL (which is really frustrating, TBH). I can easily imagine needing to do this with other origins as well.

So, what are some ways/best practices to do this sort of thing within streamsets?

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted

answered 2018-07-10 13:39:39 -0500

jeff gravatar image

The easiest way to do this is to attach an Expression Evaluator processor after your origin. You can add any output field you like, and the value can be an expression (including a constant one).

edit flag offensive delete link more


Thanks so much. I'm not 100% sure I understand all that I can do in there (or what each section actually does), but I added a field expression (not a field *attribute* expression) and got exactly what I wanted. I would upvote but I'm a n00b so I can't yet.

supahcraig gravatar imagesupahcraig ( 2018-07-10 14:17:21 -0500 )edit

answered 2020-06-02 00:47:08 -0500

sashish gravatar image

first you create an empty map/list using evaluation expression

output field = /newmap field expression = ${emptymap()}

now you can add this map/list in your path as below output field = /previousfield/newmap field expression = ${record:value('/previousfield')}

This would work. If you have nested field then you can use * for array/list. for this please read the streamsets wildcard expression.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools



Asked: 2018-07-10 13:32:01 -0500

Seen: 2,475 times

Last updated: Jun 02 '20