Ask Your Question

How to Transpose a record in SDC?

asked 2020-02-13 08:29:36 -0600

abhishek_17 gravatar image


I am new to streamsets and trying to figure out how we can transpose a dataset in SDC. My source was json which I was able to flatten it in below format as LIST MAP after using Field order stage

ItemCode       Place   Gender
A                X         M
A                X         M
A                X         F
A                Y         F

and I need a desired output as :

ItemCode      Place   M   F
A              X      2   1
A              Y      0   1

Could you please let me know how we can approach this kind of problem in SDC?


edit retag flag offensive close merge delete


Is the input a single record, or 4? Similarly, is the output a single record or 2?

metadaddy gravatar imagemetadaddy ( 2020-02-13 10:00:09 -0600 )edit

The input is 4 different record and output will be 2 records. A typical case in relational world.

abhishek_17 gravatar imageabhishek_17 ( 2020-02-13 11:49:50 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2020-02-13 11:39:27 -0600

iamontheinet gravatar image


There's a Transpose processor in our Transformer product which will let you do exactly that. See below.

image description

image description

image description

Cheers, Dash

edit flag offensive delete link more


Thanks for the quick response, So as per my understanding I cannot achieve this in Streamsets Data Collector and need to install Transformer module for this? Also as far as I know Transformer is not Open Source Component of Streamsets just like Control Hub so I will need a license for this?

abhishek_17 gravatar imageabhishek_17 ( 2020-02-13 11:48:27 -0600 )edit

Correct, there isn't a "Transpose" processor in SDC so you'll most likely need to write custom code in one of the scripting processors. And, yes, Transformer is not open source and requires a license, but you can definitely try it out for free for 30days.

iamontheinet gravatar imageiamontheinet ( 2020-02-13 11:52:37 -0600 )edit

Thanks, Could you please point me in the right direction where I can find a sample for custom code to handle such scenario

abhishek_17 gravatar imageabhishek_17 ( 2020-02-17 07:36:46 -0600 )edit

There's isn't one. Basically you know the input and output so just code up the logic in Jython, Groovy or JavaScript :)

iamontheinet gravatar imageiamontheinet ( 2020-02-17 18:03:41 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2020-02-13 08:29:36 -0600

Seen: 28 times

Last updated: Feb 13