Ask Your Question
1

How to Transpose a record in SDC?

asked 2020-02-13 08:29:36 -0600

abhishek_17 gravatar image

Hi,

I am new to streamsets and trying to figure out how we can transpose a dataset in SDC. My source was json which I was able to flatten it in below format as LIST MAP after using Field order stage

ItemCode       Place   Gender
A                X         M
A                X         M
A                X         F
A                Y         F

and I need a desired output as :

ItemCode      Place   M   F
A              X      2   1
A              Y      0   1

Could you please let me know how we can approach this kind of problem in SDC?

Thanks

edit retag flag offensive close merge delete

Comments

Is the input a single record, or 4? Similarly, is the output a single record or 2?

metadaddy gravatar imagemetadaddy ( 2020-02-13 10:00:09 -0600 )edit

The input is 4 different record and output will be 2 records. A typical case in relational world.

abhishek_17 gravatar imageabhishek_17 ( 2020-02-13 11:49:50 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted
1

answered 2020-02-13 11:39:27 -0600

iamontheinet gravatar image

Hi!

There's a Transpose processor in our Transformer product which will let you do exactly that. See below.

image description


image description


image description

Cheers, Dash

edit flag offensive delete link more

Comments

Thanks for the quick response, So as per my understanding I cannot achieve this in Streamsets Data Collector and need to install Transformer module for this? Also as far as I know Transformer is not Open Source Component of Streamsets just like Control Hub so I will need a license for this?

abhishek_17 gravatar imageabhishek_17 ( 2020-02-13 11:48:27 -0600 )edit
1

Correct, there isn't a "Transpose" processor in SDC so you'll most likely need to write custom code in one of the scripting processors. And, yes, Transformer is not open source and requires a license, but you can definitely try it out for free for 30days.

iamontheinet gravatar imageiamontheinet ( 2020-02-13 11:52:37 -0600 )edit

Thanks, Could you please point me in the right direction where I can find a sample for custom code to handle such scenario

abhishek_17 gravatar imageabhishek_17 ( 2020-02-17 07:36:46 -0600 )edit

There's isn't one. Basically you know the input and output so just code up the logic in Jython, Groovy or JavaScript :)

iamontheinet gravatar imageiamontheinet ( 2020-02-17 18:03:41 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-02-13 08:29:36 -0600

Seen: 28 times

Last updated: Feb 13