Ask Your Question
0

Streamsets help for development

asked 2020-04-13 01:55:34 -0500

Vijay Kumar gravatar image

I'm new to StreamSets and have requirement to develop for below json file to load on RDBMS.

I have tried with multiple approaches didn't work, need solution in Streamsets pipeline only. Kindly need your suggestions. Below is the source data and target data should load in that format.

for a better pls copy and paste in json viewer, sample output pasted in below "output". Kindly help meto achieve this.

source data:

{ "AdditionalInfo": { "ToolName": "Midrange Sizer", "FileID": "767126" }, "Structures": { "Qty": 1, "Name": "Test", "Bases": [ { "Id": "GMG2QZX", "Qty": 1, "Modules": [ { "Id": 1570, "Options": [ { "Id": "G8GFK25", "Qty": 16 }, { "Id": "G3N5XQS", "Qty": 5 } ] }, { "Id": 1531, "Options": [ { "Id": "GJ0H6AG", "Qty": 1 } ] }, { "Id": 1623, "Options": [ { "Id": "GMNAU37", "Qty": 1 } ] } ] }, { "Id": "GD2SYTB", "Qty": 1, "Modules": [ { "Id": 1570, "Options": [ { "Id": "G762ALR", "Qty": 15 } ] } ] }, { "Id": "GI90F1E", "Qty": 1, "Modules": [ { "Id": 1570, "Options": [ { "Id": "G8GFK25", "Qty": 3 }, { "Id": "G3N5XQS", "Qty": 6 } ] } ] }, { "Id": "GD2SYTB", "Qty": 1, "Modules": [ { "Id": 1570, "Options": [ { "Id": "G762ALR", "Qty": 2 } ] } ] } ] } }

output:

AdditionalInfo_FileID,AdditionalInfo_ToolName,Structures_index_Name,Structures_index_Qty,Structures_index_Bases_index_Id,Structures_index_Bases_index_Qty,Structures_index_Bases_index_Modules_index_Id,Structures_index_Bases_index_Modules_index_Options_index_Id,Structures_index_Bases_index_Modules_index_Options_index_Qty 767126,Midrange Sizer,Test,1,GMG2QZX,1,1570,G8GFK25,16 767126,Midrange Sizer,Test,1,GD2SYTB,1,1570,G762ALR,15 767126,Midrange Sizer,Test,1,GI90F1E,1,1570,G8GFK25,3 767126,Midrange Sizer,Test,1,GD2SYTB,1,1570,G762ALR,2 767126,Midrange Sizer,Test,1,GMG2QZX,1,1531,GJ0H6AG,1 767126,Midrange Sizer,Test,1,GMG2QZX,1,1623,GMNAU37,1 767126,Midrange Sizer,Test,1,GMG2QZX,1,1570,G3N5XQS,5 767126,Midrange Sizer,Test,1,GI90F1E,1,1570,G3N5XQS,6

edit retag flag offensive close merge delete

Comments

1

Could you edit the question to have the pretty printed JSON? It's really hard to see what you're trying to do.

metadaddy gravatar imagemetadaddy ( 2020-04-16 15:59:37 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
1

answered 2020-04-13 12:14:18 -0500

gprasaad gravatar image
edit flag offensive delete link more

Comments

Hi, I tried with this approach not worked because here data is in multiple objects and should load with recurring method, if you see that output will get an idea. Not able to attach output here. This is the data in json file -> AdditionalInfo >> Structures >> Bases >>Modules >> Options

Vijay Kumar gravatar imageVijay Kumar ( 2020-04-13 23:26:15 -0500 )edit

i am also another user like you, so cnnot solve your problem completely, but i can tell you i was in the same boat, and i used Dev Raw Data (json) and using trial and error discovered that what worked for me was a combination of Pivoter, expression evaluator, Remover, flattener and finally Renamer.

gprasaad gravatar imagegprasaad ( 2020-04-16 10:00:04 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-04-13 01:48:11 -0500

Seen: 41 times

Last updated: Apr 13