Ask Your Question

How to convert Http Client Json data into a csv file

asked 2017-11-02 11:17:02 -0500

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

I am actually getting Json data from HTTP Client and sending it to Hadoop FS. I want to convert the Json data into a CSV file or some flat file or Pipe delimited file before sending it to hadoop. Is there a way to do this? Please some one help me with this issue

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2017-11-02 12:41:42 -0500

metadaddy gravatar image

I would recommend starting by working through the tutorial - this will help you understand how the pieces fit together - and it shows how to write data to Hadoop. The key principle is that origins (such as HTTP Client) parse the incoming data into in-memory records; processors act on these records, and destinations format the records and write to the data store. Your pipeline will probably be the HTTP Client origin set to JSON format and the Hadoop FS destination set to Delimited data format. As I mentioned before, you'll need to flatten any hierarchical structure in your incoming records. This blog post is a good guide to transforming data in the pipeline: Transform Data in StreamSets Data Collector.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-11-02 11:17:02 -0500

Seen: 365 times

Last updated: Nov 02 '17