Ask Your Question
1

Extract files from zipped folder in StreamSets

asked 2017-09-29 02:25:34 -0500

prachi gravatar image

I have an SFTP FTP Client as the origin of my pipeline. The SFTP server contains csv files inside zipped folders. What configuration (Data Format etc.) should I specify to fetch data from the csv files?

edit retag flag offensive close merge delete

Comments

Any solutions to this ? Facing a similar problem.

ruchir.nishkam gravatar imageruchir.nishkam ( 2017-10-02 10:26:50 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2017-10-02 11:18:55 -0500

metadaddy gravatar image

updated 2017-10-04 14:35:12 -0500

Delimited data format with the Compressed Archive compression format will handle .tar.gz files:

image description

Can you use .tar.gz rather than .zip?

edit flag offensive delete link more

Comments

I tried this configuration, but it's not working. I tried checking the Process Subdirectories checkbox in SFTP/FTP tab shown above but in vain. Do I need to make any other changes apart from the ones shown in the figure?

prachi gravatar imageprachi ( 2017-10-03 02:47:33 -0500 )edit

Getting the following error: HTTP_00 - Cannot parse record: java.io.IOException: org.apache.commons.compress.compressors.CompressorException: No Compressor found for the stream signature. Am I missing a config change ?

ruchir.nishkam gravatar imageruchir.nishkam ( 2017-10-03 11:43:17 -0500 )edit

Try the different compression formats - it may be Compressed File or Archive.

metadaddy gravatar imagemetadaddy ( 2017-10-03 12:06:23 -0500 )edit

Tried the different combinations, and the actual name within the 'File Name Pattern'. Same error persists.

ruchir.nishkam gravatar imageruchir.nishkam ( 2017-10-03 12:22:55 -0500 )edit

Can you create a similarly formatted file with dummy data and share it? I can give it a try and see what the problem is.

metadaddy gravatar imagemetadaddy ( 2017-10-03 12:25:11 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-09-29 02:25:34 -0500

Seen: 229 times

Last updated: Oct 04 '17