Ask Your Question
0

HTTP Client + Evaluator/Script

asked 2017-11-03 05:07:59 -0500

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

Hi. I'm trying to download all the files listed in http://files.data.gouv.fr/sirene/ but my pipeline doesn't work because the first file is a text file that I don't want to keep. I just need all the csv within a zip.

I first asked my questions on stackoverflow : https://stackoverflow.com/questions/4...

From what I understand, I need to write a script or a jython evaluator to make it work but I have no idea how I should handle some variable such as "records".

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2019-10-20 04:04:07 -0500

ahmed_alashrafy gravatar image

you don't have to develop jython code to do that. the following steps achieve your goal 1- use http_client as origin to read the root page 2-use stream Selector to filter ZIP files using this expression ${str:contains(record:value('/text'),".zip")} 3-use http_client processor to read the file using data format to be delimited and the compression as Archieve, also use custom delimiter as semicolon

then store the file in the destination that you need

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-11-03 05:07:59 -0500

Seen: 369 times

Last updated: Oct 20 '19