Unzip Files on Local Machine

asked 2019-08-23 14:53:48 -0500

Schiiss gravatar image

We have a use case where files are SFTP to our server as .tar.gz and .gz. We need to be able to unzip the files before they land in our Azure Datalake so its makes them easy to consume. We are currently using a bash script to do this but it is unstable. What does everyone recommend to unzip files?

edit retag flag offensive close merge delete


Is what sense is it unstable? Are you triggering the script from an event in the pipeline?

metadaddy gravatar imagemetadaddy ( 2019-08-23 17:37:59 -0500 )edit

The script will often fail with " SHELL_002 - Script execution timed out" and will not unzip the files. So we essentially have two pipelines doing this work. 1. Listen for .gz and .tar.gz files using the "Directory" connector and unzip 2. Move files without the .gz or .tar.gz into the data lake.

Schiiss gravatar imageSchiiss ( 2019-08-27 15:23:40 -0500 )edit