Transfer file from SFTP to S3 and call Databricks Spark job

asked 2020-01-06 08:44:17 -0600

Mohit Damani gravatar image


I need to get files from SFTP to S3 and then run my Databrick job to do further processing. Challenge here is the file name, source path and target path all will be coming from RDS system.

Also once my databricks job is completed i need to create a manifest file with the details i fetched from RDS and also add few more details

edit retag flag offensive close merge delete