Streamsets transformer "Pyspark" component is giving error while i am trying to read s3 file while same pyspark script is working from emr console

asked 2020-10-06 01:36:04 -0500

satender gravatar image

image description

Any immediate help is appreciable!

edit retag flag offensive close merge delete


How are you reading the objects from S3? Or, why are you using the PySpark processor and not the Amazon S3 origin?

iamontheinet gravatar imageiamontheinet ( 2020-10-06 01:42:15 -0500 )edit

We want to use the temporary S3 credentials which is not possible with the Amazon S3 stage.

satender gravatar imagesatender ( 2020-10-06 07:14:41 -0500 )edit

Well, 403 is being thrown from/by AWS so maybe this will help -- -- cheers!

iamontheinet gravatar imageiamontheinet ( 2020-10-08 01:13:05 -0500 )edit