Date value getting converted in motion

asked 2020-11-09 13:23:29 -0500

Manoj gravatar image

Pipeline reads a timestamp field from a S3 bucket and the value is '2020-01-01 20:00:00' and it's getting automatically converted into '2020-01-02 00:00:00' while in motion.

Before I apply any conversion or processor on the field value is changing. Let me know how to retain the original value from source.

I have the set the property spark.sql.session.timeZone=UTC

edit retag flag offensive close merge delete