Ask Your Question

what value should i populate in the "DataLake Lookup" stage in the "return columns" under lookup property?

asked 2020-09-28 15:26:14 -0500

anonymous user


I am trying to build a SCD type 2 mapping where I am fetching data from oracle and storing on a "Datalake AWS S3" bucket.I am referring one of the demo from Streamsets on youtube In this demo under second scenario whatever is suggested I am following I am doing same steps but as "Datalake lookup"stage property had not been explained here and when I am using plane column name like "ID" in my "return columns" property it is giving below error while "ID" is one of the filed from my table and deltalake as well.

image description

Please explain using example.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2020-09-30 18:35:24 -0500

iamontheinet gravatar image

updated 2020-09-30 18:36:44 -0500


Check out my blog on SCD Design Patterns -- especially the third pattern. Basically you need to provide composite primary key that includes some sort of Id + the tracking field(s) that are present in your "lookup" table as shown below.

image description

Hope this helps.

Cheers, Dash

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2020-09-28 15:26:14 -0500

Seen: 233 times

Last updated: Sep 30