How does Snowflake adapter process requests into Snowflake?

asked 2020-11-27 03:42:06 -0600

EoinCollins gravatar image

We are doing CDC from Oracle into Snowflake. For the most part it is working, but we've hit upon a quirky issue. Our Oracle table doesnt have a primary key (I know, dont go there!) so we have got the SMEs to identify a column that is unique and has no nulls that we can use. This is working fine, except when the source has incompatible data i.e. user enters duplicate rows in the source, breaking the uniqueness of our key. What we are seeing is that when the user deletes the 2 rows, 2 delete statements are issued to Snowflake in what I will call a single transaction. The StreamSets Snowflake adapter appears to evaluate the 2 statements to be issued and reports an error. It does not processes either of them. I would have expected it to do the first delete, then fail on the 2nd. It doesnt. It does neither delete and a record is left in Snowflake.

Any ideas?

edit retag flag offensive close merge delete