Delete records from Azure data lake

asked 2019-03-28 01:03:49 -0500

raghav.maurya gravatar image

Hi All,

I need a help here, if someone can answer.

I have created one pipeline and due to some infrastructure issues at source system i can't use CDC functionality so in every run i am deleting the records before insert. This way in each run, it will insert all the records from source to destination which makes be able to capture both modified and newly added records but now we are suppose to use Azure data lake instead of SQL server.

Since Azure data lake is a file based database so i am not sure how i can delete the exiting records before insert. I don't want to keep duplicate records in destination.

edit retag flag offensive close merge delete