Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Delete records from Azure data lake

Hi All,

I need a help here, if someone can answer.

I have created one pipeline and due to some infrastructure issues at source system i can't use CDC functionality so in every run i am deleting the records before insert. This way in each run, it will insert all the records from source to destination which makes be able to capture both modified and newly added records but now we are suppose to use Azure data lake instead of SQL server.

Since Azure data lake is a file based database so i am not sure how i can delete the exiting records before insert. I don't want to keep duplicate records in destination.