How to load data if hashkey not in a list

asked 2018-05-16 02:56:52 -0500

davidha gravatar image

updated 2018-05-16 07:33:47 -0500

Maithri gravatar image

Hi, I am having a pipeline that uses the Field Hasher generating a Hash Key. While I am looking for a way to load the list of hash key in the destination into memory so that the pipeline can check and only load new coming data with unseen hashkey to the destination. I tried with writing my own Jython evaluator to do so. The result is accurate while the performance is incredibly slow. What would be a better way to do that?

David

edit retag flag offensive close merge delete