Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

How to load data if hashkey not in a list

Hi, I am having a pipeline that uses the Field Hasher generating a Hash Key. While I am looking for a way to load the list of hash key in the destination into memory so that the pipeline can check and only load new coming data with unseen hashkey to the destination. I tried with writing my own Jython evaluator to do so. The result is accurate while the performance is incredibly slow. What would be a better way to do that?

David

How to load data if hashkey not in a list

Hi, I am having a pipeline that uses the Field Hasher generating a Hash Key. While I am looking for a way to load the list of hash key in the destination into memory so that the pipeline can check and only load new coming data with unseen hashkey to the destination. I tried with writing my own Jython evaluator to do so. The result is accurate while the performance is incredibly slow. What would be a better way to do that?

David