Ask Your Question

How to create a unique primary key values for each record.

asked 2020-05-28 10:14:36 -0600

anonymous user


updated 2020-05-28 10:16:10 -0600

Hi All, i have job reading product_id's and call api then each product will have N of reviews. currently we don't have unique column (So i need a primary key in my table) my end destination is hadoop/hive table. can you please suggest me a best option to generate unique key values in table for new column (what processor i can use to generate a row number for each record).

Thank You.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2020-05-28 15:48:56 -0600

iamontheinet gravatar image


In StreamSets Data Collector, you can use uuid:uuid() function in Expression Evaluator to generate a UUID and add a new "ID" field for each record.

For more details on it, refer to the documentation.

Cheers, Dash

edit flag offensive delete link more


Don't use uuid:uuid(), it will slow down your pipeline after sometime. Try to think about something else.

ahmed_alashrafy gravatar imageahmed_alashrafy ( 2020-05-29 07:05:30 -0600 )edit

Thanks FYI.Can you please let me know the best solution guys. I need a unique row for each record and to just let you know that we have very very larger data set.

strem_dev gravatar imagestrem_dev ( 2020-05-29 14:20:07 -0600 )edit

@ahmed_alashrafy What you are referring to is clearly documented in the link I posted. It also highlights this "As a best practice, we recommend running the haveged daemon on any Data Collector node where the uuid:uuid() function is used."

iamontheinet gravatar imageiamontheinet ( 2020-05-29 14:27:28 -0600 )edit

@strem_dev Out of the box, that's what's available in SDC. The other option you can try for yourself is to use one of the scripting processors and use an external library that you know will work best for your use case.

iamontheinet gravatar imageiamontheinet ( 2020-05-29 14:29:47 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2020-05-28 10:14:36 -0600

Seen: 284 times

Last updated: May 29 '20