Ask Your Question
0

How to create a unique primary key values for each record.

asked 2020-05-28 10:14:36 -0500

anonymous user

Anonymous

updated 2020-05-28 10:16:10 -0500

Hi All, i have job reading product_id's and call api then each product will have N of reviews. currently we don't have unique column (So i need a primary key in my table) my end destination is hadoop/hive table. can you please suggest me a best option to generate unique key values in table for new column (what processor i can use to generate a row number for each record).

Thank You.

edit retag flag offensive close merge delete

1 Answer

Sort by » oldest newest most voted
1

answered 2020-05-28 15:48:56 -0500

iamontheinet gravatar image

Hi!

In StreamSets Data Collector, you can use uuid:uuid() function in Expression Evaluator to generate a UUID and add a new "ID" field for each record.

For more details on it, refer to the documentation.

Cheers, Dash

edit flag offensive delete link more

Comments

Don't use uuid:uuid(), it will slow down your pipeline after sometime. Try to think about something else. https://ask.streamsets.com/question/2440/why-is-uuid-generation-so-slow/

ahmed_alashrafy gravatar imageahmed_alashrafy ( 2020-05-29 07:05:30 -0500 )edit

Thanks FYI.Can you please let me know the best solution guys. I need a unique row for each record and to just let you know that we have very very larger data set.

strem_dev gravatar imagestrem_dev ( 2020-05-29 14:20:07 -0500 )edit

@ahmed_alashrafy What you are referring to is clearly documented in the link I posted. It also highlights this "As a best practice, we recommend running the haveged daemon on any Data Collector node where the uuid:uuid() function is used."

iamontheinet gravatar imageiamontheinet ( 2020-05-29 14:27:28 -0500 )edit

@strem_dev Out of the box, that's what's available in SDC. The other option you can try for yourself is to use one of the scripting processors and use an external library that you know will work best for your use case.

iamontheinet gravatar imageiamontheinet ( 2020-05-29 14:29:47 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-05-28 10:14:36 -0500

Seen: 99 times

Last updated: May 29