How to create a unique primary key values for each record.

asked 2020-05-28 10:14:36 -0600

Anonymous

updated 2020-05-28 10:16:10 -0600

Hi All, i have job reading product_id's and call api then each product will have N of reviews. currently we don't have unique column (So i need a primary key in my table) my end destination is hadoop/hive table. can you please suggest me a best option to generate unique key values in table for new column (what processor i can use to generate a row number for each record).

Thank You.

answered 2020-05-28 15:48:56 -0600

iamontheinet

2052 ●4 ●11 ●33

Hi!

In StreamSets Data Collector, you can use uuid:uuid() function in Expression Evaluator to generate a UUID and add a new "ID" field for each record.

For more details on it, refer to the documentation.

Cheers, Dash

edit flag offensive delete link

Comments

Don't use uuid:uuid(), it will slow down your pipeline after sometime. Try to think about something else. https://ask.streamsets.com/question/2440/why-is-uuid-generation-so-slow/

ahmed_alashrafy ( 2020-05-29 07:05:30 -0600 )edit

Thanks FYI.Can you please let me know the best solution guys. I need a unique row for each record and to just let you know that we have very very larger data set.

strem_dev ( 2020-05-29 14:20:07 -0600 )edit

@ahmed_alashrafy What you are referring to is clearly documented in the link I posted. It also highlights this "As a best practice, we recommend running the haveged daemon on any Data Collector node where the uuid:uuid() function is used."

iamontheinet ( 2020-05-29 14:27:28 -0600 )edit

@strem_dev Out of the box, that's what's available in SDC. The other option you can try for yourself is to use one of the scripting processors and use an external library that you know will work best for your use case.

iamontheinet ( 2020-05-29 14:29:47 -0600 )edit

add a comment

How to create a unique primary key values for each record.

1 Answer

Comments

Question Tools

Stats

Related questions

How to create a unique primary key values for each record. edit

1 Answer

Comments

Question Tools

Stats

Related questions

How to create a unique primary key values for each record.