Ask Your Question

Sequence Value Generation

asked 2020-03-23 05:14:49 -0500

chandra gravatar image

In Expression Evaluator create a new column with sequence value for each record.

Could you share you guidance.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2020-03-24 12:29:36 -0500

metadaddy gravatar image

You should do this in the target system, for example, you could use an AUTO_INCREMENT field in MySQL. It is tricky to do this reliably in StreamSets Data Collector, since you would have to discover the starting value when the pipeline starts, and any writes from other clients would break the sequence.

If you really need to do this in the pipeline, you're writing the data in a single pipeline run, and there are no other database clients, you could use a script evaluator. Here is an example in Groovy:

Init Script

# state persists between batches, but NOT across multiple pipeline runs
sdc.state['counter'] = 0


records = sdc.records
for (record in records) {
    try {
        record.value['id'] = sdc.state['counter']++
    } catch (e) {
        sdc.log.error(e.toString(), e)
        sdc.error.write(record, e.toString())
edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2020-03-23 05:14:49 -0500

Seen: 14 times

Last updated: Mar 24