Ask Your Question
2

Streamsets Transformer - JDBC Origin without offset column

asked 2019-12-17 11:43:05 -0600

andremachado gravatar image

updated 2019-12-19 12:26:45 -0600

metadaddy gravatar image

I'm testing platforms that can allow any user to easily create data processing pipelines. This platform has to meet certain requirements and one of them is to be capable of moving data from Oracle/SQL Server to HDFS.

Streamsets Transformer (v3.11) meets all requirements including the one referred above. I just can't get it to work in a very specific case: When ingesting a table that contains no numeric columns.

In these cases I want the pipeline to process all data so, in the JDBC Origin, I enabled the "Skip Offset Tracking" property. I thought that by skipping the offset tracking there would be no need to set the "Offset Column" property (guess I was wrong).

JDBC_05 - Table doesn't have compatible primary key configuration - supporting exactly one column but table have 0

If a numeric column exists, a possible workaround is to set it as the offset column but I can't find a way of doing this when none exists.

Am I missing something?

Thanks

( original question here: https://stackoverflow.com/questions/5... )

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2019-12-17 16:23:14 -0600

metadaddy gravatar image

We are looking at providing this functionality in Transformer in a future release. I'll come back and update this answer with any news.

In the meantime, you might want to look at using StreamSets Data Collector for these tables. It does not have the 'numeric offset column' requirement.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-12-17 11:43:05 -0600

Seen: 30 times

Last updated: Dec 17 '19