Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Streamsets Transformer - JDBC Origin without offset column

I'm testing platforms that can allow any user to easily create data processing pipelines. This platform has to meet certain requirements and one of them is to be capable of moving data from Oracle/SQL Server to HDFS.

Streamsets Transformer (v3.11) meets all requirements including the one referred above. I just can't get it to work in a very specific case: When ingesting a table that contains no numeric columns.

In these cases I want the pipeline to process all data so, in the JDBC Origin, I enabled the "Skip Offset Tracking" property. I thought that by skipping the offset tracking there would be no need to set the "Offset Column" property (guess I was wrong).

JDBC_05 - Table doesn't have compatible primary key configuration - supporting exactly one column but table have 0

If a numeric column exists, a possible workaround is to set it as the offset column but I can't find a way of doing this when none exists.

Am I missing something?

Thanks

( original question here: https://stackoverflow.com/questions/59378072/streamsets-transformer-jdbc-origin-without-offset-column )

Streamsets Transformer - JDBC Origin without offset column

I'm testing platforms that can allow any user to easily create data processing pipelines. This platform has to meet certain requirements and one of them is to be capable of moving data from Oracle/SQL Server to HDFS.

Streamsets Transformer (v3.11) meets all requirements including the one referred above. I just can't get it to work in a very specific case: When ingesting a table that contains no numeric columns.

In these cases I want the pipeline to process all data so, in the JDBC Origin, I enabled the "Skip Offset Tracking" property. I thought that by skipping the offset tracking there would be no need to set the "Offset Column" property (guess I was wrong).

JDBC_05 - Table doesn't have compatible primary key configuration - supporting exactly one column but table have 0

If a numeric column exists, a possible workaround is to set it as the offset column but I can't find a way of doing this when none exists.

Am I missing something?

Thanks

( original question here: https://stackoverflow.com/questions/59378072/streamsets-transformer-jdbc-origin-without-offset-column )