JDBC Stages and On Unknown Type => Convert to String

asked 2018-09-14 14:41:15 -0500

__jeff__
11 ●1 ●1 ●3

updated 2018-09-17 10:44:43 -0500

not sure what i don't understand, but it looks like streamsets hardcode this parameter to always be STOP_PIPELINE and not configurable? This is from the configBean...

 @ConfigDef(
      required = true,
      type = ConfigDef.Type.MODEL,
      label = "On Unknown Type",
      description = "Action that should be performed when an unknown type is detected in the result set.",
      defaultValue = "STOP_PIPELINE",
      displayPosition = 220,
      group = "ADVANCED"
  )
  @ValueChooserModel(UnknownTypeActionChooserValues.class)
  public UnknownTypeAction unknownTypeAction = UnknownTypeAction.STOP_PIPELINE;

Error:

java.lang.IllegalArgumentException: Type:-155 is not a valid Types.java value. at java.sql.JDBCType.valueOf(Unknown Source) at com.streamsets.pipeline.lib.jdbc.multithread.TableContext.isPartitionable(TableContext.java:215) at com.streamsets.pipeline.lib.jdbc.multithread.TableContext.isPartitionable(TableContext.java:186) at com.streamsets.pipeline.lib.jdbc.multithread.TableContext.<init>(TableContext.java:109) at...

Yes, I understand that the datatimeoffset from SQL is a type -155, which isn't in the java.sql library StreamSets is using. Hence why I wanted to use TO_STRING on the unknown datatypes. Are you saying that StreamSets has no solution for this problem? I tried this and couldn't get the TO_STRING on UNKNOWN_TYPES to work, and thus I started to dig into the code to see why it wasn't working. Is there not a way to move forward?

answered 2018-09-14 14:54:19 -0500

metadaddy

5464 ●26 ●41 ●81 https://about.me/patpa...

updated 2018-09-16 18:47:43 -0500

STOP_PIPELINE is just the default value. If you look in UnknownTypeAction.java you'll see both STOP_PIPELINE and CONVERT_TO_STRING.

I just checked and I was able to select 'Convert to String' in the JDBC Query Consumer origin's advanced tab without a problem:

image description

JdbcUtil.resultToField() handles unknown types according to the configured action:

      default:
        if(unknownTypeAction == null) {
          return null;
        }
        switch (unknownTypeAction) {
          case STOP_PIPELINE:
            throw new StageException(JdbcErrors.JDBC_37, md.getColumnType(columnIndex), md.getColumnLabel(columnIndex));
          case CONVERT_TO_STRING:
            Object value = rs.getObject(columnIndex);
            if(value != null) {
              field = Field.create(Field.Type.STRING, rs.getObject(columnIndex).toString());
            } else {
              field = Field.create(Field.Type.STRING, null);
            }
            break;
          default:
            throw new IllegalStateException("Unknown action: " + unknownTypeAction);
        }

The specific error you are seeing seems to be related to trying to use a column with a non-standard JDBC type (-155 is MS SQL Server's DateTimeOffset type) to partition the data.

edit flag offensive delete link

Comments

The pipeline fails to validate with datetimeoffset types even with CONVERT_TO_STRING selected. This pipeline works fine on other tables. How do I move forward?

__jeff__ ( 2018-09-14 16:03:09 -0500 )edit

My question about the hardcoding is on the backend, not whether or not the user can select a value from the drop down.

__jeff__ ( 2018-09-14 16:11:50 -0500 )edit

Updated my answer with the back end code that tests the unknown type action. Could you edit the error message you see at validation into your question?

metadaddy ( 2018-09-14 16:31:11 -0500 )edit

Added some more detail related to the error you are seeing

metadaddy ( 2018-09-16 18:48:02 -0500 )edit

add a comment

JDBC Stages and On Unknown Type => Convert to String

1 Answer

Comments

Question Tools

Stats

JDBC Stages and On Unknown Type => Convert to String edit

1 Answer

Comments

Question Tools

Stats

JDBC Stages and On Unknown Type => Convert to String