Ask Your Question
1

JDBC Stages and On Unknown Type => Convert to String

asked 2018-09-14 14:41:15 -0600

__jeff__ gravatar image

updated 2018-09-17 10:44:43 -0600

not sure what i don't understand, but it looks like streamsets hardcode this parameter to always be STOP_PIPELINE and not configurable? This is from the configBean...

 @ConfigDef(
      required = true,
      type = ConfigDef.Type.MODEL,
      label = "On Unknown Type",
      description = "Action that should be performed when an unknown type is detected in the result set.",
      defaultValue = "STOP_PIPELINE",
      displayPosition = 220,
      group = "ADVANCED"
  )
  @ValueChooserModel(UnknownTypeActionChooserValues.class)
  public UnknownTypeAction unknownTypeAction = UnknownTypeAction.STOP_PIPELINE;

Error:

java.lang.IllegalArgumentException: Type:-155 is not a valid Types.java value. at java.sql.JDBCType.valueOf(Unknown Source) at com.streamsets.pipeline.lib.jdbc.multithread.TableContext.isPartitionable(TableContext.java:215) at com.streamsets.pipeline.lib.jdbc.multithread.TableContext.isPartitionable(TableContext.java:186) at com.streamsets.pipeline.lib.jdbc.multithread.TableContext.<init>(TableContext.java:109) at...

Yes, I understand that the datatimeoffset from SQL is a type -155, which isn't in the java.sql library StreamSets is using. Hence why I wanted to use TO_STRING on the unknown datatypes. Are you saying that StreamSets has no solution for this problem? I tried this and couldn't get the TO_STRING on UNKNOWN_TYPES to work, and thus I started to dig into the code to see why it wasn't working. Is there not a way to move forward?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2018-09-14 14:54:19 -0600

metadaddy gravatar image

updated 2018-09-16 18:47:43 -0600

STOP_PIPELINE is just the default value. If you look in UnknownTypeAction.java you'll see both STOP_PIPELINE and CONVERT_TO_STRING.

I just checked and I was able to select 'Convert to String' in the JDBC Query Consumer origin's advanced tab without a problem:

image description

JdbcUtil.resultToField() handles unknown types according to the configured action:

      default:
        if(unknownTypeAction == null) {
          return null;
        }
        switch (unknownTypeAction) {
          case STOP_PIPELINE:
            throw new StageException(JdbcErrors.JDBC_37, md.getColumnType(columnIndex), md.getColumnLabel(columnIndex));
          case CONVERT_TO_STRING:
            Object value = rs.getObject(columnIndex);
            if(value != null) {
              field = Field.create(Field.Type.STRING, rs.getObject(columnIndex).toString());
            } else {
              field = Field.create(Field.Type.STRING, null);
            }
            break;
          default:
            throw new IllegalStateException("Unknown action: " + unknownTypeAction);
        }

The specific error you are seeing seems to be related to trying to use a column with a non-standard JDBC type (-155 is MS SQL Server's DateTimeOffset type) to partition the data.

edit flag offensive delete link more

Comments

The pipeline fails to validate with datetimeoffset types even with CONVERT_TO_STRING selected. This pipeline works fine on other tables. How do I move forward?

__jeff__ gravatar image__jeff__ ( 2018-09-14 16:03:09 -0600 )edit

My question about the hardcoding is on the backend, not whether or not the user can select a value from the drop down.

__jeff__ gravatar image__jeff__ ( 2018-09-14 16:11:50 -0600 )edit

Updated my answer with the back end code that tests the unknown type action. Could you edit the error message you see at validation into your question?

metadaddy gravatar imagemetadaddy ( 2018-09-14 16:31:11 -0600 )edit

Added some more detail related to the error you are seeing

metadaddy gravatar imagemetadaddy ( 2018-09-16 18:48:02 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-09-14 14:41:15 -0600

Seen: 37 times

Last updated: Sep 17