Ask Your Question

Is it possible for JDBC Producer to process MySQLBinLog without specifying field mappings ?

asked 2018-06-28 04:19:48 -0500

akianna gravatar image

updated 2018-06-28 08:21:49 -0500

metadaddy gravatar image

I am testing streamsets right now and trying to move data from MySQL to MySQL and I have a simple setup of MySQL Binary Log -> JDBC Producer (with ChangeLog specified). On both ends I have created a table with following fields:

  name varchar(255),
  surname varchar(255)

It says in the documentation that "By default, fields are written to columns of the same name." and Field to Column Mapping is required when column name differs. Hovewer the setup works for me only if I specify field mappings for each field.

If I don't specify the mappings then I have these issues:

  • if I have AUTO_INCREMENT on id field, then the insert will be marked as successful, when in fact only a new empty row with autogenerated id is inserted and other columns are left blank.
  • if I don't have AUTO_INCREMENT insert will fail immediatly with "Field 'id' doesn't have a default value"

Is it possible for JDBC Producer to process MySQLBinLog without specifying field mappings when columns are named the same?

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted

answered 2018-07-13 10:12:27 -0500

todd gravatar image

I'm running into this as well.

There is an open issue

edit flag offensive delete link more


Perfect, thx for info

peter_f gravatar imagepeter_f ( 2018-07-13 11:02:00 -0500 )edit

answered 2018-07-12 07:14:53 -0500

peter_f gravatar image

It seems that the JDBC producer is currently broken and does not work without defining a complete field to column mapping.

I was able to reproduce this issue using Streamsets SDC 3.3.1 and MySql 5.7, with MySQL Binary Log -> JDBC Producer. Without a field to column map the producer stage tries to insert/update null values to all columns of the target table.

I think this question is also caused by the same issue.

Apart of that, the drives needed by the two stages differ, the MySQL Binary Log works with the latest MySql 8 drivers, for the producer you have to stick with 5.1 ones. Otherwise the producer has problems looking up the target table in the info schema, this is also ... at least ... sub-par.

If there is no open issue regarding this on Jira, I'll be happy to create one, add a sample and logs to track this down.

Best Peter

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-06-28 04:19:48 -0500

Seen: 19,250 times

Last updated: Jul 13 '18