Ask Your Question
1

I have to create a pipeline which should trigger automatically, whenever new data arrived or existing columns get modify

asked 2018-11-19 23:05:01 -0600

raghav.maurya gravatar image

updated 2018-11-20 23:35:15 -0600

Shruthi gravatar image

I am am using MySQL as a database. Please mention which origin i should use.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-11-20 23:13:23 -0600

Shruthi gravatar image

updated 2018-12-06 07:05:53 -0600

You can use 'Mysql Binary Log' as origin for CDC.

For example,

  • Create pipeline with origin 'Mysql Binary Log' and choose your proper destination(Here i used 'Local FS' for eg). Also you can add processors in between according to your requirements(like i used 'Field remover' )

image description

  • You can configure 'Mysql Binary Log' as follows, image description

    Give your database credentials on 'Credential' tab and configure 'Advanced' tab as, image description

  • Before doing this you need to enable binlog in your streamsets... i.e, do following changes to /etc/mysql/mariadb.conf.d/50-server.cnf (enable the following by removing # tag)

    server-id = 1

    log_bin = /var/log/mysql/mariadb-bin

    log_bin_index = /var/log/mysql/mariadb-bin.index

    sync_binlog = 1

    binlog_format=row

    innodb_flush_log_at_trx_commit=0

edit flag offensive delete link more

Comments

Hi Shruthi, Thanks for sharing the details. My admin installed the the MySQL driver but while using MySQL Binary Log i am getting runtime exception "unable to get driver instance for jdbcURL". Bu i can use the same database source with JDBC Query consumer and JDBC Multitable Consumer. Please suggest

raghav.maurya gravatar imageraghav.maurya ( 2018-11-21 02:57:56 -0600 )edit

Did you enable Binlog in your configuration file?

Shruthi gravatar imageShruthi ( 2018-11-21 03:10:00 -0600 )edit

On the database side or it is required somewhere in streamsets ?

raghav.maurya gravatar imageraghav.maurya ( 2018-11-21 03:11:16 -0600 )edit

I have updated my answer please do the following changes in config file like i mentioned.

Shruthi gravatar imageShruthi ( 2018-11-21 03:24:23 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-11-19 23:05:01 -0600

Seen: 84 times

Last updated: Dec 06