Ask Your Question

Which processor need to be used to query on CDC data from MySQL Binlog to any destination?

asked 2018-01-21 23:18:21 -0500

Maithri gravatar image

updated 2018-01-29 00:29:30 -0500

I want to fetch table data (2-3 tables) from mariadb,achieve CDC,apply some filter condition or query on this data using any processor and dump it to multiple destinations based on that conditions( or query).

I used MysqlBinary log origin and fetched data(CDC),I am stuck to choose a processor for this requirement.Please someone suggest me the solution for this requirement.


edit retag flag offensive close merge delete


I've given a basic answer based on the filtering requirement. Can you expand on what you mean by 'query on this data'. Do you mean query MariaDB for additional data based on the CDC records?

metadaddy gravatar imagemetadaddy ( 2018-01-24 11:09:31 -0500 )edit

2 Answers

Sort by ยป oldest newest most voted

answered 2018-09-10 23:53:10 -0500

Maithri gravatar image

updated 2018-09-11 00:32:02 -0500

I could achieve this requirement by using MysqlBinary log origin for CDC where I included all the required tables.Then used a JDBC lookup processor to query on the tables included in the origin where i could apply JOIN as well and a condition like:

 WHERE a.emp_id =${record:value('/Data/emp_id')}
edit flag offensive delete link more

answered 2018-01-24 11:08:25 -0500

metadaddy gravatar image

You can filter records and send them to different destinations using the Stream Selector processor - just write conditions like:

${record:value('/fieldname') == 'bob'}

and send the records to different streams.

edit flag offensive delete link more


I want to fetch data from mariadb by joining tables with where conditions.I could achieve this by using JDBC query consumer.

Maithri gravatar imageMaithri ( 2018-01-24 23:27:11 -0500 )edit

But I want to achieve CDC as well, so JDBC query consumer did not support these changes from mariadb and I was suggested to use Mysql Binary Log.The problem I am facing here is that I cannot write the join query and the where conditions how I used for Jdbc Query consumer in Mysql Binary Log.

Maithri gravatar imageMaithri ( 2018-01-24 23:30:13 -0500 )edit

Example: select x,y,(select a from table b where a.b1=some value) from table a join table b on(a.somevaule=c.somevalue)

Maithri gravatar imageMaithri ( 2018-01-24 23:31:51 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-01-21 23:18:21 -0500

Seen: 2,126 times

Last updated: Sep 11 '18