Ask Your Question

Crud operations in a single pipeline

asked 2017-09-13 01:17:16 -0600

sujeet gravatar image

I want to create a pipeline that detect changes in my mysql db and write data into another table in mysql. But only new item detection is happening why not update item and delete item ?

edit retag flag offensive close merge delete


Which origin are you using?

metadaddy gravatar imagemetadaddy ( 2017-09-13 16:07:05 -0600 )edit

Origin- jdbc multitable destination- hadoop fs

sujeet gravatar imagesujeet ( 2017-09-15 06:57:37 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2017-09-15 12:20:10 -0600

jeff gravatar image

How does the source table get updated? There is nothing magical about the multitable consumer, and it's not fundamentally a change tracking mechanism. It's fundamentally just running a select * from table where offset > X and X gets updated as data is consumed. It sounds like you probably need a last_modified_timestamp column in your table that gets updated on every update/delete, and to use that as the offset column for the table.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-09-13 01:17:16 -0600

Seen: 146 times

Last updated: Sep 15 '17