Ask Your Question
1

HadoopFS destination not firing Event to "invalidate metadata" for Impala

asked 2018-01-04 09:36:28 -0500

gaurav gravatar image

Hi Everyone, I am trying to replicate a database (MySQL) to Hive using the following blog post: https://streamsets.com/blog/blogrepli...

As I add a new record in one of the source tables in MySQL it gets replicated to Hive (verified successfully), however the event which should be fired by HadoopFS destination to "invalidate metadata" for Impala by Hive Query executor is not working. I checked the dashboard of workflow and there is NO event getting fired.

I have checked and verified each config details from the blog post and it's all good.

Can anyone guide me as to what could be the issue.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-01-04 18:03:51 -0500

metadaddy gravatar image

If the Hadoop FS destination is not firing the event, then it likely isn't closing the file. Take a look at the destination's idle timeout configuration; by default, the Hadoop FS destination will wait for an hour before closing the output file.

edit flag offensive delete link more

Comments

Thanks Pat, it worked.Quick question. Stopped and reran pipeline & tried to add a new record in "departments" table, it got appended in hive. Impala could fetch it(without me running "invalidate metadata") but in dashboard there was NO event fired. I inserted 1 more rec and event was fired. Why?

gaurav gravatar imagegaurav ( 2018-01-04 22:21:39 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-01-04 09:34:59 -0500

Seen: 64 times

Last updated: Jan 04