Ask Your Question

MySQL to Apache Phoenix

asked 2017-11-01 08:00:29 -0500

casel.chen gravatar image

updated 2017-11-02 12:45:32 -0500

metadaddy gravatar image

Does StreamSets DC support migrating data from MySQL to Apache Phoenix directly?

After trying with JDBC Producer destination, the data pipeline reports the following error:

JDBC_00 - Cannot connect to specified database: com.zaxxer.hikari.pool.PoolInitializationException: Exception during pool initialization: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator

I added the following jars in sdc-extras/streamsets-datacollector-jdbc-lib/lib directory:

  • mysql-connector-java-5.1.42-bin.jar
  • commons-configuration-1.6.jar
  • hbase-common-1.2.3.jar
  • phoenix-core-4.11.0-HBase-1.2.jar
  • hadoop-auth-2.5.1.jar
  • zookeeper-3.4.6.jar
  • hadoop-common-2.7.1.jar
  • phoenix-4.11.0-HBase-1.2-client.jar
  • hbase-client-1.2.3.jar
  • phoenix-4.11.0-HBase-1.2-thin-client.jar

Any idea how to fix it?

edit retag flag offensive close merge delete


Should I use JDBC Producer to write data into apache phoenix?

casel.chen gravatar imagecasel.chen ( 2017-11-01 09:18:07 -0500 )edit

See my updated answer

metadaddy gravatar imagemetadaddy ( 2017-11-02 12:50:24 -0500 )edit

2 Answers

Sort by ยป oldest newest most voted

answered 2017-11-10 19:46:16 -0500

casel.chen gravatar image

I upgraded hbase to 1.3.1 version and apache phoenix to apache-phoenix-4.12.0-HBase-1.3-bin, then tried again with JDBC Producer destination, it shows the following error. I guess apache phoenix doesn't support "INSERT" and "UPDATE" statements but only "UPSERT" statement. So, How can I work around it? Thanks!

Ps: I also tried config "Default Operation" as "UPDATE", a new error occurs: "Pipeline Status: RUNNING_ERROR: com.streamsets.pipeline.api.StageException: JDBC_14 - Error processing batch. java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixParserException: ERROR 602 (42P00): Syntax error. Missing "STATISTICS" at line 1, column 8."

image description

edit flag offensive delete link more


Should I need use Flume Destination as I found there is a Phoenix Flume plugin? But how to use Flume with StreamSets DC? Is there any example?

casel.chen gravatar imagecasel.chen ( 2017-11-12 20:07:52 -0500 )edit

answered 2017-11-01 10:43:08 -0500

metadaddy gravatar image

updated 2017-11-02 12:49:41 -0500

I don't know of anyone that's tried it, but it should work via the JDBC Producer destination.

[UPDATE] - It looks like you're hitting HBASE-14963, which is fixed in HBase versions 1.3.0, 1.4.0 and 2.0.0. I don't think there's a way to work around this in HBase 1.2.3.

edit flag offensive delete link more


Please see my feedback.

casel.chen gravatar imagecasel.chen ( 2017-11-10 19:46:10 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-11-01 08:00:29 -0500

Seen: 3,187 times

Last updated: Nov 10 '17