Ask Your Question
1

streamsets-datacollector-mapr_5_2-lib install issue

asked 2018-05-07 18:18:23 -0500

itsmahi gravatar image

updated 2018-05-15 14:57:49 -0500

metadaddy gravatar image

I have installed streamsets-datacollector-mapr_5_2-lib multiple times but after i restarted it still says uninstalled. Is there any setting i need to set to make it available?

I am getting below error after mapr setup:

Logging initialized @2340ms to org.eclipse.jetty.util.log.Slf4jLog Abnormal exit: java.lang.NoClassDefFoundError: org/apache/hadoop/io/compress/GzipCodecCheck STDERR for more details java.lang.NoClassDefFoundError: org/apache/hadoop/io/compress/GzipCode

after the mapr-setup completes it removes the entry from the below property and streamsets won't restart. If i add this entry back i am able to restart streamsets but i don't see mapr setup. system.stagelibs.blacklist=streamsets-datacollector-mapr_5_2-lib

[thread:main] ERROR ClassLoaderStageLibraryTask - The following stages have invalid classpath: streamsets-datacollector-mapr_5_2-lib 2018-05-08 16:26:13,999 [user:] [pipeline:] [runner:] [thread:main] ERROR Main - Abnormal exit: java.lang.NoClassDefFoundError: org/apache/hadoop/io/compress/GzipCodec

Below command returned nothing. I am not sure if GzipCodec is part of maprfs 5.2.2. jar file. Is it a bug in streamsets?

./jar -tf /opt/mapr/lib/maprfs-5.2.2-mapr.jar | grep GzipCodec

Output from script:

+ BLACKLIST_PROP=system.stagelibs.blacklist
+ PROP_FILENAME=sdc.properties
+ POLICY_FILENAME=sdc-security.policy
+ MAPR_LIB_BASE=streamsets-datacollector-mapr
+ MAPR_SPARK_LIB='streamsets-datacollector-mapr_spark*-lib'
+ '[' -z /infadm/streamsets/streamsets-datacollector-3.2.0.0 ']'
+ '[' '!' -f /infadm/streamsets/streamsets-datacollector-3.2.0.0/bin/streamsets ']'
+ '[' -z /infadm/streamsets/streamsets-datacollector-3.2.0.0/etc ']'
+ '[' '!' -f /infadm/streamsets/streamsets-datacollector-3.2.0.0/etc/sdc.properties ']'
+ SDC_PROP_FILE=/infadm/streamsets/streamsets-datacollector-3.2.0.0/etc/sdc.properties
+ SDC_POLICY_FILE=/infadm/streamsets/streamsets-datacollector-3.2.0.0/etc/sdc-security.policy
+ '[' '!' -w /infadm/streamsets/streamsets-datacollector-3.2.0.0/etc/sdc.properties ']'
+ [[ linux-gnu == \d\a\r\w\i\n* ]]
+ [[ linux-gnu == \c\y\g\w\i\n ]]
+ [[ -z '' ]]
+ read -p 'Please enter the MapR version (default 6.0.0): ' MAPR_VERSION
Please enter the MapR version (default 6.0.0): 5.2.2
+ MAPR_VERSION=5.2.2
+ [[ 5.2.2 =~ ^[1-9][0-9]*\.[0-9]*(\.[0-9]*)*$ ]]
++ expr 5.2.2 : '\([1-9][0-9]*\.[0-9]*\)'
+ MAPR_VERSION=5.2
+ _MAPR_VERSION=5_2
+ MAPR_LIB=streamsets-datacollector-mapr_5_2-lib
+ '[' '!' -d /infadm/streamsets/streamsets-datacollector-3.2.0.0/streamsets-libs/streamsets-datacollector-mapr_5_2-lib ']'
+ '[' '!' -z ']'
++ ls /infadm/streamsets/streamsets-datacollector-3.2.0.0/streamsets-libs
++ grep 'streamsets-datacollector-mapr_spark_*'
+ SDC_MAPR_SPARK_LIB=
++ echo
++ cut -d_ -f 3,4
++ sed s/_/./g
+ SDC_SPARK_VERSION=
+ [[ -z '' ]]
+ read -p 'Please enter the absolute path of MapR Home (default /opt/mapr): ' MAPR_HOME
Please enter the absolute path of MapR Home (default /opt/mapr): /opt/mapr
+ MAPR_HOME=/opt/mapr
+ '[' '!' -e /opt/mapr/MapRBuildVersion ']'
+ '[' '!' -d /opt/mapr/spark ']'
+ echo 'Warning: Spark is not installed. Skipping MapR Spark enablement.'
Warning: Spark is not installed. Skipping MapR Spark enablement.
+ echo ''

+ echo Configuration:
Configuration:
+ echo SDC_HOME=/infadm/streamsets/streamsets-datacollector-3.2.0.0
SDC_HOME=/infadm/streamsets/streamsets-datacollector-3.2.0.0
+ echo SDC_CONF=/infadm/streamsets/streamsets-datacollector-3.2.0.0/etc
SDC_CONF=/infadm/streamsets/streamsets-datacollector-3.2.0.0/etc
+ echo MAPR_HOME=/opt/mapr
MAPR_HOME=/opt/mapr
+ echo MAPR_VERSION=5.2
MAPR_VERSION=5.2
+ echo MAPR_MEP_VERSION=
MAPR_MEP_VERSION=
+ '[' '!' -z ']'
+ echo ''

+ echo 'Updating sdc.properties file ....'
Updating sdc.properties file ....
+ blacklist_remove streamsets-datacollector-mapr_5_2-lib
+ STAGE_LIB=streamsets-datacollector-mapr_5_2-lib
+ echo 'Enabling streamsets-datacollector-mapr_5_2-lib'
Enabling streamsets-datacollector-mapr_5_2-lib
++ grep -i system.stagelibs.blacklist /infadm/streamsets/streamsets-datacollector-3.2.0.0/etc/sdc ...
(more)
edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted
1

answered 2018-05-25 13:15:44 -0500

junko_urata gravatar image

updated 2018-05-25 13:33:31 -0500

metadaddy gravatar image

GzipCodec is in hadoop-common-*.jar and that's what you need (not streamsets-datacollector-hadoop-common-*.jar).

Can you please share output of ls -l /opt/mapr/lib?

edit flag offensive delete link more

Comments

1

Filed https://issues.streamsets.com/browse/SDC-9081 and mentioned a workaround there.

junko_urata gravatar imagejunko_urata ( 2018-05-25 13:47:18 -0500 )edit
0

answered 2018-05-07 18:39:51 -0500

metadaddy gravatar image

There are additional steps to setting up MapR support, since we cannot ship their libraries due to their license - see https://streamsets.com/documentation/...

edit flag offensive delete link more

Comments

I have completed all prerequisites and i see the library installed on linux server and all soft links created but i don't see origins and destinations for mapr.

itsmahi gravatar imageitsmahi ( 2018-05-07 20:12:01 -0500 )edit

Did you run the setup-mapr script? If so, please send the output of the script.

metadaddy gravatar imagemetadaddy ( 2018-05-08 13:42:56 -0500 )edit

I am not able to add everything in the comments, so i posted the whole content as an answer.

itsmahi gravatar imageitsmahi ( 2018-05-08 16:14:49 -0500 )edit

I am getting below error after mapr setup Logging initialized @2340ms to org.eclipse.jetty.util.log.Slf4jLog Abnormal exit: java.lang.NoClassDefFoundError: org/apache/hadoop/io/compress/GzipCodecCheck STDERR for more details java.lang.NoClassDefFoundError: org/apache/hadoop/io/compress/GzipCode

itsmahi gravatar imageitsmahi ( 2018-05-08 16:19:24 -0500 )edit

after the mapr-setup completes it removes the entry from the below property and streamsets won't restart. If i add this entry back i am able to restart streamsets but i don't see mapr setup. system.stagelibs.blacklist=streamsets-datacollector-mapr_5_2-lib

itsmahi gravatar imageitsmahi ( 2018-05-08 16:21:28 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-05-07 18:17:59 -0500

Seen: 118 times

Last updated: May 25