Ask Your Question
0

DC not starting on Hortonworks HDP 2.6

asked 2018-03-22 21:04:53 -0600

AdeelH gravatar image

updated 2018-03-23 11:32:38 -0600

jeff gravatar image

I am trying to install Streamsets on a single node Hadoop box (Hortonworks Sandbox). The install process is quite straightforward on the Streamsets website

Download the core tar file, untar it and then run "streamsets-datacollector-3.1.2.0/bin/streamsets dc" to start the DataCollector on port 18630.

Somehow this port is never opened and so connection fails. I used netcat to verify that this port remains closed. I have read through the logs and it does show "StandaloneAndClusterPipelineManager - Stopped Production Pipeline Manager" but I am not sure if this a problem or how to fix it.

Screen Output

[root@sandbox-hdp streams]# streamsets-datacollector-3.1.2.0/bin/streamsets dc
Java 1.8 detected; adding $SDC_JAVA8_OPTS of "-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Djdk.nio.maxCachedBufferSize=262144" to $SDC_JAVA_OPTS
Logging initialized @2385ms to org.eclipse.jetty.util.log.Slf4jLog
Running on URI : 'http://sandbox-hdp.hortonworks.com:18630'

Log

2018-03-21 11:03:20,046 [user:] [pipeline:] [runner:] [thread:main] INFO  Main - -----------------------------------------------------------------
2018-03-21 11:03:20,047 [user:] [pipeline:] [runner:] [thread:main] INFO  Main - Build info:
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Version        : 3.1.2.0
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Date           : 2018-03-20T18:19Z
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Built by       : root
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Repo SHA       : 2f3225224e5ce2beff355254e44ffa9e3c48863a
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Source MD5     : 1aa7555e88e9e601b82336af48d3f97
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main - -----------------------------------------------------------------
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main - Runtime info:
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Java version : 1.8.0_161-b14
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   SDC ID       : 76d2066d-2cf7-11e8-adc1-4d8940914dad
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Runtime dir  : /streams/streamsets-datacollector-3.1.2.0
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Config dir   : /streams/streamsets-datacollector-3.1.2.0/etc
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Data dir     : /streams/streamsets-datacollector-3.1.2.0/data
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Log dir      : /streams/streamsets-datacollector-3.1.2.0/log
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main - -----------------------------------------------------------------
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Security Manager : ENABLED, policy file: file:///streams/streamsets-datacollector-3.1.2.0/etc/sdc-security.policy
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main - -----------------------------------------------------------------
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO  Main - Starting ...
2018-03-21 11:03:20,066 [user:] [pipeline:] [runner:] [thread:main] INFO  Main - -----------------------------------------------------------------
2018-03-21 11:03:20,066 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Kerberos enabled: false
2018-03-21 11:03:20,073 [user:] [pipeline:] [runner:] [thread:main] INFO  Main -   Unlimited cryptography enabled: true
2018-03-21 11:03:20,073 [user:] [pipeline:] [runner:] [thread ...
(more)
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-03-23 11:33:54 -0600

jeff gravatar image

It seems like perhaps your VM is not able to provide enough memory for the JVM process running Data Collector to allocate its heap (which is by default 1GB, and you don't appear to have overridden that). Can you post the output of top and free commands, run from a separate terminal session in the VM?

edit flag offensive delete link more

Comments

Hi Jeff, I have uploaded the required screenshots to this dropbox link. I could not upload them on the forum. https://www.dropbox.com/sh/rd0o4f1894z9q5d/AADjDiUQ3tMtfNG0JXc3HDHWa?dl=0 Thanks for your help. Regards, Adeel

AdeelH gravatar imageAdeelH ( 2018-03-23 18:50:22 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-03-22 21:04:53 -0600

Seen: 88 times

Last updated: Mar 23