Ask Your Question
1

unable to start StreamSets using streamsets dc

asked 2017-09-09 13:07:18 -0500

sushilindia gravatar image

updated 2017-09-20 13:09:13 -0500

LC gravatar image
  1. Not able to start streamsets using command /bin/streamsets dc, tried with sudo also but giving a message that streamsets: command not found .i am using ubuntu 17.04.

  2. Then i tried with /bin/sudo ./streamsets dc ,which gives below error which i am not able to understand and fix after reffering the link and dont know how and where to set the limit 32768
    Error:
    Configuration of maximum open file limit is too low: 1024 (expected at least 32768). Please consult https://goo.gl/LgvGF

Thanks in Advance

edit retag flag offensive close merge delete

5 Answers

Sort by ยป oldest newest most voted
2

answered 2017-09-11 13:11:28 -0500

jeff gravatar image

updated 2017-09-11 13:27:33 -0500

Please refer to the Configuring the Open File Limit of the documentation here.

edit flag offensive delete link more
2

answered 2017-09-11 13:14:05 -0500

You don't need to start DC as root. But you were typing /bin/streamsets dc -- in fact, if you're in the main Data Collector directory, you'd just type something like ./bin/streamsets dc (a relative path, not the absolute path you were using). In Ubuntu, you'll need to edit /etc/security/limits.conf and add two lines, something like:

* soft nofiles 32768

* hard nofiles 32768

To make this take effect for your current session, run ulimit -n 32768

edit flag offensive delete link more

Comments

I am using a company mac , where I am not given a sudo access. Is there any workaround to increase the limits without being a sudo.

raghuramg gravatar imageraghuramg ( 2019-12-10 03:16:42 -0500 )edit
2

answered 2017-09-22 03:21:59 -0500

sujeet gravatar image

Before starting streamsets use "ulimit -n 32768" . It will fix the issue

edit flag offensive delete link more
1

answered 2017-09-11 13:32:11 -0500

kirti gravatar image

updated 2017-09-11 13:32:37 -0500

Also, you can try StreamSets Data Collector with Docker. If interested, check out Getting Started with StreamSets Data Collector on Docker.

edit flag offensive delete link more
0

answered 2020-04-20 12:18:25 -0500

Paul Tipper gravatar image

I'm trying to start the streamsets-dc docker container, but the container is exiting on start-up. Here's what I see when I run docker ps -a:

18:06 $ docker ps -a
CONTAINER ID  IMAGE                                           COMMAND   CREATED        STATUS                    PORTS                             NAMES
ec48521394f6  docker.io/streamsets/datacollector:latest       dc        8 seconds ago  Exited (1) 7 seconds ago  0.0.0.0:18630->18630/tcp          streamsets-dc
82bdce9df314  localhost/identityserverui:latest                         2 months ago   Created                   0.0.0.0:5000-5001->5000-5001/tcp  identity-server_ui_1
6bc11a3bd09e  identityserverregistry.azurecr.io/idxapi:2.3.4            2 months ago   Created                   0.0.0.0:5000-5001->5000-5001/tcp  identity-server_api_1
469d193f09c5  localhost/identityserver:latest                           2 months ago   Created                   0.0.0.0:5000-5001->5000-5001/tcp  identity-server_idp_1
f2be3820eed9  docker.io/library/postgres:latest               postgres  2 months ago   Created                   0.0.0.0:5000-5001->5000-5001/tcp  identity-server_db_1
da3006a3c4b3  k8s.gcr.io/pause:3.1                                      2 months ago   Created                   0.0.0.0:5000-5001->5000-5001/tcp  d5a2b56c021b-infra

You can see that the streamsets-dc container exits immediately on start-up. If I run the docker logs streamsets-dc command, I get the following

Java 1.8 detected; adding $SDC_JAVA8_OPTS of "-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Djdk.nio.maxCachedBufferSize=262144" to $SDC_JAVA_OPTS
Configuration of maximum open file limit is too low: 1024 (expected at least 32768). Please consult https://goo.gl/6dmjXd

I have to admit I'm not all that familiar with docker, but I would have assumed that the streamsets-dc container would have everything configured to just run out of the box.

Can anyone suggest what I need to do to get the container to run?

edit flag offensive delete link more

Comments

Hi Paul - you've posted this as an answer to another question. Could post it as a new question? That way more people will see it. https://ask.streamsets.com/questions/ask/

metadaddy gravatar imagemetadaddy ( 2020-04-20 16:55:52 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-09-09 13:05:48 -0500

Seen: 2,516 times

Last updated: Sep 22 '17