Ask Your Question
0

Hadoop URI from origin not connecting to StreamSets

asked 2019-07-09 10:11:18 -0600

Sunjana K. gravatar image

I set up my StreamSets and Cloudera VM environments using Docker. The URL I'm using to access Hue (GUI of Hadoop) is https://localhost:8888. I then accessed Cloudera VM thru bash, and did Hadoop fs -ls hdfs://localhost:8888/, but got this error:

Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message tag had invalid wire type.; Host Details : local host is: "quickstart.cloudera/172.17.0.3"; destination host is: "localhost":8888;

This is my first issue. I'm not sure why this isn't working bc I verified numerous times that my working port for Hue is 8888. It does, however work on hdfs://localhost:8020, and I was able to see the path to my file from there.

I've tried all of these options:

hdfs://172.17.0.3:8888 hdfs://localhost:8888 hdfs://localhost:8020 hdfs://quickstart.cloudera:8020 hdfs://quickstart.cloudera:8888

All with Input Path:

/user/cloudera/trade_file.csv

With this result: Cannot connect to the filesystem. Check if the Hadoop FS location: '/user/cloudera/trade_file.csv' is valid or not: 'java.net.ConnectException: Call From bdf224f41340/172.17.0.2 to localhost:8888 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/Connect...

I'm guessing my issue has to be with hadoop fs -ls not working on localhost:8888, and localhost:8020 not existing in web form. However, neither am I able to establish Hue in https://localhost:8020, nor am I able to figure out why 8888 isn't able to be accessed despite it working perfectly on the browser.

Could someone please help?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2019-07-11 12:07:33 -0600

Sunjana K. gravatar image

SOLVED: connect Cloudera VM and StreamSets containers(give each container a name) thru a Docker bridge, which you must define yourself. See these resources:

https://docs.docker.com/v17.09/engine... https://stackoverflow.com/questions/5... (with this, use docker network connect instead of just docker connect)

Then your URI should be hdfs://[Cloudera container name]:[bridge] and your Input Path should be the directory to the file you want

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-07-09 10:11:18 -0600

Seen: 258 times

Last updated: Jul 11