Difficulty Producing from localhost to Kafka in HDP Sandbox 2.5

[#1]

We are trying to produce to Kafka in the HDP 2.5 sandbox from my host machine, and I can't seem to get it to work. I am doing this from a C# service running in Visual Studio. It seems that the connection isn't being made between the host machine and the Kafka Borker host. We had this working before with HDP 2.4, but now that it is in a docker container, there seems to be another layer of complexity that is screwing us up. Here is what we have tried so far:

- Added 6667 to 6667 between host and VM. Killed the sandbox container, removed it, added 6667:6667 between VM and Container to /root/start_scripts/start_sandbox.sh, restarted the VM, saw the port forwarding was added with docker ps. We then created the Kafka topic in the container, (tested produce and consume within container; which worked ok) and then tried to produce from the host machine with the following uri: http://127.0.0.1:6667. Doing it this way, I don't get an error message, it just doesn't show up on the consumer. Not sure what is happening there.

- We also tried switching the system over to Host Only, but that seemed to screw with Ambari somehow, that is a different issue.

- We also tried importing the image in our own OS environment in VBox, that didn't work either

I don't mean post this question without a lot of detailed error messages and command line output, but there were just so many ways we have tried this that I wanted to see if anyone could help us hone in on the correct way, and then I can post the output from that. I suspect that the problem may have something to do with the config for Kafka using a hostname of 'sandbox.hortonworks.com' but i'm not sure.

Any help would be GREATLY appreciated, we've really been banging our heads against the wall. We must be missing something.

Many thanks,

T

[#2]

@T D,

You are running the C# and virtualbox with the sandbox all on the same windows host, right?

From a Windows CMD prompt, what do you get when you do the following?

netstat -ano | findstr "6667 8080"

I would expect lines that say "listening" for both ports 6667 and 8080, and that the last number (Process ID) would be the same. I think that would verify that the Virtualbox is listening and forwarding the ports as configured.

Also, what happens if you connect to http://127.0.0.1:6667 from a browser on the same host? You should at least get a connection and not a timeout.

John

[#3]

John,

Thank you for the response. Yes, I am running the C# service and VirtualBox on the same windows host.

Here is what I get with those two experiments.

netstat -ano | findstr "6667 8080"
  TCP    127.0.0.1:6667         0.0.0.0:0              LISTENING       3344
  TCP    127.0.0.1:8080         0.0.0.0:0              LISTENING       3344
  TCP    127.0.0.1:18080        0.0.0.0:0              LISTENING       3344
  TCP    [::1]:62738            [::1]:49154            ESTABLISHED     28080

and for the browser

This site can’t be reached
The webpage at <a href="http://127.0.0.1:6667/">http://127.0.0.1:6667/</a> might be temporarily down or it may have moved permanently to a new web address.
ERR_UNSAFE_PORT

@jwhitmore

[#4]

The browser test did not work since Chrome blocked 6667 as unsafe. If you have telnet or putty installed, try to telnet to 127.0.0.1 port 6667 and see if it connects. We're just looking to see if we have the connectivity, we're not trying to produce a record on the topic.

[#5]

Hmmm, when I try to telnet from host machine at 127.0.01:6667 I get:

Connection closed by remote host

When I try to ssh from host machine at 127.0.0.1:6667 I get:

server unexpectedly closed network connection

----------------------------------

Also tried these experiments from the virtual box with the container running and Kafka started

telnet from virtual box

telnet 127.0.0.1 6667
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.

and

netstat -tan | grep :6667
tcp6       0      0 :::6667                 :::*                    LISTEN
tcp6       0      0 :::8080                 :::*                    LISTEN
tcp6       0      0 :::2222                 :::*                    LISTEN

So it looks like the port forwarding is working between host and VM, and VM and container, but not between host machine and container? Of note, I still get the output between the VM and the Container even if the services at those ports are not running, however, the connection is automatically closed as opposed to the CLI waiting for me to input Ctrl-C. So I guess what this shows is that the port forwarding is working, however, I'm not sure it elucidates where the error with Producing to Kafka is, or why the host machine can't make the same connections as the VM with regards to the container.

The Windows service configuration is:

Kafka uri: http://127.0.0.1:6667

and when I run the service on my host machine and a consumer in the container, this is what I see:

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic test-topic {metadata.broker.list=sandbox.hortonworks.com:6667, request.timeout.ms=30000, client.id=console-consumer-59653, security.protocol=PLAINTEXT}

^CProcessed a total of 0 messages

But no errors on the host machine...

[#6]

@T D,

ssh into the sandbox and start a loop that will show connections to the Kafka service. Ctrl-C will stop the loop. Otherwise while it runs, you can see if new connections are made when you start your service and when you do the telnet test.

while true; do (netstat -tan | grep -vi listen | grep 6667); sleep 1; don

[#7]

@T D,

Since you get a connection that does not immediately close when you connect from the VM to the container, I think the issue is still in getting the port forwarded from the host to the VM through Virtualbox. It looks like it is listening, but it is not working. When you added 6667 between the host and VM, did you stop and restart the VM after? I don't know if it's needed, but I really think that's where the issue is, since you get a connection from the VM to the container. If you stop the VM, I think it should stop listening on 6667 (and all of the other ports). Maybe even restarting VirtualBox once the VM is down so you can validate that none of the ports are forwarded anymore. Then start it all back up again and see if it gets connected correctly.

Let me know, I'm really curious about it!

John

[#8 · Accepted solution]

Well, we have it working now.

@jwhitmore

What we ended up doing is creating a VM in virtualbox with a hostonly network adapter. Added the hostonly IP addr with mappings to the 'sandbox.hortonworks.com' FQDN to the hosts file in Windows for the local machine, added the port forwarding/binding to the container at 6667 - 6667, and it started working.

Short answer, but overall it was the lack of mapping from the ip addr to the FQDN.

[#9]

@T D The original problem is related to the fact the 6667 port is not mapper in the hosting VM. As you a running your cluster in docker container, the hosting VM should perform its launch with proper port mappings. In order to enable this port connect to your VM using ssh -p 2122 root@localhost and perform the following steps:

1. Disable sandbox service:

    systemctl disable sandbox.service  

    init 6

2. Alter the launch script:

    vi /root/start_scripts/start_sandbox.sh

3. Add this line after the last -p operation and save the script:

    -p 6667:6667 \

4. Delete existing container

    docker rm sandbox

5. Enable sandbox service:

    systemctl enable sandbox.service    

    init 6

After this you will be able to check that the port is opened using this command:

    docker ps | grep 6667

Now add port forwarding in the NAT configuration of your VM: kafka=>TCP=>127.0.0.1=>6667=> =>6667

After this you will be able to connect to the kafka broker on your sandbox.

[#10]

Step 4 from the instructions above will lead to data loss if you was using the VM, so please backup your VM first!

I am supporting training VM for my organization, where users have imported some data and installed apps.

Now we have to restore the VM, because the default image does not contained user data and administration changes.