Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hadoop h2o attempting to use invalid “Possible callback IP address”

Highlighted

Hadoop h2o attempting to use invalid “Possible callback IP address”

Contributor

Attempting to use hadoop h2o for hdp 3.1 and when running any command to try to start the h2o cluster and getting errors that appear to be due to memory, but looking at the output a bit more, I see that h2o appears to be checking an IP that does not actually exist on the hadoop cluster for resources. Eg.

➜  h2o-3.26.0.2-hdp3.1 hadoop jar h2odriver.jar -nodes 3 -mapperXmx 1g -output /home/ml1/hdfsOutputDir
Determining driver host interface for mapper->driver callback...
    [Possible callback IP address: 192.168.122.1]
    [Possible callback IP address: 172.18.4.49]
    [Possible callback IP address: 127.0.0.1]
Using mapper->driver callback IP address and port: 172.18.4.49:40771
(You can override these with -driverif and -driverport/-driverportrange and/or specify external IP using -extdriverif.)
Memory Settings:
    mapreduce.map.java.opts:     -Xms1g -Xmx1g -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Dlog4j.defaultInitOverride=true
    Extra memory percent:        10
    mapreduce.map.memory.mb:     1126
Hive driver not present, not generating token.
19/08/08 09:37:52 INFO client.RMProxy: Connecting to ResourceManager at hw01.co.local/172.18.4.46:8050
19/08/08 09:37:52 INFO client.AHSProxy: Connecting to Application History server at hw02.co.local/172.18.4.47:10200
19/08/08 09:37:53 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/ml1/.staging/job_1565057088651_0014
19/08/08 09:37:54 INFO mapreduce.JobSubmitter: number of splits:3
19/08/08 09:37:54 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1565057088651_0014
19/08/08 09:37:54 INFO mapreduce.JobSubmitter: Executing with tokens: []
19/08/08 09:37:54 INFO conf.Configuration: found resource resource-types.xml at file:/etc/hadoop/3.1.0.0-78/0/resource-types.xml
19/08/08 09:37:54 INFO impl.YarnClientImpl: Submitted application application_1565057088651_0014
19/08/08 09:37:54 INFO mapreduce.Job: The url to track the job: http://HW01.co.local:8088/proxy/application_1565057088651_0014/
Job name 'H2O_62688' submitted
JobTracker job ID is 'job_1565057088651_0014'
For YARN users, logs command is 'yarn logs -applicationId application_1565057088651_0014'
Waiting for H2O cluster to come up...
19/08/08 09:38:11 INFO client.RMProxy: Connecting to ResourceManager at hw01.co.local/172.18.4.46:8050
19/08/08 09:38:11 INFO client.AHSProxy: Connecting to Application History server at hw02.co.local/172.18.4.47:10200

----- YARN cluster metrics -----
Number of YARN worker nodes: 4

----- Nodes -----
Node: http://HW03.co.local:8042 Rack: /default-rack, RUNNING, 0 containers used, 0.0 / 15.0 GB used, 0 / 3 vcores used
Node: http://HW04.co.local:8042 Rack: /default-rack, RUNNING, 0 containers used, 0.0 / 15.0 GB used, 0 / 3 vcores used
Node: http://hw05.co.local:8042 Rack: /default-rack, RUNNING, 0 containers used, 0.0 / 15.0 GB used, 0 / 3 vcores used
Node: http://HW02.co.local:8042 Rack: /default-rack, RUNNING, 1 containers used, 5.0 / 15.0 GB used, 1 / 3 vcores used

----- Queues -----
Queue name:            default
    Queue state:       RUNNING
    Current capacity:  0.08
    Capacity:          1.00
    Maximum capacity:  1.00
    Application count: 1
    ----- Applications in this queue -----
    Application ID:                  application_1565057088651_0014 (H2O_62688)
        Started:                     ml1 (Thu Aug 08 09:37:54 HST 2019)
        Application state:           FINISHED
        Tracking URL:                http://HW01.co.local:8088/proxy/application_1565057088651_0014/
        Queue name:                  default
        Used/Reserved containers:    1 / 0
        Needed/Used/Reserved memory: 5.0 GB / 5.0 GB / 0.0 GB
        Needed/Used/Reserved vcores: 1 / 1 / 0

Queue 'default' approximate utilization: 5.0 / 60.0 GB used, 1 / 12 vcores used

Note the

[Possible callback IP address: 192.168.122.1]

Yet, my host IPs on the cluster are as shown... 110284-1565294485230.png

Is this normal? Why might this be happening? Any debugging suggestions or fixes?