Support Questions
Find answers, ask questions, and share your expertise

How to do remote debugging for Apache Hadoop Client e.g (NamenodeWebHdfsMethods.java) module using Intellij.

How to do remote debugging for Apache Hadoop Client e.g (NamenodeWebHdfsMethods.java) module using Intellij.

New Contributor

I am new to Apache Hadoop, so please excuse or apologies me, if my query is silly or this is not the right platform to ask for the same. I know how to do remote debugging using intellij IDE, but not sure how to connect my Hadoop Client (using 2.8.1 binary) by adding JVM parms. My hadoop client is running on some Virtual Machine. I want to do remote debugging so that I can put breakpoints in the code and see the flow of the code. Thanks in advance

3 REPLIES 3

Re: How to do remote debugging for Apache Hadoop Client e.g (NamenodeWebHdfsMethods.java) module using Intellij.

Super Mentor

@Bhushan kumar

In the terminal where you are running the Hadoop Cline you can set the "HADOOP_OPTS" to add the remote debugging option.

Example:

# export HADOOP_OPTS = $HADOOP_OPTS -agentlib:jdwp=transport=dt_socket,address=9999,server=y,suspend=n

# hdfs dfs -ls /

.

Re: How to do remote debugging for Apache Hadoop Client e.g (NamenodeWebHdfsMethods.java) module using Intellij.

New Contributor

@Jay SenSharma
Thank you for your response, I tried this but could not connect remotely. The error on IntelliJ - Unable to run debugger.


echo $HADOOP_OPTS on my hadoop client terminal gives me:

-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9999

Am I missing something?

Re: How to do remote debugging for Apache Hadoop Client e.g (NamenodeWebHdfsMethods.java) module using Intellij.

Explorer

I have the same problem (HDP 2.6, vmware).

Reading a lot of related answers I believe that the problem is port forwarding and/or docker configuration.

What has to be done to enable acess to a specific port (9999 in the example above) of the sandbox?

(Probably the same is done for scp - the sandbox is accessed via port 2222 though there is nothing listening at port 2222 in the sandbox.)

Unfortunately the most useful commands for examining this problem are not installed in the docker machine (e.g. ifconfig, netstat, ...)