Remote debug HiveServer2

ThomasLarsson — Thu, 07 Jan 2016 18:10:52 GMT

Hello,

I'm running HDP-2.2.6.0-2800 and am trying to remote debug the HiveServer2 process.

My problem is that I don't get the line numbers to match when setting breakpoints in the code and stepping through the code.

This usually happens when the source code on my local machine from which I am running the debugger does not match the compiled code running on the server I'm debugging.

Has anyone got this to work? In that case how?

Here are my instructions to reproduce this problem using the HDP-2.2.4 sandbox (there is no 2.2.6 sandbox afaik):

======================================

1. Create a new HDP-2.2.4 sandbox instance.

2. Add the following snippet to Advanced hive-env -> hive-env template

# Enable remote debugging of hiveserver2.
if [ "$SERVICE" = "hiveserver2" ]; then
  export HADOOP_OPTS="$HADOOP_OPTS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005"
fi

3. Modify the /usr/hdp/2.2.4.2-2/hive/bin/hive.distro file by replacing the following section

# Make sure we're using a compatible version of Hadoop
if [ "x$HADOOP_VERSION" == "x" ]; then
  HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
fi

With this: (When you start the hiveserver with the agent flags, it print an additional line to stdout which confuses this awk script)

# Make sure we're using a compatible version of Hadoop
if [ "$SERVICE" == 'hiveserver2' ]; then
  if [ "x$HADOOP_VERSION" == "x" ]; then
    HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 2) {print $2;}}');
  fi
else
  if [ "x$HADOOP_VERSION" == "x" ]; then
    HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
  fi
fi

4. Clone the hive git repo and change to the branch-0.14 branch.

5. Create a remote debug connection to the hiveserver2 java process

(i'm using IntelliJ IDEA to setup the project)

6. Set a breakpoint in org.apache.hive.service.cli.session.SessionManager, line 268, i.e.

if (withImpersonation) {
	HiveSessionImplwithUGI sessionWithUGI = new HiveSessionImplwithUGI(protocol, username, password,hiveConf, ipAddress, delegationToken);
	session = HiveSessionProxy.getProxy(sessionWithUGI, sessionWithUGI.getSessionUgi());
	sessionWithUGI.setProxySession(session);
} else {
      session = new HiveSessionImpl(protocol, username, password, hiveConf, ipAddress);
}

session.setSessionManager(this);
session.setOperationManager(operationManager); // <--- Set breakpoint here for example
try {
      session.initialize(sessionConf);
      if (isOperationLogEnabled) {
      	session.setOperationLogSessionDir(operationLogRootDir);
      }
      session.open();
} catch (Exception e) {
      throw new HiveSQLException("Failed to open new session", e);
}

7. Start a new hive session, I'm using beeline

8. See that the hiveserver2 execution is halted at the breakpoint.

9. Try to "Step into" the session.setOperationManager method, and you actually end up in

org.apache.hive.service.cli.session.HiveSessionImpl.getSessionHandle()

An obvious line mismatch here as you can see.

Perhaps I am missing something here.

Grateful for any tips.

/Thomas

Re: Remote debug HiveServer2

ThomasLarsson — Thu, 07 Jan 2016 20:46:25 GMT

I found the source code, here:

https://github.com/hortonworks/hive-release/releases

question Remote debug HiveServer2 in Archives of Support Questions (Read Only)

Remote debug HiveServer2

Re: Remote debug HiveServer2