Hive connecting to node that does not exist

[#1]

Hi,

I am encountering a network issue with Hive where is trying to connect to a node that does not exist when trying to do a SELECT * query from either Hue or Beeline CLI:

Error while compiling statement: FAILED: SemanticException Unable to determine if hdfs://ip-172-19-36-94.ap-southeast-2.compute.internal:8020/user/hive/warehouse/allowed is encrypted: org.apache.hadoop.hive.ql.metadata.HiveException: java.net.NoRouteToHostException: No Route to Host from ip-172-19-36-68/172.19.36.68 to ip-172-19-36-94.ap-southeast-2.compute.internal:8020 failed on socket timeout exception: java.net.NoRouteToHostException: No route to host; For more details see: http://wiki.apache.org/hadoop/NoRouteToHost

The node that is specified by ip-172-19-36-94.ap-southeast-2.compute.internal does not exist in the cluster or anywhere in our environment.

I have been looking for the hive-site.xml and core-site.xml but there are no references to this DNS entry.

I was just wondering if there was anywhere else Hive could reference this DNS entry? Could it reference DNSs of prior existing Hadoop nodes from the Hive meta-store for example?

Cheers,

David

[#2 · Accepted solution]

Hi @d_liu_ 

The error shows that  problem with the network connectivity between the Hive server (ip-172-19-36-68) and the HDFS namenode (ip-172-19-36-94.ap-southeast-2.compute.internal) on port 8020.

So as per your comment above HDFS node does not exist in your cluster right? May i know how you are running this query like from beeline or Hue or from any third party tool?

Search the above host in all the configuration files(hive-site.xml,hive-env.sh,hive.metastore.uris etc..) 

Try to search the above host name at client side from where you are running the query and there is a chances if you have any custom scripts or config files causing the issue, so check if you have any custom scripts to run the jobs.

Regards,

Chethan YM

[#3 · Accepted solution]

Hi @d_liu_ 

The error shows that  problem with the network connectivity between the Hive server (ip-172-19-36-68) and the HDFS namenode (ip-172-19-36-94.ap-southeast-2.compute.internal) on port 8020.

So as per your comment above HDFS node does not exist in your cluster right? May i know how you are running this query like from beeline or Hue or from any third party tool?

Search the above host in all the configuration files(hive-site.xml,hive-env.sh,hive.metastore.uris etc..) 

Try to search the above host name at client side from where you are running the query and there is a chances if you have any custom scripts or config files causing the issue, so check if you have any custom scripts to run the jobs.

Regards,

Chethan YM

[#4 · Accepted solution]

Hi @ChethanYM,

I did a pg_dump and grep'd the old namenode DNS on my hive metastore and found the table locations in there referenced the old DNS. Setting the table location to the new namenode with 'alter table <table> set location <new location>' in Hive fixed the issue.

Thanks for your help!

David