Hue error: TTransportException("Could not connect to any of [('10.83.35.142', 10000)]",)

[#1]

Please advise-Getting this error in Hue - Could not connect to any of [('10.83.35.142', 10000)] (code THRIFTTRANSPORT): TTransportException("Could not connect to any of [('10.83.35.142', 10000)]",)

[#2]

Hi @vijaypabothu ,

First try to Check if the  Hive Server is running and if it's on the address 10.83.35.142 with the 10000 port. This message indicates that Hive Server is down or HUE Can't reach the server.

Luiz

[#3]

Hi @carrossoni ,

You are Right . Hive server 2 is down on Hive on Tez. Please find the Error log below. Please advise how to overcome this issue.

20/09/24 02:39:04 ERROR server.HiveServer2: [shutdown-hook-0]: Error stopping schq
java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid
	at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:308) ~[hive-exec-3.1.3000.7.1.3.0-100.jar:3.1.3000.7.1.3.0-100]
	at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:886) ~[hive-service-3.1.3000.7.1.3.0-100.jar:3.1.3000.7.1.3.0-100]
	at org.apache.hive.service.server.HiveServer2.lambda$init$0(HiveServer2.java:433) ~[hive-service-3.1.3000.7.1.3.0-100.jar:3.1.3000.7.1.3.0-100]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]

s 

[#4]

Hi @vijaypabothu ,

This error means that the node is on a invalid state but not the root cause, more debug is needed. Does the ZK logs shows something?

Also I would look at https://community.cloudera.com/t5/Support-Questions/hiveserver2-alway-shut-down/td-p/155439

Regards,

Luiz

[#5]

Below error in Zookeeper 

2020-09-18 14:30:38,331 ERROR org.apache.zookeeper.server.quorum.LearnerHandler: Unexpected exception causing shutdown while sock still open
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:113)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:87)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:118)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:379)
2020-09-18 14:30:38,332 WARN org.apache.zookeeper.server.quorum.LearnerHandler: ******* GOODBYE /64.102.14.234:55960 ********
2020-09-18 14:30:38,337 ERROR org.apache.zookeeper.server.quorum.LearnerHandler: Unexpected exception causing shutdown while sock still open
java.io.IOException: Unreasonable length = 541611092
at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:146)
at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:111)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:87)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:118)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:379)
2020-09-18 14:30:38,338 WARN org.apache.zookeeper.server.quorum.LearnerHandler: ******* GOODBYE /64.102.14.234:56947 ********
2020-09-18 14:31:41,053 INFO org.apache.zookeeper.server.ZooKeeperServer: Submitting global closeSession request for session 0x300261365d7006e

[#6]

This seems related to https://community.cloudera.com/t5/Support-Questions/zookeeper-error-Unexpected-exception-causing-shu...

I would also try to look which application is causing this.

Regards,

Luiz

[#7]

Hi @carrossoni ,

I followed the same as mentioned in below . But still getting this error and Not able to find the RC . Please advise.

SERVICE=[zookeeper], SERVICE_TYPE=[ZOOKEEPER], LOG_LEVEL=[WARN], SEVERITY=[IMPORTANT]}, content=Cannot open channel to 3 at election address matrix-mlsec-05.cisco.com/10.83.35.145:4181, timestamp=1599765151156} - 1 of 56 failure(s) in last 1831s
java.io.IOException: Error connecting to matrix-mlsec-01.cisco.com/10.83.35.141:7184
at com.cloudera.cmf.event.shaded.org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:292)
at com.cloudera.cmf.event.shaded.org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:206)
at com.cloudera.cmf.event.shaded.org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:141)
at com.cloudera.cmf.event.publish.AvroEventStorePublishProxy.checkSpecificRequestor(AvroEventStorePublishProxy.java:135)
at com.cloudera.cmf.event.publish.AvroEventStorePublishProxy.publishEvent(AvroEventStorePublishProxy.java:210)
at com.cloudera.cmf.event.publish.EventStorePublisherWithRetry$PublishEventTask.run(EventStorePublisherWithRetry.java:242)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at com.cloudera.cmf.event.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at com.cloudera.cmf.event.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at com.cloudera.cmf.event.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at com.cloudera.cmf.event.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at com.cloudera.cmf.event.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at com.cloudera.cmf.event.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at com.cloudera.cmf.event.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more .  

[#8]

Hi @vijaypabothu , this looks a network error. can you check with your network team if the ports are open?

[#9]

Hi @carrossoni  

I met this problem this days.

The server of hive on tez shut down after it restarted. 

2021-02-25 14:20:06,800 INFO org.apache.hadoop.hive.common.ZooKeeperHiveHelper: [shutdown-hook-0]: Server instance removed from ZooKeeper.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopping/Disconnecting tez sessions.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopped tez session pool manager.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Shutting down HiveServer2
2021-02-25 14:20:06,800 ERROR org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Error stopping schq
java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid
at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-203]
at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:887) ~[hive-service-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-203]
at org.apache.hive.service.server.HiveServer2.lambda$init$0(HiveServer2.java:434) ~[hive-service-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-203]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Web UI has stopped
2021-02-25 14:20:06,800 INFO org.apache.hadoop.hive.common.ZooKeeperHiveHelper: [shutdown-hook-0]: Server instance removed from ZooKeeper.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopping/Disconnecting tez sessions.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopped tez session pool manager.
2021-02-25 14:20:06,807 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at acdu02datvm0024.acsdmzuat.cc.cmbchina.cn/172.22.5.22
************************************************************/

But I cannot find error message from ZK’s log.

Could you please give some advice?

[#10]

Hi @Ant5566 

Can you take a look at /opt/cloudera/cm-agent/service/hive/hive.sh specially the TEZ_JARS classpath?

Let me know if the path exists or it's correct, if not maybe it's needed to change to the correct location and them restart the services.

Thanks,

Luiz