question Re: Please advise-Getting this error in Hue - Could not connect to any of [('10.83.35.142', 10000)] (code THRIFTTRANSPORT): TTransportException("Could not connect to any of [('10.83.35.142', 10000)]",) in Support Questions

Hue error: TTransportException("Could not connect to any of [('10.83.35.142', 10000)]",)

vijaypabothu — Wed, 23 Sep 2020 21:01:13 GMT

Please advise-Getting this error in Hue - Could not connect to any of [('10.83.35.142', 10000)] (code THRIFTTRANSPORT): TTransportException("Could not connect to any of [('10.83.35.142', 10000)]",)

Re: Please advise-Getting this error in Hue - Could not connect to any of [('10.83.35.142', 10000)] (code THRIFTTRANSPORT): TTransportException("Could not connect to any of [('10.83.35.142', 10000)]",)

carrossoni — Wed, 23 Sep 2020 18:38:57 GMT

Hi @vijaypabothu ,

First try to Check if the Hive Server is running and if it's on the address 10.83.35.142 with the 10000 port. This message indicates that Hive Server is down or HUE Can't reach the server.

Luiz

Re: Please advise-Getting this error in Hue - Could not connect to any of [('10.83.35.142', 10000)] (code THRIFTTRANSPORT): TTransportException("Could not connect to any of [('10.83.35.142', 10000)]",)

vijaypabothu — Thu, 24 Sep 2020 07:46:47 GMT

Hi @carrossoni ,

You are Right . Hive server 2 is down on Hive on Tez. Please find the Error log below. Please advise how to overcome this issue.

20/09/24 02:39:04 ERROR server.HiveServer2: [shutdown-hook-0]: Error stopping schq
java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid
	at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:308) ~[hive-exec-3.1.3000.7.1.3.0-100.jar:3.1.3000.7.1.3.0-100]
	at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:886) ~[hive-service-3.1.3000.7.1.3.0-100.jar:3.1.3000.7.1.3.0-100]
	at org.apache.hive.service.server.HiveServer2.lambda$init$0(HiveServer2.java:433) ~[hive-service-3.1.3000.7.1.3.0-100.jar:3.1.3000.7.1.3.0-100]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]

Re: Please advise-Getting this error in Hue - Could not connect to any of [('10.83.35.142', 10000)] (code THRIFTTRANSPORT): TTransportException("Could not connect to any of [('10.83.35.142', 10000)]",)

carrossoni — Thu, 24 Sep 2020 14:26:46 GMT

Hi @vijaypabothu ,

This error means that the node is on a invalid state but not the root cause, more debug is needed. Does the ZK logs shows something?

Also I would look at https://community.cloudera.com/t5/Support-Questions/hiveserver2-alway-shut-down/td-p/155439

Regards,

Luiz

Re: Please advise-Getting this error in Hue - Could not connect to any of [('10.83.35.142', 10000)] (code THRIFTTRANSPORT): TTransportException("Could not connect to any of [('10.83.35.142', 10000)]",)

vijaypabothu — Sat, 26 Sep 2020 13:50:54 GMT

Below error in Zookeeper

2020-09-18 14:30:38,331 ERROR org.apache.zookeeper.server.quorum.LearnerHandler: Unexpected exception causing shutdown while sock still open
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:113)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:87)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:118)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:379)
2020-09-18 14:30:38,332 WARN org.apache.zookeeper.server.quorum.LearnerHandler: ******* GOODBYE /64.102.14.234:55960 ********
2020-09-18 14:30:38,337 ERROR org.apache.zookeeper.server.quorum.LearnerHandler: Unexpected exception causing shutdown while sock still open
java.io.IOException: Unreasonable length = 541611092
at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:146)
at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:111)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:87)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:118)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:379)
2020-09-18 14:30:38,338 WARN org.apache.zookeeper.server.quorum.LearnerHandler: ******* GOODBYE /64.102.14.234:56947 ********
2020-09-18 14:31:41,053 INFO org.apache.zookeeper.server.ZooKeeperServer: Submitting global closeSession request for session 0x300261365d7006e

Re: Please advise-Getting this error in Hue - Could not connect to any of [('10.83.35.142', 10000)] (code THRIFTTRANSPORT): TTransportException("Could not connect to any of [('10.83.35.142', 10000)]",)

carrossoni — Sat, 26 Sep 2020 17:51:38 GMT

I would also try to look which application is causing this.

Regards,

Luiz

Re: Please advise-Getting this error in Hue - Could not connect to any of [('10.83.35.142', 10000)] (code THRIFTTRANSPORT): TTransportException("Could not connect to any of [('10.83.35.142', 10000)]",)

vijaypabothu — Tue, 29 Sep 2020 05:53:08 GMT

Hi @carrossoni ,

I followed the same as mentioned in below . But still getting this error and Not able to find the RC . Please advise.

SERVICE=[zookeeper], SERVICE_TYPE=[ZOOKEEPER], LOG_LEVEL=[WARN], SEVERITY=[IMPORTANT]}, content=Cannot open channel to 3 at election address matrix-mlsec-05.cisco.com/10.83.35.145:4181, timestamp=1599765151156} - 1 of 56 failure(s) in last 1831s
java.io.IOException: Error connecting to matrix-mlsec-01.cisco.com/10.83.35.141:7184
at com.cloudera.cmf.event.shaded.org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:292)
at com.cloudera.cmf.event.shaded.org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:206)
at com.cloudera.cmf.event.shaded.org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:141)
at com.cloudera.cmf.event.publish.AvroEventStorePublishProxy.checkSpecificRequestor(AvroEventStorePublishProxy.java:135)
at com.cloudera.cmf.event.publish.AvroEventStorePublishProxy.publishEvent(AvroEventStorePublishProxy.java:210)
at com.cloudera.cmf.event.publish.EventStorePublisherWithRetry$PublishEventTask.run(EventStorePublisherWithRetry.java:242)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at com.cloudera.cmf.event.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
at com.cloudera.cmf.event.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
at com.cloudera.cmf.event.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
at com.cloudera.cmf.event.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at com.cloudera.cmf.event.shaded.org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at com.cloudera.cmf.event.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at com.cloudera.cmf.event.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
... 3 more .

Re: Please advise-Getting this error in Hue - Could not connect to any of [('10.83.35.142', 10000)] (code THRIFTTRANSPORT): TTransportException("Could not connect to any of [('10.83.35.142', 10000)]",)

carrossoni — Fri, 02 Oct 2020 16:09:09 GMT

Hi @vijaypabothu , this looks a network error. can you check with your network team if the ports are open?

Re: Please advise-Getting this error in Hue - Could not connect to any of [('10.83.35.142', 10000)] (code THRIFTTRANSPORT): TTransportException("Could not connect to any of [('10.83.35.142', 10000)]",)

Ant5566 — Fri, 26 Feb 2021 02:29:53 GMT

Hi @carrossoni

I met this problem this days.

The server of hive on tez shut down after it restarted.

2021-02-25 14:20:06,800 INFO org.apache.hadoop.hive.common.ZooKeeperHiveHelper: [shutdown-hook-0]: Server instance removed from ZooKeeper.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopping/Disconnecting tez sessions.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopped tez session pool manager.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Shutting down HiveServer2
2021-02-25 14:20:06,800 ERROR org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Error stopping schq
java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid
at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-203]
at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:887) ~[hive-service-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-203]
at org.apache.hive.service.server.HiveServer2.lambda$init$0(HiveServer2.java:434) ~[hive-service-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-203]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Web UI has stopped
2021-02-25 14:20:06,800 INFO org.apache.hadoop.hive.common.ZooKeeperHiveHelper: [shutdown-hook-0]: Server instance removed from ZooKeeper.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopping/Disconnecting tez sessions.
2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopped tez session pool manager.
2021-02-25 14:20:06,807 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at acdu02datvm0024.acsdmzuat.cc.cmbchina.cn/172.22.5.22
************************************************************/

But I cannot find error message from ZK’s log.

Could you please give some advice?

Re: Please advise-Getting this error in Hue - Could not connect to any of [('10.83.35.142', 10000)] (code THRIFTTRANSPORT): TTransportException("Could not connect to any of [('10.83.35.142', 10000)]",)

carrossoni — Thu, 04 Mar 2021 16:37:23 GMT

Hi @Ant5566

Can you take a look at /opt/cloudera/cm-agent/service/hive/hive.sh specially the TEZ_JARS classpath?

Let me know if the path exists or it's correct, if not maybe it's needed to change to the correct location and them restart the services.

Thanks,

Luiz