Member since
10-01-2016
156
Posts
8
Kudos Received
6
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 7520 | 04-04-2019 09:41 PM | |
| 2994 | 06-04-2018 08:34 AM | |
| 1402 | 05-23-2018 01:03 PM | |
| 2863 | 05-21-2018 07:12 AM | |
| 1750 | 05-08-2018 10:48 AM |
06-04-2018
08:34 AM
Initially SupportKB recomended solution doesn't work. But then I have realized my amazing mistake: echo -e "[https]\nverify=disable" > /etc/python/cert-verification.cfg command somehow generated: [https]nverify=disable which should be: [https]
verify=disable After fixing this wierd mistake LLAP successfully started. Thanks @Geoffrey Shelton Okot for your amazing response.
... View more
06-04-2018
08:14 AM
Thank you @Geoffrey Shelton Okot. slider 0.92 and python 2.7.5.
... View more
06-04-2018
07:13 AM
I use HDP 2.6.3, no kerberos, no ssl. Hive LLAP doesn't start. I tried many suggestions including llap_heap_size and hive.llap.daemon.yarn.container.mb change, JAVA_HOME change etc. but nothing worked. ERRORs from YARN LLAP appllication logs: ERROR 2018-06-04 09:32:30,173 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:32:30,173 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:32:30,175 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:32:40,185 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:32:40,290 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:32:40,290 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:32:40,292 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:32:50,298 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:32:50,408 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:32:50,408 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:32:50,410 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:33:00,422 connection.py:573 - Connecting to analitik02.gelbim.gov.tr:2181
INFO 2018-06-04 09:33:00,424 client.py:439 - Zookeeper connection established, state: CONNECTED
INFO 2018-06-04 09:33:00,428 connection.py:540 - Closing connection to analitik02.gelbim.gov.tr:2181
INFO 2018-06-04 09:33:00,428 client.py:443 - Zookeeper session lost, state: CLOSED
INFO 2018-06-04 09:33:00,428 Registry.py:69 - AM Host = hadooptest7.datalonga.com, AM Secured Port = 40573, ping port = 43336
INFO 2018-06-04 09:33:00,429 main.py:292 - Connecting to the server at: https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:33:00,429 NetUtil.py:76 - DEBUG: Trying to connect to the server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:33:00,429 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:33:00,531 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:33:00,532 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:33:00,534 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:33:10,537 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:33:10,633 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:33:10,634 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:33:10,636 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:33:20,636 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:33:20,735 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:33:20,736 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:33:20,738 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:33:30,743 connection.py:573 - Connecting to analitik01.gelbim.gov.tr:2181
INFO 2018-06-04 09:33:30,745 client.py:439 - Zookeeper connection established, state: CONNECTED
INFO 2018-06-04 09:33:30,748 connection.py:540 - Closing connection to analitik01.gelbim.gov.tr:2181
INFO 2018-06-04 09:33:30,748 client.py:443 - Zookeeper session lost, state: CLOSED
INFO 2018-06-04 09:33:30,749 Registry.py:69 - AM Host = hadooptest7.datalonga.com, AM Secured Port = 40573, ping port = 43336
INFO 2018-06-04 09:33:30,749 main.py:292 - Connecting to the server at: https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:33:30,749 NetUtil.py:76 - DEBUG: Trying to connect to the server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:33:30,749 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:33:30,848 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:33:30,849 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:33:30,851 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:33:40,861 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:33:40,964 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:33:40,964 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:33:40,966 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:33:50,976 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:33:51,076 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:33:51,076 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:33:51,078 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:34:01,081 connection.py:573 - Connecting to analitik03.gelbim.gov.tr:2181
INFO 2018-06-04 09:34:01,084 client.py:439 - Zookeeper connection established, state: CONNECTED
INFO 2018-06-04 09:34:01,087 connection.py:540 - Closing connection to analitik03.gelbim.gov.tr:2181
INFO 2018-06-04 09:34:01,087 client.py:443 - Zookeeper session lost, state: CLOSED
INFO 2018-06-04 09:34:01,088 Registry.py:69 - AM Host = hadooptest7.datalonga.com, AM Secured Port = 40573, ping port = 43336
INFO 2018-06-04 09:34:01,088 main.py:292 - Connecting to the server at: https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:34:01,088 NetUtil.py:76 - DEBUG: Trying to connect to the server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:34:01,089 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:34:01,186 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:34:01,186 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:34:01,188 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:34:11,197 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:34:11,286 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:34:11,286 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:34:11,288 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:34:21,298 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:34:21,400 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:34:21,400 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:34:21,402 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:34:31,414 connection.py:573 - Connecting to analitik02.gelbim.gov.tr:2181
INFO 2018-06-04 09:34:31,417 client.py:439 - Zookeeper connection established, state: CONNECTED
INFO 2018-06-04 09:34:31,420 connection.py:540 - Closing connection to analitik02.gelbim.gov.tr:2181
INFO 2018-06-04 09:34:31,420 client.py:443 - Zookeeper session lost, state: CLOSED
INFO 2018-06-04 09:34:31,420 Registry.py:69 - AM Host = hadooptest7.datalonga.com, AM Secured Port = 40573, ping port = 43336
INFO 2018-06-04 09:34:31,421 main.py:292 - Connecting to the server at: https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:34:31,421 NetUtil.py:76 - DEBUG: Trying to connect to the server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:34:31,421 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:34:31,520 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:34:31,521 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:34:31,523 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:34:41,533 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:34:41,634 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:34:41,634 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:34:41,636 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:34:51,647 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:34:51,745 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:34:51,745 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:34:51,747 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:35:01,759 connection.py:573 - Connecting to analitik01.gelbim.gov.tr:2181
INFO 2018-06-04 09:35:01,761 client.py:439 - Zookeeper connection established, state: CONNECTED
INFO 2018-06-04 09:35:01,765 connection.py:540 - Closing connection to analitik01.gelbim.gov.tr:2181
INFO 2018-06-04 09:35:01,765 client.py:443 - Zookeeper session lost, state: CLOSED
INFO 2018-06-04 09:35:01,765 Registry.py:69 - AM Host = hadooptest7.datalonga.com, AM Secured Port = 40573, ping port = 43336
INFO 2018-06-04 09:35:01,766 main.py:292 - Connecting to the server at: https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:35:01,766 NetUtil.py:76 - DEBUG: Trying to connect to the server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
INFO 2018-06-04 09:35:01,766 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:35:01,865 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:35:01,865 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:35:01,867 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:35:11,877 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:35:11,979 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:35:11,979 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:35:11,981 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
INFO 2018-06-04 09:35:21,991 NetUtil.py:40 - Connecting to the following url https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/
ERROR 2018-06-04 09:35:22,093 NetUtil.py:61 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
ERROR 2018-06-04 09:35:22,093 NetUtil.py:63 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
INFO 2018-06-04 09:35:22,094 NetUtil.py:85 - Server at https://hadooptest7.datalonga.com:43336/ws/v1/slider/agents/ is not reachable, sleeping for 10 seconds...
On Ambari interface: 2018-06-04 09:56:26,643 - LLAP app 'llap0' deployment unsuccessful.
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 626, in <module>
HiveServerInteractive().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute
method(env)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 978, in restart
self.start(env, upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py", line 119, in start
raise Fail("Skipping START of Hive Server Interactive since LLAP app couldn't be STARTED.")
resource_management.core.exceptions.Fail: Skipping START of Hive Server Interactive since LLAP app couldn't be STARTED. SupportKB handles the same issue and asserts that he has solved the problem which is not.
... View more
Labels:
- Labels:
-
Apache Hive
05-28-2018
08:53 AM
@Geoffrey Shelton Okot Yes it did. But it stopped saturday morning again. Some of logs: - org.apache.hadoop.hbase.DoNotRetryIOException: hconnection-0x55af174a closed
- ERROR [phoenix-update-statistics-3] stats.StatisticsScanner: Failed to update statistics table!
org.apache.hadoop.hbase.DoNotRetryIOException: hconnection-0x10e9f278 closed
- ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
- java.io.IOException: Connection reset by peer
- org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
... View more
05-24-2018
11:01 AM
Thank you very much @Geoffrey Shelton Okot for your time. I have followed your suggestion step by step except Atlas. I don't have Atlas, even so, I have dropped the 'atlas_titan' table in HBase. I will monitor RegionServers behaviours from now on. I will notify the results in here.
... View more
05-23-2018
01:03 PM
I have solved it. It was a connection problem as logs stated. I had to modify /etc/hosts file. servers have two network card. I switched ip-names of two card.
... View more
05-23-2018
12:07 PM
I have added two more hosts as datanodes via Ambari. Installation is completed successfully. When I return back Ambari dashboard tab I see DataNodes Live: 8/10 I have checked HDFS and seen Installed DataNodes 10/10. There is no apparent error. Screeshots are below. I can see cpu cores and memory but can't disks. Total hdfs disk space didn't change. How to solve this? /var/log/hadoop/hdfs/hadoop-hdfs-datanode-hadooptest12.log 2018-05-23 15:20:28,176 WARN datanode.DataNode (BPServiceActor.java:retrieveNamespaceInfo(227)) - Problem connecting to server: hadooptest12.datalonga.com/StandByNN_IP:8020
2018-05-23 15:20:28,176 WARN datanode.DataNode (BPServiceActor.java:retrieveNamespaceInfo(227)) - Problem connecting to server: hadooptest13.datalonga.com/ActiveNN_IP:8020
datanodes-live.png datanodes-dead.png
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
05-23-2018
06:38 AM
Thanks @schhabra The region server logs are below: 2018-05-23 07:42:13,313 WARN [RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=16020] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=hadooptest1.datalonga.com:2181,hadooptest2.datalonga.com:2181,hadooptest3.datalonga.com:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/meta-region-server
2018-05-23 07:42:13,313 ERROR [RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=16020] zookeeper.RecoverableZooKeeper: ZooKeeper getData failed after 7 attempts
2018-05-23 07:42:13,313 WARN [RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=16020] zookeeper.ZKUtil: hconnection-0x42873e880x0, quorum=hadooptest1.datalonga.com:2181,hadooptest2.datalonga.com:2181,hadooptest3.datalonga.com:2181, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/meta-region-server
2018-05-23 07:42:13,314 ERROR [RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=16020] zookeeper.ZooKeeperWatcher: hconnection-0x42873e880x0, quorum=hadooptest1.datalonga.com:2181,hadooptest2.datalonga.com:2181,hadooptest3.datalonga.com:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/meta-region-server
2018-05-23 07:42:13,315 ERROR [RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=16020] write.KillServerOnFailurePolicy: Could not update the index table, killing server region because couldn't write to an index table
org.apache.phoenix.hbase.index.exception.SingleIndexWriteFailureException: Failed 4 actions: Table 'CITY_I' was not found, got: ATLAS_ENTITY_AUDIT_EVENTS.: 4 times,
at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:168)
at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:132)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 4 actions: Table 'CITY_I' was not found, got: ATLAS_ENTITY_AUDIT_EVENTS.: 4 times,
at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:235)
at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:215)
at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1625)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:913)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:927)
at org.apache.phoenix.execute.DelegateHTable.batch(DelegateHTable.java:94)
at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:164)
... 5 more
2018-05-23 07:42:13,315 FATAL [RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=16020] regionserver.HRegionServer: ABORTING region server hadooptest8.datalonga.com,16020,1526882638915: Could not update the index table, killing server region because couldn't write to an index table
org.apache.phoenix.hbase.index.exception.SingleIndexWriteFailureException: Failed 4 actions: Table 'CITY_I' was not found, got: ATLAS_ENTITY_AUDIT_EVENTS.: 4 times,
at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:168)
at org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter$1.call(ParallelWriterIndexCommitter.java:132)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 4 actions: Table 'CITY_I' was not found, got: ATLAS_ENTITY_AUDIT_EVENTS.: 4 times,
2018-05-23 07:42:13,315 FATAL [RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=16020] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.phoenix.coprocessor.ScanRegionObserver, org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver, org.apache.phoenix.hbase.index.Indexer, org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver, org.apache.phoenix.coprocessor.ServerCachingEndpointImpl, org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint]
2018-05-23 07:42:13,327 INFO [RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=16020] regionserver.HRegionServer: Dump of metrics as JSON on abort:
2018-05-23 07:42:13,330 INFO [RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=16020] util.IndexManagementUtil: Rethrowing org.apache.hadoop.hbase.DoNotRetryIOException: Failed 4 actions: Table 'CITY_I' was not found, got: ATLAS_ENTITY_AUDIT_EVENTS.: 4 times,
2018-05-23 07:42:13,331 ERROR [RpcServer.FifoWFPBQ.default.handler=0,queue=0,port=16020] coprocessor.UngroupedAggregateRegionObserver: IOException during rebuilding: org.apache.hadoop.hbase.DoNotRetryIOException: Failed 4 actions: Table 'CITY_I' was not found, got: ATLAS_ENTITY_AUDIT_EVENTS.: 4 times,
at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:77)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 4 actions: Table 'CITY_I' was not found, got: ATLAS_ENTITY_AUDIT_EVENTS.: 4 times,
at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:235)
There is no zookeeper log.
... View more
05-22-2018
02:57 PM
Is it normal that AMS randomly restart itself? Is it workaround or permenant solution? I experience same issue.
... View more
05-21-2018
07:12 AM
I have solved it with the help of this article Working shiro_ini_content: # Sample LDAP configuration, for Active Directory user Authentication, currently tested for single Realm
[main]
ldapRealm=org.apache.zeppelin.realm.LdapRealm
ldapRealm.contextFactory.systemUsername=cn=hadoop_srv,ou=hadoop,dc=datalonga,dc=com
ldapRealm.contextFactory.systemPassword=hadoop_srv_password
ldapRealm.contextFactory.authenticationMechanism=simple
ldapRealm.contextFactory.url=ldap://datalonga.ldap:389
# Ability to set ldap paging Size if needed; default is 100
ldapRealm.pagingSize=200
ldapRealm.authorizationEnabled=true
ldapRealm.searchBase=OU=hadoop,dc=datalonga,dc=com
ldapRealm.userSearchBase=dc=datalonga,dc=com
ldapRealm.groupSearchBase=OU=hadoop,dc=datalonga,dc=com
ldapRealm.userObjectClass=person
ldapRealm.groupObjectClass=group
ldapRealm.userSearchAttributeName = sAMAccountName
# Set search scopes for user and group. Values: subtree (default), onelevel, object
ldapRealm.userSearchScope = subtree
ldapRealm.groupSearchScope = subtree
ldapRealm.userSearchFilter=(&(objectclass=person)(sAMAccountName={0}))
ldapRealm.memberAttribute=member
# Format to parse & search group member values in 'memberAttribute'
ldapRealm.memberAttributeValueTemplate=CN={0},OU=hadoop,dc=datalonga,dc=com
# No need to give userDnTemplate if memberAttributeValueTemplate is provided
#ldapRealm.userDnTemplate=
# Map from physical AD groups to logical application roles
#ldapRealm.rolesByGroup = "hadoop_grp":admin_role,"hadoop":hadoop_users_role
# Force usernames returned from ldap to lowercase, useful for AD
ldapRealm.userLowerCase = true
# Enable support for nested groups using the LDAP_MATCHING_RULE_IN_CHAIN operator
ldapRealm.groupSearchEnableMatchingRuleInChain = true
sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager
### If caching of user is required then uncomment below lines
cacheManager = org.apache.shiro.cache.MemoryConstrainedCacheManager
securityManager.cacheManager = $cacheManager
securityManager.sessionManager = $sessionManager
securityManager.realms = $ldapRealm
# 86,400,000 milliseconds = 24 hour
securityManager.sessionManager.globalSessionTimeout = 86400000
shiro.loginUrl = /api/login
[urls]
# This section is used for url-based security.
# You can secure interpreter, configuration and credential information by urls. Comment or uncomment the below urls that you want to hide.
# anon means the access is anonymous.
# authc means Form based Auth Security
# To enfore security, comment the line below and uncomment the next one
#/api/version = anon
/api/interpreter/** = authc, roles[admin_role,hadoop_users_role]
/api/configurations/** = authc, roles[admin_role]
/api/credential/** = authc, roles[admin_role,hadoop_users_role]
#/** = anon
/** = authc
... View more