Member since
06-08-2017
5
Posts
0
Kudos Received
0
Solutions
09-27-2017
06:13 PM
I too had this issue on HDP 2.6 ambari-server.log reported ERROR: invalid page header in block 13760 of relation base/16995/67484
Just below that error, there was the query which threw that error. Runnig the same on Postgre threw the same error. In my case ambri.alert_history was corrupt. Below step fixed the issue database=# SET zero_damaged_pages = on;
SET
database=# VACUUM FULL damaged_table;
WARNING: invalid page header in block 13748 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13749 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13757 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13758 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13759 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13760 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13762 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13782 of relation base/16995/67484; zeroing out page
(...)
WARNING: index "damaged_table_site_id" contains 14697831 row versions, but table contains 14709258 row versions
HINT: Rebuild the index with REINDEX.
WARNING: invalid page header in block 13762 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13816 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13817 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13818 of relation base/16995/67484; zeroing out page
(...)
WARNING: index "damaged_table_site_id" contains 14697498 row versions, but table contains 14709258 row versions
HINT: Rebuild the index with REINDEX.
VACUUM
database=# REINDEX TABLE damaged_table;
... View more
08-09-2017
08:08 PM
After Upgrading HDP from 2.5 to 2.6 everything was running fine but after installing SAM , Nifi and Registry , HDFS is not starting HDFS is configure for High Availablity. Ambari-Error 2017-08-09 15:41:06,084 - Getting jmx metrics from NN failed. URL: http://mst-1.us590.corpintra.net:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/jmx.py", line 42, in get_value_from_jmx
return data_dict["beans"][0][property]
IndexError: list index out of range
2017-08-09 15:41:10,767 - Getting jmx metrics from NN failed. URL: http://mst-2.us590.corpintra.net:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/jmx.py", line 41, in get_value_from_jmx
data_dict = json.loads(data)
File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 335, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 353, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
2017-08-09 15:41:20,101 - Getting jmx metrics from NN failed. URL: http://mst-1.us590.corpintra.net:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/jmx.py", line 41, in get_value_from_jmx
data_dict = json.loads(data)
File "/usr/lib/python2.6/site-packages/ambari_simplejson/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 335, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.6/site-packages/ambari_simplejson/decoder.py", line 353, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded The last two Traceback just keeps repeating and the HDFS never starts. Any input from your end will be really valuable @Sedat Kestepe @Brandon Wilson
... View more
Labels:
08-04-2017
02:01 PM
Run service check on HDFS, MapReduce2 and HIVE fails with almost similar error Message HDFS Run Check failure File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 199, in run_command raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/etc/passwd -H 'Content-Type: application/octet-stream' 'http://{ACTIVE_NAME_NODE}:50070/webhdfs/v1/tmp/idd735ac17_date490417?op=CREATE&user.name=hdfs&overwrite=True'' returned status_code=404.
<HTML><HEAD><TITLE>HTTP 404 File not found</TITLE></HEAD><BODY TEXT=BLACK BGCOLOR=WHITE>The requested file was not found</BODY></HTML> MapReduce2 File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 199, in run_command
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/etc/passwd -H 'Content-Type: application/octet-stream' 'http://mst-2.us590.corpintra.net:50070/webhdfs/v1/user/ambari-qa/mapredsmokeinput?op=CREATE&user.name=hdfs&overwrite=True'' returned status_code=404.
<HTML><HEAD><TITLE>HTTP 404 File not found</TITLE></HEAD><BODY TEXT=BLACK BGCOLOR=WHITE>The requested file was not found</BODY></HTML>
... View more
- Tags:
- service-check
06-08-2017
04:22 AM
Is there i document to setup by multinode cluster on google cloud. I am not able to do passwordless SSH which i reuqired for Ambari Cluster deployment.
... View more
Labels:
- Labels:
-
Apache Ambari
