Member since
06-23-2014
9
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
90220 | 10-10-2014 08:09 AM |
10-10-2014
08:09 AM
2 Kudos
I was able to remove the corrupt files using hdfs@sandy-quad-1:~$ hdfs fsck / -delete Now I still need to find out how to fix the 'Under-Replicated Blocks' problem...
... View more
10-10-2014
07:59 AM
1 Kudo
In my HDFS status summary in Cloudera Manager, I see the followign messages about missing and underreplicated blocks: "2 missing blocks in the cluster. 8,062 total blocks in the cluster. Percentage missing blocks: 0.02%. Critical threshold: any." "6,498 under replicated blocks in the cluster. 8,062 total blocks in the cluster. Percentage under replicated blocks: 80.60%. Critical threshold: 40.00%." The 2 corrupt files are the following: hdfs@sandy-quad-1:~$ hdfs fsck -list-corruptfileblocks
14/10/10 16:51:59 WARN ssl.FileBasedKeyStoresFactory: The property 'ssl.client.truststore.location' has not been set, no TrustStore will be loaded
Connecting to namenode via http://sandy-quad-1.sslab.lan:50070
The list of corrupt files under path '/' are:
blk_1074173133 /user/history/done/2014/10/07/000001/job_1412322902461_1076-1412674739294-bart-hadoop%2Dmapreduce%2Dclient%2Djobclient%2D2.3.0%2Dcdh5.1.2%2Dt-1412674771395-10-1-SUCCEEDED-root.bart-1412674749886.jhist
blk_1074173134 /user/history/done/2014/10/07/000001/job_1412322902461_1076_conf.xml
The filesystem under path '/' has 2 CORRUPT files What is the best way to fix these two corrupt files and also fix the underreplicated block problem?
... View more
Labels:
- Labels:
-
HDFS
09-10-2014
01:24 AM
I am using Cloudera Express 5.0.2 (#297 built by jenkins on 20140606-2222 git: 80907df78ba6b50c21a598f0caff8b00685d5961) with the CDH 5.1.2-1.cdh5.1.2.p0.3 parcel on an Ubuntu 12.04.4 LTS machine in a 4-node cluster. I want to get rid of the following Hive warning: 14/09/09 16:31:36 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore. and I'm therefore now looking at how to configure the Hive metastore. If I go to the Hive configuration in my Cloudera Manager UI, and search for 'hive.metastore.uris', this configuration property is not found and I don't see a way to add it. Furthermore, if I search for the file hive-site.xml, I find multiple versions of this file: root@sandy-quad-1:~# find / -name hive-site.xml
/run/cloudera-scm-agent/process/456-hive-HIVEMETASTORE/hive-site.xml
/run/cloudera-scm-agent/process/455-hive-HIVESERVER2/hive-site.xml
/run/cloudera-scm-agent/process/454-hue-HUE_SERVER/hive-conf/hive-site.xml
/run/cloudera-scm-agent/process/453-hive-HIVEMETASTORE/hive-site.xml
/run/cloudera-scm-agent/process/452-hive-HIVESERVER2/hive-site.xml
/run/cloudera-scm-agent/process/431-hue-HUE_SERVER/hive-conf/hive-site.xml
/run/cloudera-scm-agent/process/423-deploy-client-config/hive-conf/hive-site.xml
/run/cloudera-scm-agent/process/407-hive-HIVEMETASTORE/hive-site.xml
/run/cloudera-scm-agent/process/406-hive-HIVESERVER2/hive-site.xml
/run/cloudera-scm-agent/process/403-hive-metastore-create-tables/hive-site.xml
/run/cloudera-scm-agent/process/394-hue-HUE_SERVER/hive-conf/hive-site.xml
/run/cloudera-scm-agent/process/393-deploy-client-config/hive-conf/hive-site.xml
/run/cloudera-scm-agent/process/363-impala-CATALOGSERVER/hive-conf/hive-site.xml
/run/cloudera-scm-agent/process/362-hive-HIVEMETASTORE/hive-site.xml
/run/cloudera-scm-agent/process/361-hive-HIVESERVER2/hive-site.xml
/run/cloudera-scm-agent/process/331-impala-CATALOGSERVER/hive-conf/hive-site.xml
/run/cloudera-scm-agent/process/330-hive-HIVEMETASTORE/hive-site.xml
/run/cloudera-scm-agent/process/329-hive-HIVESERVER2/hive-site.xml
/run/cloudera-scm-agent/process/305-hue-HUE_SERVER/hive-conf/hive-site.xml
/run/cloudera-scm-agent/process/304-hue-HUE_SERVER/hive-conf/hive-site.xml
/run/cloudera-scm-agent/process/303-hue-HUE_SERVER/hive-conf/hive-site.xml
/run/cloudera-scm-agent/process/296-impala-CATALOGSERVER/hive-conf/hive-site.xml
/run/cloudera-scm-agent/process/295-hive-HIVEMETASTORE/hive-site.xml
/run/cloudera-scm-agent/process/294-hive-HIVESERVER2/hive-site.xml
/run/cloudera-scm-agent/process/269-impala-CATALOGSERVER/hive-conf/hive-site.xml
/run/cloudera-scm-agent/process/267-deploy-client-config/hive-conf/hive-site.xml
/run/cloudera-scm-agent/process/247-hue-HUE_SERVER/hive-conf/hive-site.xml
/run/cloudera-scm-agent/process/244-hive-HIVEMETASTORE/hive-site.xml
/run/cloudera-scm-agent/process/243-hive-HIVESERVER2/hive-site.xml
/run/cloudera-scm-agent/process/216-hue-HUE_SERVER/hive-conf/hive-site.xml
/run/cloudera-scm-agent/process/207-hive-HIVEMETASTORE/hive-site.xml
/run/cloudera-scm-agent/process/206-hive-HIVESERVER2/hive-site.xml
/run/cloudera-scm-agent/process/204-hue-HUE_SERVER/hive-conf/hive-site.xml
/opt/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/etc/hive/conf.dist/hive-site.xml
/etc/hive/conf.cloudera.hive1/hive-site.xml
/etc/hive/conf.cloudera.hive/hive-site.xml and I don't know which one is the one I need to change. My questions are: Is it possible to add configuration options that are not there in my Hive config in the Cloudera Manager UI? What is the main hive-site.xml config file that I need to change, and what services do I need to restart in order for these changes to take effect? Thanks for your help, Bart
... View more