HI tarek  you can try running this , this will fix the issue.    service cloudera-scm-agent restart ... View more

Hi Gautam My issue is I think with filled space , after troubleshooting i found i was earlier using /dfs/dn for HDFS block storage , later i added non OS partition under /home (/home/hdfs/dfs/dn) and then started importing  100 of GB data.  Looks like some how my old path /dfs/dn had also sotred some of HDFS blocks and filled that root partition. Sais so if now by chnagging the configuration  remove (/dfs/dn)  dfs.data.dir  and restart cluster will it do automatic move data to only left location /home/hdfs/dfs/dn or how to handle that. I guess this will fix my problem for now.   Do not worry about data much what ever best and quick will be fine .     [root@hadoop-vm2 /]# du -sh ./* 7.9M ./bin 61M ./boot 4.0K ./cgroup 196K ./dev 40G ./dfs 30M ./etc 55G ./home 12K ./impala 263M ./lib 27M ./lib64 16K ./lost+found 4.0K ./media 0 ./misc 4.0K ./mnt 0 ./net 3.5G ./opt du: cannot access `./proc/20676/task/20676/fd/4': No such file or directory du: cannot access `./proc/20676/task/20676/fdinfo/4': No such file or directory du: cannot access `./proc/20676/fd/4': No such file or directory du: cannot access `./proc/20676/fdinfo/4': No such file or directory 0 ./proc 92K ./root 15M ./sbin 4.0K ./selinux 4.0K ./srv 0 ./sys 1.2M ./tmp 2.9G ./usr 387M ./var 223M ./yarn [root@hadoop-vm2 /]# ls /dfs/dn/current/ BP-1505211549-172.28.172.30-1424252944658 VERSION ... View more

yes file does exist and it has below content    [root@hadoop-vm3 ~]# cat /opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/bin/hadoop #!/bin/bash # Reference: http://stackoverflow.com/questions/59895/can-a-bash-script-tell-what-directory-its-stored-in SOURCE="${BASH_SOURCE[0]}" BIN_DIR="$( dirname "$SOURCE" )" while [ -h "$SOURCE" ] do SOURCE="$(readlink "$SOURCE")" [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE" BIN_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )" done BIN_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )" LIB_DIR=$BIN_DIR/../lib # Autodetect JAVA_HOME if not defined . $LIB_DIR/bigtop-utils/bigtop-detect-javahome export HADOOP_LIBEXEC_DIR=//$LIB_DIR/hadoop/libexec exec $LIB_DIR/hadoop/bin/hadoop "$@" [root@hadoop-vm3 ~]#     ... View more

HI Gautam   please find the conetent as below, let me know next...   [root@hadoop-vm3 alternatives]# pwd /var/lib/alternatives [root@hadoop-vm3 alternatives]# cat hadoop auto /usr/bin/hadoop /opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/bin/hadoop 10 [root@hadoop-vm3 alternatives]# ... View more

Hi Gautam  thanks alot for your reply , need some more info to fix this. can this be fixed from cloudera manager or i need to fix it on each machine by running some command   I checked  /etc/alternatives/hadoop and it points to  hadoop -> /opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/bin/hadoop while in /var/lib/alternatives   also i found that hadoop is there . Now I am not sure what to delete here..?   So my question is how to restore this for all hadoop command like hadoop/hive etc....just to give you backgroud when this issue started, i got some configuration warning in cloudera manager and it was referring some outdated parcel etc, i just fixed that warning and this issue started. so is there any way i can fix this from cloudera manager. your help will be really appericiated ....   one interesting thing i noticed that when i gave the complete path , its able to list files   [root@hadoop-vm3 bin]# /opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/bin/hadoop fs -ls / Found 4 items drwxr-xr-x - hbase hbase 0 2015-02-20 05:41 /hbase drwxrwxr-x - solr solr 0 2015-02-18 04:50 /solr drwxrwxrwt - hdfs supergroup 0 2015-02-20 06:44 /tmp drwxr-xr-x - hdfs supergroup 0 2015-02-20 00:54 /user     ========================================================================== [root@hadoop-vm3 bin]# ls /var/lib/alternatives/ avro-tools hadoop-httpfs-conf hiveserver2 jre_openjdk mapred solr-conf sqoop-create-hive-table sqoop-version beeline hadoop-kms-conf hive-webhcat-conf kite-dataset mta solrctl sqoop-eval statestored catalogd hbase hue-conf libnssckbi.so.x86_64 oozie spark-conf sqoop-export whirr cli_mt hbase-conf impala-conf links oozie-conf spark-executor sqoop-help yarn cli_st hbase-indexer impalad llama pig spark-shell sqoop-import zookeeper-client flume-ng hbase-solr-conf impala-shell llamaadmin pig-conf spark-submit sqoop-import-all-tables zookeeper-conf flume-ng-conf hcat ip6tables.x86_64 llama-conf print sqoop sqoop-job zookeeper-server hadoop hdfs iptables.x86_64 load_gen pyspark sqoop2 sqoop-list-databases zookeeper-server-cleanup hadoop-0.20 hive java mahout sentry sqoop2-conf sqoop-list-tables zookeeper-server-initialize hadoop-conf hive-conf jre_1.6.0 mahout-conf sentry-conf sqoop-codegen sqoop-merge hadoop-fuse-dfs hive-hcatalog-conf jre_1.7.0 mail senty-conf sqoop-conf sqoop-metastore ========================================================================== ... View more