Have you clean up files under dfs.datanode.data.dirs that is not being written by HDFS for blocks? If not, the non-dfs used won't change. Similar question has been answered here: https://community.hortonworks.com/questions/42122/hdfs-non-dfs-used.html. ... View more

@Vinit Pandey, for the encrypted files under HDFS encryption zone. They are only allowed to rename within the same encryption zone. You may copy to/from encryption zone which had an additional decrpt (or encrypt) overhead compared with rename. Please refer to the document for more details: "HDFS restricts file and directory renames across encryption zone boundaries. This includes renaming an encrypted file / directory into an unencrypted directory (e.g., hdfs dfs mv /zone/encryptedFile /home/bob), renaming an unencrypted file or directory into an encryption zone (e.g., hdfs dfs mv /home/bob/unEncryptedFile /zone), and renaming between two different encryption zones (e.g., hdfs dfs mv /home/alice/zone1/foo /home/alice/zone2). In these examples, /zone, /home/alice/zone1, and /home/alice/zone2 are encryption zones, while /home/bob is not. A rename is only allowed if the source and destination paths are in the same encryption zone, or both paths are unencrypted (not in any encryption zone)." ... View more

Like other MR jobs, you can specify the number of map tasks for the job via JobConf.setNumMapTasks(). However, this is only a hint and the actual number of spawned map tasks depends on the number of input splits. If you have set NumMapTasks=100 with 50 nodes, then the number of tasks in parallel per node is about 100/50 = 2 assuming the splits are evenly distributed across the nodes. ... View more

@Saurabh Saurabh, have you check the following HCC article to see if it applies to your case? https://community.hortonworks.com/articles/16144/write-or-append-failures-in-very-small-clusters-un.html ... View more

@Yog Prabhhu, you can get the file block information from WebHDFS REST API like curl -i "http://<HOST>:<PORT>/webhdfs/v1/<FilePath>?op=GETFILEBLOCKLOCATIONS The corresponding JAVA API is FileSystem.getFileBlockLocations: public BlockLocation[] getFileBlockLocations(FileStatus file, long start, long len) You will get an array of block locations like below: [BlockLocation(offset: 0, length: BLOCK_SIZE,* hosts: {"host1:9866", "host2:9866, host3:9866"},...,] ... View more

@Michael Bronson It depends on whether the compoennts are going to use the new disks or not. If not, they don't need to restart. For those services that need to use the new disk. Some of them, such as HDFS datanode supported Hot-Swap, which means you can add disks by the following steps without a restart of datanode service. 1> changing the dfs.datanode.data.dir from hdfs-site.xml to include new disk locations (e.g., /data/disk2). <property> <name>dfs.datanode.data.dir</name> <value>/data/disk1,/data/disk2</value> </property> 2> Run hdfs CLI to reconfig datanode service without a restart. hdfs dfsadmin-reconfig datanode dn1.hdp.com:9820 start Other services might need a restart to use the new disks if Hot-Swap is not supported. ... View more

@Prateek Behera The disk quota in your case works fine as expected. HDFS by default has a replication factor of 3 as you can see in the 3rd column of your CLI output. 500MB *3 (replication factor) = 1.5 GB > 1GB (quota). ... View more

@szetszwo, you can do this with haodop impersonation by setting the env variable $HADOOP_PROXY_USER=foo ... View more

hdfs debug recoverLease command should be used to close the file. The complete syntax is like below: hdfs debug recoverLease -path <path-of-the-file>[-retries <retry-times>] ... View more

Apache Hadoop has an official wiki page that summarized major changes across 1.x, 2.x. Hope that helps. https://wiki.apache.org/hadoop/Roadmap ... View more

You will need to add auth_to_local rule in core-site.xml to map dgiri@mytestdomain.com to dgiri RULE:[2:$1](.*@mytestdomain.com)s/@.*// More defailt of how to write auth_to_local rules can be found here: https://hortonworks.com/blog/fine-tune-your-apache-hadoop-security-settings/ ... View more

bq. From what I understood every files use a minimum of 1 block in HDFS. No. It is not true. You can have file of size 0, they are likely to be created on NN but no block being allocated and/or streamed to DNs yet. Check your files and you will have 4 files (8694-8690=4) of size 0. ... View more

decrypt operation needs POST instead of GET. Check the samples form KMS doc below. Decrypt Encrypted Key REQUEST: POST http://HOST:PORT/kms/v1/keyversion/<version-name>/_eek?ee_op=decrypt Content-Type: application/json { "name" : "<key-name>", "iv" : "<iv>", //base64 "material" : "<material>", //base64 } RESPONSE: 200 OK Content-Type: application/json { "name" : "EK", "material" : "<material>", //base64 } ... View more

Please check hadoop wiki at https://wiki.apache.org/hadoop though some of the info maybe outdated. ... View more

@Xiong Duan - fully reload configuration files like hdfs-site.xml/core-site.xml without a restart is not supported. However, you can change certain configuration keys without a restart using hdfs dfsadmin [-reconfig <namenode|datanode> <host:ipc_port> <start|status|properties>] ... View more