Member since
09-28-2015
51
Posts
31
Kudos Received
17
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
880 | 04-13-2018 11:36 PM | |
2545 | 04-13-2018 11:03 PM | |
660 | 04-13-2018 10:56 PM | |
2493 | 04-10-2018 03:12 PM | |
2895 | 02-13-2018 07:23 PM |
10-05-2018
12:10 AM
1 Kudo
These seems to be bogus replay exception when running solr service. Changes hadoop-env.sh or solr JVM option with -Dsun.security.krb5.rcache=none should fix the problem. # # Extra Java runtime options. Empty by default.
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Dsun.security.krb5.rcache=none ${HADOOP_OPTS}"
... View more
08-27-2018
06:14 PM
@Daniel Muller, can you grep "Safe mode is" from hdfs namenode log? That will tell the reason why namenode does not exit safemode directly.
... View more
08-27-2018
06:07 PM
@Simran kaur, we don't usually format a production cluster unless it is absolutely necessary? Can you elaborate on what happened to the namenode (like hdfs command that had been executed) before the NN fail to start?
... View more
04-13-2018
11:36 PM
Have you clean up files under dfs.datanode.data.dirs that is not being written by HDFS for blocks? If not, the non-dfs used won't change. Similar question has been answered here: https://community.hortonworks.com/questions/42122/hdfs-non-dfs-used.html.
... View more
04-13-2018
11:03 PM
@Vinit Pandey, for the encrypted files under HDFS encryption zone. They are only allowed to rename within the same encryption zone. You may copy to/from encryption zone which had an additional decrpt (or encrypt) overhead compared with rename. Please refer to the document for more details: "HDFS restricts file and directory renames across encryption zone boundaries. This includes renaming an encrypted file / directory into an unencrypted directory (e.g., hdfs dfs mv /zone/encryptedFile /home/bob), renaming an unencrypted file or directory into an encryption zone (e.g., hdfs dfs mv /home/bob/unEncryptedFile /zone), and renaming between two different encryption zones (e.g., hdfs dfs mv /home/alice/zone1/foo /home/alice/zone2). In these examples, /zone, /home/alice/zone1, and /home/alice/zone2 are encryption zones, while /home/bob is not. A rename is only allowed if the source and destination paths are in the same encryption zone, or both paths are unencrypted (not in any encryption zone)."
... View more
04-13-2018
10:56 PM
Like other MR jobs, you can specify the number of map tasks for the job via JobConf.setNumMapTasks(). However, this is only a hint and the actual number of spawned map tasks depends on the number of input splits. If you have set NumMapTasks=100 with 50 nodes, then the number of tasks in parallel per node is about 100/50 = 2 assuming the splits are evenly distributed across the nodes.
... View more
04-10-2018
03:12 PM
@Saurabh Saurabh, have you check the following HCC article to see if it applies to your case? https://community.hortonworks.com/articles/16144/write-or-append-failures-in-very-small-clusters-un.html
... View more
03-21-2018
07:04 PM
@John Liao , thanks for reporting the issue. Have you checked the hdfs audit log during that period of time when the NN flipped? It is likely you have a spike in the number of RPC requests (e.g., large recursive delete, getContentSummary call, etc) toward the active NN that overwhelm the call queue. As a result of that, ZKFC cannot get any response within the configured timeout (default=45s) and triggered failover. Please check @Arpit Agarwal's previous articles on best practice to scale your namenode to handle similar situations: https://community.hortonworks.com/articles/43838/scaling-the-hdfs-namenode-part-1.html
... View more
02-13-2018
07:23 PM
@Yog Prabhhu, you can get the file block information from WebHDFS REST API like curl -i "http://<HOST>:<PORT>/webhdfs/v1/<FilePath>?op=GETFILEBLOCKLOCATIONS The corresponding JAVA API is FileSystem.getFileBlockLocations: public BlockLocation[] getFileBlockLocations(FileStatus file,
long start, long len)
You will get an array of block locations like below: [BlockLocation(offset: 0, length: BLOCK_SIZE,* hosts: {"host1:9866", "host2:9866, host3:9866"},...,]
... View more
02-01-2018
04:17 PM
Can you check you hadoop.kms.authentication.kerberos.name.rules settings from kms-site.xml? Try "DEFAULT" if you have a customized setting that is invalid. You mentioned that the kms principle is changed. Can you also post your hadoop.kms.authentication.kerberos.principal and hadoop.security.auth_to_local settings from core-site.xml?
... View more
01-08-2018
07:53 PM
This is a known limitation of wholeTextFiles as reported in https://issues.apache.org/jira/browse/SPARK-18965. Try using binaryFiles as suggested in https://issues.apache.org/jira/browse/SPARK-22225.
... View more
01-02-2018
06:29 PM
@Michael Bronson It depends on whether the compoennts are going to use the new disks or not. If not, they don't need to restart. For those services that need to use the new disk. Some of them, such as HDFS datanode supported Hot-Swap, which means you can add disks by the following steps without a restart of datanode service. 1> changing the dfs.datanode.data.dir from hdfs-site.xml to include new disk locations (e.g., /data/disk2).
<property>
<name>dfs.datanode.data.dir</name>
<value>/data/disk1,/data/disk2</value>
</property>
2> Run hdfs CLI to reconfig datanode service without a restart.
hdfs dfsadmin-reconfig datanode dn1.hdp.com:9820 start
Other services might need a restart to use the new disks if Hot-Swap is not supported.
... View more
01-02-2018
06:13 PM
1 Kudo
@Prateek Behera The disk quota in your case works fine as expected. HDFS by default has a replication factor of 3 as you can see in the 3rd column of your CLI output. 500MB *3 (replication factor) = 1.5 GB > 1GB (quota).
... View more
11-28-2017
12:16 AM
1 Kudo
@szetszwo, you can do this with haodop impersonation by setting the env variable $HADOOP_PROXY_USER=foo
... View more
11-20-2017
11:34 PM
1 Kudo
There is only one namenode per HFDS cluster assuming you don't have federation. The Format namenode command you mentioned will clear all the HDFS namespace info and affect all the other works no matter where the command is being executed.
... View more
11-08-2017
06:39 PM
1 Kudo
Have you kinit before running the command? The debug log showed that you are running as a non-kerberos user root. Can you run kdestroy/kinit/klist and ensure your local ticket cache has a valid Kerberos ticket before running the hdfs command? 17/11/08 11:10:48 DEBUG security.UserGroupInformation: hadoop login commit
17/11/08 11:10:48 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: root
17/11/08 11:10:48 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: root" with name root
17/11/08 11:10:48 DEBUG security.UserGroupInformation: User entry: "root"
17/11/08 11:10:48 DEBUG security.UserGroupInformation: UGI loginUser:root (auth:SIMPLE)
... View more
11-02-2017
07:56 PM
1 Kudo
Can you try change the replication factor to see if that help? In the meanwhile, can you provide namenode log segment matching with the time above? hadoop fs -setrep
... View more
11-02-2017
06:15 PM
3 Kudos
hdfs debug recoverLease command should be used to close the file. The complete syntax is like below: hdfs debug recoverLease -path <path-of-the-file>[-retries <retry-times>]
... View more
09-28-2017
08:06 PM
Apache Hadoop has an official wiki page that summarized major changes across 1.x, 2.x. Hope that helps. https://wiki.apache.org/hadoop/Roadmap
... View more
09-19-2017
07:28 PM
1 Kudo
You will need to add auth_to_local rule in core-site.xml to map dgiri@mytestdomain.com to dgiri RULE:[2:$1](.*@mytestdomain.com)s/@.*// More defailt of how to write auth_to_local rules can be found here: https://hortonworks.com/blog/fine-tune-your-apache-hadoop-security-settings/
... View more
09-19-2017
07:14 PM
@Eon kitex have you change the owner of /hdfs/hadoopdir to hdfs:hadoop?
... View more
09-18-2017
06:57 PM
@Ajit Sonawane, can you check your zookeeper servers (abctestlab0512, abctestlab0513, abctestlab0515) and logs to see if they are running properly not overloaded?
... View more
09-08-2017
07:55 PM
bq. From what I understood every files use a minimum of 1 block in HDFS. No. It is not true. You can have file of size 0, they are likely to be created on NN but no block being allocated and/or streamed to DNs yet. Check your files and you will have 4 files (8694-8690=4) of size 0.
... View more
07-20-2017
06:54 PM
decrypt operation needs POST instead of GET. Check the samples form KMS doc below. Decrypt Encrypted Key REQUEST: POST http://HOST:PORT/kms/v1/keyversion/<version-name>/_eek?ee_op=decrypt
Content-Type: application/json
{
"name" : "<key-name>",
"iv" : "<iv>", //base64
"material" : "<material>", //base64
}
RESPONSE: 200 OK
Content-Type: application/json
{
"name" : "EK",
"material" : "<material>", //base64
}
... View more
07-20-2017
06:21 PM
dfs.data.dir is deprecated by dfs.datanode.data.dir in new version of Hadoop. Both should work in HDP 2.6.1. Please check and ensure entries are comma separated without any extra space. bq. "When I attempt to specify the dfs.datanode.data.dir using the URI (i.e [RAM_DISK]file:///ramdisk/hdfs) as detailed in the doc mentioned above,the datanode fails to start." @Constantin Stanca, can you post the datanode log when datanode failed to start?
... View more
06-23-2017
06:48 AM
2 Kudos
Please check hadoop wiki at https://wiki.apache.org/hadoop though some of the info maybe outdated.
... View more
06-12-2017
11:00 PM
1 Kudo
@Michael DeGuzis we have customer hit a similar issue and a hotfix with HDFS-11499 solved the problem. The fix has been back ported to HDP 2.6.1.
... View more
06-06-2017
07:57 PM
1 Kudo
@Xiong Duan - fully reload configuration files like hdfs-site.xml/core-site.xml without a restart is not supported. However, you can change certain configuration keys without a restart using hdfs dfsadmin [-reconfig <namenode|datanode> <host:ipc_port> <start|status|properties>]
... View more
04-19-2017
07:12 PM
It is likely another instance of HFDS-11608 where the block size is set too big (> 2GB). The overflow issue was recently fixed by https://issues.apache.org/jira/browse/HDFS-11608.
... View more
03-09-2017
04:22 PM
I don't major issue with hue from the attached image. Try curl from the sandbox itself to isolate virtual box port forwarding issue if any?
... View more