Member since
10-21-2015
59
Posts
31
Kudos Received
16
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1531 | 03-09-2018 06:33 PM | |
1373 | 02-05-2018 06:52 PM | |
10603 | 02-05-2018 06:41 PM | |
2705 | 11-30-2017 06:46 PM | |
1020 | 11-22-2017 06:20 PM |
11-05-2018
06:26 PM
is both clusters, datalab and juggernaut in azure?
... View more
10-25-2018
07:02 PM
Perhaps, you need to return an error from the Server so that S3AFS slows down, and retries? would that be an option for you, that would be more generic way of solving your isssue.
... View more
10-25-2018
07:01 PM
But you do realize that S3AFile system is just one application among
thousands of applications that can run against the S3 head, right?
... View more
10-25-2018
06:39 PM
> In my object store code if you have access to your code base and know where the issue is; won't it be easier to fix the issue you have? If you fix S3 via parameters it is just a matter of time before you run into other issues.
... View more
08-31-2018
06:44 PM
Thanks for the update. Glad you were able to make it work. Thanks for the comments and sharing it with the community.
... View more
04-26-2018
06:22 PM
@Sriram Hadoop >How can I change the block size for the existing files in HDFS? I want to increase the block size. May I ask what you are trying to achieve? We might be able to make better suggestions if we know what is the problem you are trying to solve?
... View more
04-26-2018
06:19 PM
1 Kudo
@Michael Bronson 1. is it safty to run e2fsck -y /dev/sdf in order torepair a disk /dev/sdf file-system ? Datanodes need to be able to read and write to the underlying file system. So if there is an error in the file system, we have no choice but to fix it. That said, HDFS will have the same blocks on other machines. So you can put this node into maintenance mode in Ambari and fix the file system errors. There is a possibility of losing some data blocks. So if you have this error in more than one datanode, please do this one by one, with some time in between. I would run fcsk and then reboot the Datanode machine to make sure everything is okay, before starting work on the next node. 2. is it necessary to do some other steps after running - e2fsck -y /dev/sdf ? Not from the HDFS point of view, as I said, I would make sure I am doing this datanode by datanode and not in parallel.
... View more
03-09-2018
06:34 PM
@smdas Sorry forgot to tag you.
... View more
03-09-2018
06:33 PM
A quick search in the code base tells me that we have these following policies AvailableSpaceBlockPlacementPolicy BlockPlacementPolicyDefault BlockPlacementPolicyRackFaultTolerant BlockPlacementPolicyWithNodeGroup BlockPlacementPolicyWithUpgradeDomain > yet didn't find any documentation listing the available choices. You are absolutely right, we can certainly do better on documenting this, Thanks for pointing this out. I will address this in an Apache JIRA.
... View more
02-27-2018
06:15 PM
can you please post the log statement that tells you what the error is ?
... View more
02-14-2018
05:07 PM
@PJ I am guessing that it could be related to "dfs.namenode.startup.delay.block.deletion" value since you mention that you restarted the cluster.
... View more
02-05-2018
06:52 PM
@Malay Sharma Really depends on your workload. The best case is if you can get good block sizes, that if you have really large files, you should set your block size to 256MB( this reduces the number of blocks in the file system, since you are storing more data per block), or use the default 128 MB. If you have a running cluster, using smart sense, you can get a view of the file size distribution of your current cluster. That will give you an idea if you need to tune the block size for performance. From experience, I can tell you that the performance of your cluster is not going to be dependent on block size unless you have a large number of files in the system. Also, unlike a physical file system, setting the block size to 128MB does not mean that each block write will use up 128MB. HDFS will only use the number of bytes actually written, so there is no waste of space because of the block size.
... View more
02-05-2018
06:41 PM
2 Kudos
@Malay Sharma HDFS is very good at caching all file names and block addresses in memory. This is done at the Namenode level. This makes HDFS incredibly fast. So if you make modifications to the file system or read a file location, all of this can be served with no disk I/Os. This design choice of keeping all metadata in memory at all times has certain trade-offs. One of them is that we need to spend a couple of bytes (think 100s of bytes) per file and blocks. This leads to an issue in HDFS -- when you have a file system with 500 million to 700 million -- the amount of RAM that needs to be reserved by the Namenode becomes large. Typically, in sizes of 256 GB or more. At this size, the JVM is hard at work too; since it has to do things like garbage collection. There is also another dimension to this when you have 700 million files, it is quite possible that your cluster is serving 30-40K or more requests per second. This also creates lots of memory churn. So a large number of files, combined with lots of file system requests makes Namenode a bottleneck in HDFS or in other words, the metadata that we need to keep in memory creates a bottleneck in HDFS. There are several solutions / work in progress to address this problem -- HDFS federation -- That is being shipped as part of HDP 3.0, allows many names nodes to work against a set of Datanodes. In HDFS-7240, where we are trying to separate the block space from the namespace, that allows us to immediately double or quadruple the effective size of the cluster. Here is good document that tracks various issues and different approaches to scaling the namenode -- Uber scaling namenode There is another approach where we try to send the read workloads to the secondary namenode, freeing up the active namenode and thus scaling it better. That work is tracked in Consistent Reads from Standby Node Please let me know if you have any other questions. Thanks Anu
... View more
12-23-2017
06:54 PM
@zkfs YARN Resource Manager has a configuration that prevents you from launching jobs when the data node is around 90-95% full. So that is the first level of error you will see. At the HDFS level, you will error since block allocations and writes would fail. If you have Amabari, then Ambari will be firing alerts much before you reach 100%. I don't think HDFS will fire alerts, instead, you will see failures at the HDFS level.
... View more
12-05-2017
06:18 PM
From the error message, it looks like some of the services might not be running. Can you please make sure that zookeeper and journal nodes are indeed running before starting NN?
... View more
12-01-2017
06:23 PM
1 Kudo
@Sedat Kestepe Since you don't care about data, from an HDFS perspective it is easier to reinstall your cluster. If you insist I can lead you through the recovery steps, but if I were you I would just reinstall at this point.
... View more
12-01-2017
06:19 PM
@Michael Bronson The steps described by you looks good. If you have Ambari running against this cluster, you should be able to find an option called "Maintenance mode" in the menus. Here is some documentation about that, https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-operations/content/setting_maintenance_mode.html It is not needed for your replace your disks, but it will avoid spurious alerts in your system.
... View more
11-30-2017
06:54 PM
@kskp This is a strange problem. I would check the following things. 1. Check if anyone has done a massive delete operation? 2. Did you mount a disk with data already -- that is a set of data blocks pull out from some other machine/cluster? 3. Please keep the cluster in safe mode, till you puzzle out what happened.
... View more
11-30-2017
06:46 PM
@Michael Bronson Assuming you are talking about a data node. If you are able to replace this disk trivially, that is the operation is simply to pull the disk out of a JBOD, then you can shut down this datanode, replace the disk, format and mount it back. HDFS will detect that it has lost a set of blocks (probably it has already done it since the disk is faulty and no i/o is happening to that disk), and replicate them correctly. You can check if you have any under-replicated blocks in your cluster. You can replace the disk and things will return to normal. There is, however, a small hitch, this new disk will not have the same amount of data as the other disks. If you are running Hadoop 3.0-- it is still in beta and not production ready -- you can run the diskBalancer tool, which will move data from other disks to this new disk. Generally, this will not be an issue. If the disk replacement in your machines is not as straightforward as I described, you can ask Ambari to put this machine into a maintenance state. That will tell HDFS not to replicate all the blocks after the 10 mins window ( by default) when a machine is declared dead. You can do that and then do the operation. Just so that you are aware, HDFS supports a notion of failed volumes. So if you have a data node with a large number of disks, say 8 disks, then you can set the failed volume tolerance, say to something like 2. This will make sure that node works well even in the face of two disks failing. if you do that, you can replace disks when you have a scheduled maintenance window with downtime. Please let me know if you have any more questions or need more help on this.
... View more
11-28-2017
05:53 AM
1 Kudo
@satya gaurav You will need to change a parameter inside hadoop-env.sh, the parameter is called HADOOP_DATANODE_OPTS, you can pass standard JVM commands via this parameters. now to specifically answer your questions. 1. We need to increase the heap size using command line (as GUI is
not here) for data node and our cluster using hadoop version is hadoop-common 2.4.0.2.1.2.0-402. change the value of HADOOP_DATANODE_OPTS inside hadoop-env.sh. 2. So if we change the parameter in hadoop-env.sh on name node will it
reflect to all nodes or we have to do it manually in all data nodes. No, if you change in namenode, it will not automatically reflect in all datanodes. That is where a tool like Amabri is useful, in this case you will have to copy the hadoop-env.sh yourself. 3.whether we need down time or we can change it without stoping any services. You can achieve this goal via a rolling restart, since this is a manual operation I suggest that you make sure that all datanodes are up and running and reboot 2 nodes at a time. That will ensure maximum availability of your cluster. 4.If we need to stop the services then please let us know the services name i.e which services need to stop. you need to restart datanode service on all datanodes. Hope this helps. Please let me know if you have any more questions.
... View more
11-28-2017
12:24 AM
@satya gaurav Do you have any tools like Ambari or similar tools that allow you to manage the cluster? if you have, the UI will have settings to make this change and do a rolling restart of your datanodes. Please let me know if that is not the case, and you really want to do this from command line, it is possible, but wanted to make sure that I am not giving wrong instructions, hence the query.
... View more
11-22-2017
06:20 PM
@ARUN it is very dependent on the type and nature of your data. It is also influenced by the applications that consume data.
... View more
11-20-2017
11:40 PM
1 Kudo
@Michael Bronson Just to add to what @Xiaoyu Yao said; if you want to clean only a data node (worker23), you will need to ssh to that node. Then you can either format the data disks or delete data. You can see in Ambari which paths map to the data store. But you need to make sure that are actual disks if you want to run format. A safer approach would be to run "rm -rf /data/disk1", "rm -rf /data/disk2", assuming data node has stored data in a path called /data/disk1 and /data/disk2 Just as @Xiaoyu Yao mentioned, please do NOT format the namenode, the whole cluster will be lost.
... View more
09-08-2017
05:28 PM
Since you said only one node, I am not sure if you mean there is only one physical machine in the cluster. if that is the case, you want to enable something called pseudo-distributed mode, where all processes, that is namenode, datanode etc. are going to run on the same machine. You can find instructions here. https://hortonworks.com/hadoop-tutorial/introducing-apache-ambari-deploying-managing-apache-hadoop/ On the other hand, if you want to run namenode on a single machine -- but you have a set of physical nodes, you can use Ambari normally. if you want to use Amabri, You can select HDFS and the standard installation does not assume that you have HA. If you want to turn on HA then use HA Wizard.
... View more
02-09-2017
05:33 PM
2 Kudos
@suresh krish These two links contain most of the information that is needed for tuning the Namenode heap. Alerts come from Ambari, so if you want to tune that instead of Namenode heap, you can do that in Ambari. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_installing_manually_book/content/ref-80953924-1cbf-4655-9953-1e744290a6c3.1.html https://community.hortonworks.com/articles/14170/namenode-garbage-collection-configuration-best-pra.html
... View more
01-12-2017
06:19 PM
@Tomomichi Hirano Yes, it does.
... View more
01-12-2017
02:28 AM
@Tomomichi Hirano Would it be possible that Standby namenode is having too much Garbage collections going on ? You might want to look for GC's and if you see a GC happening each time your check point is happening, then it might explain why the standby namenode is not able to keep up. If that is the case, then tuning the memory setting of Namenode might be the right approach instead of changing the block report frequency. When we do a check point we have to decode lot of small objects and that can create memory pressure on Standby Namenode, Can you please check if there is a correlation between GC and checkpointing ?
... View more
01-10-2017
06:15 PM
@Tomomichi Hirano Without understanding why you are seeing a "block report storm", it is hard to say if increasing this parameter will help. Typically most clusters -- even very large clusters seem to work fine with this parameter. Would you be able to share -- How many datanodes are in this cluster / how many data blocks it has ? If you have too many blocks, then block reports might slow down the active namenode. I am surprised that you are impacted by performance issues in the standby namenode. Would it be that you have GC issues and you are seeing some kind of alerts from the standby ?
... View more
01-10-2017
03:16 AM
1 Kudo
@Tomomichi Hirano No you cannot. Block reports serve an essential function that allows Namenode to reconcile the state of the cluster. That is these block reports tell namenode what blocks are needed and what blocks are to be deleted, if the block is under replicated etc. etc. Full block reports are expensive for Namenode process( there are both incremental and full reports) so it is set to a longer value. However, if you set it to really long values like 1 month, your namenode might not work correctly. Typically the only reason to change this value is if your Namenode is under severe load, so if you are not experiencing such a problem, I would suggest that you don't change this parameter.
... View more
12-22-2016
06:52 PM
1 Kudo
@Namit Maheshwari yes, it is possible - couple of ways you can do this are: 1. You can do it from JAVA code by using JMX Apis. 2. You can connect to namenode:httpport;/jmx and get a json based response. 3. You can configure a Namenode/Datanode Metrics log file which will dump into a file which you can parse I am probably missing some more ways :), hope this helps you.
... View more