Member since
01-17-2017
13
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3000 | 05-02-2017 06:33 PM |
05-02-2017
06:43 PM
During the initial setup of HDFS using Ambari, configuring the pathway to /home/hadoop/hdfs/data, or any /home pathway for that matter, is illegal. After further digging and research, it is written as a security measure to prevent writing to the /home directory. I had to literally take the long way home.
... View more
05-02-2017
06:38 PM
@slachterman while this didn't directly help me solve the problem, it did help me think of a way to come to the solution. Thank you for the help!
... View more
05-02-2017
06:33 PM
I figured it out. Ambari is not allowing the DataNodes to access the /home (where the bulk of the free space is on a default Linux install) directories to store data. However, I created a folder pathway /home/hadoop/hdfs/data and placed a link to folder in the default /hadoop/hdfs pathway, so now the datanode directory technically reads /hadoop/hdfs/data, it executes a redirect to /home/hadoop/hdfs/data. I had to do this for every datanode. I hope they fix this in the near future. Thanks for the help!
... View more
05-02-2017
04:45 PM
@slachterman, each VM has the OS (CentOS7) running on their respective hdd and partitioned accordingly. The DN directory is pointing to /hadoop/hdfs/data. However, according to the partition sizes of the OS, this folder is limited to 50 gb. The rest is dedicated to the /home partition, just over 1.7tb on each DN. I did notice during the install of Ambari I was not allowed to point my NameNode or DataNode to /home. Is this the problem?
... View more
05-02-2017
02:28 PM
@slachterman Thank you for responding. The CRIT alert is stating my NameNode capacity is maxed out and the dfsadmin report on the NameNode is stating the same: all 28kb of the dfs are used. As I run the same report on the DataNodes, I get the same results, they only have 28kb allocated to dfs and they are full as well. As for the setup, I am using 1x physical machine with 1TB hdd to run the Ambari server/Edge Server/Secondary Master, 1x VM w/ 1TB hdd to run as Primary Master, and 3x VMs (each with a dedicated 2TB hdd) to run as the DataNodes. The purpose of this cluster is to act as a DEV/Admin Sandbox until I am given the funds to build a proper cluster. On a side note, is there much difference in setting up a cluster with physical machines vs VMs? I wouldn't think so as they both require separated networks and similar design and deployments. Then again, nothing about this setup has gone right so far...
... View more
05-01-2017
05:25 PM
I just installed Ambari 2.5.0.3 with HDP 2.6 and I am getting a critical alert stating my NameNode capacity is already 100%. After running a report, its only allocated 28kb! How do I fix this?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
04-27-2017
08:19 PM
This is what I was expecting, but this is what I am given to work with, sadly. Thank you for your input. I am hoping if this build goes well, then I can convence the "powers at be" for a bigger budget to build a proper cluster. Thank you again.
... View more
04-27-2017
04:47 PM
I'm setting up a 4 node cluster (1x phsical and 3x virtual). The physical machine is to be the master/edge and the 3x VMs are to be the datanodes. My question is when using Ambari 2.5.0.3 "assigning the masters" do I keep everything on the intended master (including the secondary namenode) and only run one zookeeper server/metrics collector/activity analyzer/activity explorer, or do I place the secondary namenode on one of the datanodes along with a zookeeper server/metrics collector/activity analyzer/activity explorer on each of the datanodes? My intent is to have the physical machine act as the client/edgenode and the VMs to just handle data. Any advice is appreciated. Thanks in advance.
... View more
Labels:
- Labels:
-
Apache Ambari
03-24-2017
12:50 PM
I am using 2.5 and still getting the same error when I try to delete the auto generated Hive View Instance. Does this patch work? How do I patch Ambari? Is there a way to manually remove the instance outside of Ambari (ie through the terminal)? Thank you in advance.
... View more
01-25-2017
03:28 PM
Thank you for your response. I have the other three already mounted to the base system (/mnt/...), named Data1, Data2, Data3 already. How do I mount them to hadoop? Is this a configuration in Ambari/HDFS? Or is this something I must do in terminal? Again, still learning the finer details of this system. Thanks.
... View more