Member since
03-14-2016
45
Posts
22
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
653 | 09-21-2018 10:02 AM | |
1669 | 09-11-2018 10:44 AM | |
1366 | 07-06-2016 01:14 PM |
10-04-2018
11:25 AM
1 Kudo
We have seen performance issue and stability issue in heavy workload system, this is due to disk latency/Shared Disk. I.e frequent namenode failover, longer boot time, slower checkpoint, slower logging, Higher fsync will cause session expiry, etc. We never recommend a shared disk for Namenode, Journal node and Zookeeper. All of these services should have a dedicated disk. You can configure following disk type according to your HDFS workload.
[dfs.namenode.name.dir]
Namenode fsimage directory, dedicated disk => HDD 15K RPM
[dfs.journalnode.edits.dir]
Namenode edit log directory, dedicated disk => SSD
[dataDir from zoo.cfg]
Zoookeeper snapshot and transaction logs, for normal usage from ZKFC and HBase => HDD 15K RPM
Zoookeeper snapshot and transaction logs, If it used by Nifi/ Accumulo/ Kafka/ Storm/ HBase and ZKFC => SSD
If you are using RAID for meta-directory(dfs.namenode.name.dir & dfs.journalnode.edits.dir), then disable RAID and check the Non-RAID performance. There is a strong redundant for meta-directory (fsimage & edits are available from Standby NameNode and remaining QJNs). If RAID is not disabled for JN, then consider using different RAID. RAID 1 and RAID 10 are also good for the dfs.journalnode.edits.dir set rather than RAID 5, due to increased in write latency for small block writes. if you don't have a faster disk, then don't consider using fsimage replication. It will impact write performance even if one of the disks slower.
... View more
- Find more articles tagged with:
- Cloud & Operations
- Disks
- FAQ
- HDFS
- journalnode
- namenode
- Zookeeper
Labels:
10-04-2018
09:53 AM
@Elias Abacioglu You can refer below guidance for configuring service port. https://community.hortonworks.com/articles/223817/how-do-you-enable-namenode-service-rpc-port-withou.html
... View more
10-04-2018
09:42 AM
1 Kudo
The service RPC port gives the DataNodes a dedicated port to report their status via block reports and heartbeats. The port is also used by Zookeeper Failover Controllers for periodic health checks by the automatic failover logic. The port is never used by client applications hence it reduces RPC queue contention between client requests and DataNode messages. Steps: 1) Ambari -> HDFS -> Configs -> Advanced -> Custom hdfs-site -> Add Property dfs.namenode.servicerpc-address.<dfs.internal.nameservices>.nn1=<namenode1 host:rpc port> dfs.namenode.servicerpc-address.<dfs.internal.nameservices>.nn2=<namenode2 host:rpc port> dfs.namenode.service.handler.count=(dfs.namenode.handler.count / 2) This RPC port receives all DN and ZKFC requests like block report, heartbeat, liveness report, etc.. Example from hdfs-site.xml,
dfs.nameservices=shva
dfs.internal.nameservices=shva
dfs.ha.namenodes.shva=nn1,nn2
dfs.namenode.rpc-address.shva.nn1=hwxunsecure2641.openstacklocal:8020
dfs.namenode.rpc-address.shva.nn2=hwxunsecure2642.openstacklocal:8020
dfs.namenode.handler.count=200 Service RPC host, port and handler threads:
dfs.namenode.servicerpc-address.shva.nn1=hwxunsecure2641.openstacklocal:8040
dfs.namenode.servicerpc-address.shva.nn2=hwxunsecure2642.openstacklocal:8040
dfs.namenode.service.handler.count=100
2) Restart Standby Namenode. You must wait till Standby Namenode out of safemode. Note: You can check Safemode status in Standby Namenode UI. 3) Restart Active Namenode. 4) Stop Standby Namenode ZKFC controller. 5) Stop Active Namenode ZKFC controller. 6) Login Active Namenode and reset Namenode HA state. #su - hdfs
$hdfs zkfc -formatZK
7) Login Standby Namenode and reset Namenode HA state. #su - hdfs
$hdfs zkfc -formatZK 😎 Start Active Namenode ZKFC controller. 9) Start Standby Namenode ZKFC controller. 10) Rolling restart the Datanodes. Note: Please check, Nodemanager should not be installed in Namenode box because it uses same port 8040. If installed then you need to change service RPC port from 8040 to different port. Ref: scaling-the-hdfs-namenode
... View more
- Find more articles tagged with:
- HDFS
- How-ToTutorial
- namenode
- Sandbox & Learning
Labels:
10-04-2018
07:41 AM
@Muthukumar Somasundaram Formatting is not an ideal option to solve this issue. In this case, you lost all your data.
... View more
09-21-2018
10:02 AM
@Santanu Ghosh The branch 1.x do not have namenode HA with QJN based. It is production ready and available only from hadoop-2.x. You can refer HDFS-HA umbrella jira, HDFS-3278.
... View more
09-18-2018
09:26 AM
Oops! It looks these are the spam account. Thx @Jordan Moore for your info.
... View more
09-17-2018
12:56 PM
2 Kudos
@Harshali Patel HDFS data is distributed across datanodes in local file system storage. You can configure list of storage disk dfs.datanode.data.dir in hdfs-site.xml. dfs.datanode.data.dir - Determines where on the local filesystem an HDFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored.
... View more
09-11-2018
10:44 AM
2 Kudos
You can safely ignore this warning if you don't have enabled service RPC. This is dedicated port in Namenode, datanode will send liveness and block report to this queue. The port is never used by client applications hence it reduces RPC queue contention between client requests and DataNode messages. You can ref this link for more, Scaling Namenode.
... View more
09-11-2018
09:22 AM
@Jagatheesh Ramakrishnan Appreciate your effort and writing this data recovery part. Can you please add a note on this article? Namenode should be stopped very immediate after file deletion otherwise it's hard to recover because namenode already send out block deletion request to datanode. So physical block might get deleted by datanode.
... View more
09-06-2018
07:28 AM
You can do tail in namenode and datanode log, also you can redirect output to dummy log file during restart. #tailf <namenode log> >/tmp/namenode-`hostname`.log #tailf <datanode log> >/tmp/datanode-`hostname`.log
... View more
09-06-2018
07:12 AM
Thanks for the confirmation. I need namenode and datanode log after HDFS service restart.
... View more
09-06-2018
07:07 AM
@Muthukumar Somasundaram Namenode will be in safemode until it receives the specified percentage(dfs.namenode.safemode.threshold-pct=0.999f) of blocks that should satisfy minimal replication and it should be reported to namenode. In your case, Namenode still waiting for block report from datanodes. Please ensure that all datanodes are up and running, and check if datanode is sending block report. Addition, Check how many blocks so far reported to namenode? ie. The reported blocks 71 needs additional 17 blocks to reach the threshold 1.0000 of total blocks 87.
... View more
09-06-2018
06:48 AM
@Michael Bronson Looks good to me. Just do one more check, what was the config that getting loaded into NN in-memory? http://<active nn host>:50070/conf and find it "dfs.datanode.data.dir". You must share us the logs. No point in going with assumptions. 🙂
... View more
09-05-2018
07:10 PM
As I said earlier, It's hard to tell you the exact cause without reviewing namenode and datanode log regarding disk registration. As we see in the UI that configured capacity is 154 GB which means it registered only two disks from each datanode. If you don't have any concern then share your logs which should be after the service restart. I still waiting for your reply to my previous question, Did you validated in local machine, hdfs-site.xml without ambari? # grep dfs.datanode.data.dir -A1 /etc/hadoop/conf/hdfs-site.xml
... View more
09-05-2018
04:26 PM
Ok. Share you namenode and one of the datanode log after service restart. Did you validate in the local machine, hdfs-site.xml without ambari? # grep dfs.datanode.data.dir -A1 /etc/hadoop/conf/hdfs-site.xml
... View more
09-05-2018
03:16 PM
Did you restarted HDFS after adding disks? It's hard to tell you exact cause without analysis namenode and datanode log. If possible attach namenode and one of the datanode log which must be after service restart. We can validate the disk registration in HDFS. Also /etc/hadoop/conf.
... View more
09-05-2018
02:52 PM
@Michael Bronson As I said earlier that your configured capacity is 154 GB, not 320 GB. This can be seen in NN UI. http://<active namenode host>:50070/dfshealth.html#tab-overview You must check "dfs.datanode.data.dir", the no of disks that are configured for HDFS. It looks you configured only two disks. Ambari -HDFS -> Config -> Settings -> DataNode directories You must do HDFS restart if you have not done after commissioning disks.
... View more
09-05-2018
02:11 PM
Ambari clearly shows, total configured capacity is 152 GB. But you need to double check from Namenode UI. Share namenode screenshot, http://<active namenode host>:50070/dfshealth.html#tab-overview http://<active namenode host>:50070/dfshealth.html#tab-datanode http://<active namenode host>:50070/dfshealth.html#tab-datanode-volume-failures Also, attach active namenode log and one of the datanode log after service restart. We have to find what are the disks are getting registered during startup. Can you get /etc/hadoop/conf/hdfs-site.xml?
... View more
09-05-2018
12:53 PM
5 Kudos
Missing Block Mark missing if all of the block replicas of that file is not reported to Namenode. Corrupt Block Mark corrupt if all of the block replicas of that file is corrupted (Or) none of them are reported to Namenode. The checklist must be done before you confirm block is corrupted/missing. Check if all datanodes are running in the cluster Check if you see dead datanodes Check if disk failure from multiple datanode Check if disk out of space from multiple datanode Check if block report is rejected by namenode (It can be seen from namenode log as a warning/error) Check if you changed any config groups Check if block physically exists in local filesystem or removed by users unknowingly. Ex: "find <dfs.datanode.data.dir> -type f -iname <blkid>*". Repeat the same step in all datanodes Check if too many blocks hosted in a single datanode Check if block report fails with "exceeding max RPC size", default 64 MB. You can see this warning from namenode log "Protocol message was too large. May be malicious" Check if mount point is unmounted because of filesystem failure Check if block is written into root volume because of disk auto unmount. Data might be hidden if you remount the filesystem on top of existing datanode dir. Note: You will lose data if you run "hdfs fsck / -delete". Please ensure you have done all checklist
... View more
Labels:
09-05-2018
10:25 AM
Yes, you can use remaining space to other blocks.
... View more
09-05-2018
09:51 AM
@Michael Bronson 134+18=152 GB is your total configured capacity. It is not 320 GB. Please confirm if all volumes (/dev/sdb,/dev/sdc,/dev/sdd,/dev/sde) are added in "dfs.datanode.data.dir" (hdfs-site.xml) to sum up 320 GB as configured capacity.
... View more
09-05-2018
09:46 AM
@rabbit s -1. I totally disagree. HDFS calculates exactly what it used. Please don't confuse it.
... View more
06-05-2018
09:23 AM
4 Kudos
Simple API to find region name for given row key. It will also give you hint about the region - Where key will be placed. To compile class: $JAVA_HOME/bin/javac -cp `hbase classpath`: Regionfinder.java To run tool: $JAVA_HOME/bin/java -cp `hbase classpath`: Regionfinder <tablename> <rowkey> Regionfinder.java import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HRegionLocation;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.RegionLocator;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.util.Bytes;
public class Regionfinder {
private static void usage(String[] args) {
if (args.length < 2) {
System.out.println("$JAVA_HOME/bin/java -cp `hbase classpath`: Regionfinder <tablename> <rowkey>");
System.exit(-1);
}
}
public static void main(String[] args) throws IOException {
usage(args);
TableName tablename = TableName.valueOf(args[0]);
String rowkey = args[1];
byte[] keyInBytes = Bytes.toBytes(rowkey);
Configuration conf = HBaseConfiguration.create();
Connection connection = ConnectionFactory.createConnection(conf);
Table table = connection.getTable(tablename);
RegionLocator regionLocater = connection.getRegionLocator(tablename);
HRegionLocation regionLocation = regionLocater.getRegionLocation(keyInBytes);
Result result = table.get(new Get(keyInBytes));
if(result.isEmpty()){
System.out.println("Rowkey "+rowkey+" is not exist in any region. It will be placed in region : "+regionLocation.getRegionInfo().getRegionNameAsString());
}else{
System.out.println("Table Name = " + tablename + "\n" + "Row Key = " + rowkey + "\n" + "Region Server = "
+ regionLocation.getServerName() + "\n" + "Region Name = "
+ regionLocation.getRegionInfo().getRegionNameAsString() + "\n" + "Encoded Region Name = "
+ regionLocation.getRegionInfo().getEncodedName());
}
}
}
Ex Output:
#java -cp `hbase classpath`: Regionfinder sp 100
Table Name = sp
Row Key = 100
Region Server = hwx2633.openstacklocal,16020,1528109888044
Region Name = sp,100,1521182497105.6e87f8a4f3bf7c2762d644dba8e98022.
Encoded Region Name = 6e87f8a4f3bf7c2762d644dba8e98022
Non existing row key:
#java -cp `hbase classpath`: Regionfinder sp 10
Rowkey 10 is not exist in any region. It will be placed in region : sp,10,1521182497105.ecc8568cb94776ad83ee04cbb422bff0
... View more
- Find more articles tagged with:
- Data Processing
- FAQ
- HBase
Labels:
01-30-2018
12:56 PM
Appreciated @D Giri. Your step is perfect for changing the Journal directory without affecting any existing service. But my article has steps for doing it offline. Why we need to re-initialize? I haven't copied the edits and meta from old directory. So there will not be any Journal Layout. Let me add your steps in another section in the same article. Thank you for your contribution.
... View more
01-30-2018
07:57 AM
There are two sections which illustrate moving the Journal directory on the same Host.
You can follow one of the section according to your need.
Section 1: ( Service Downtime is not required )
1. Change the settings in Ambari for journal node edits directory(dfs.journalnode.edits.dir) from /hadoop/hdfs/journal/ to /data/1/journal/
2. Don't restart any services immediately.
3. Stop the journal node on
NODE1.
a. SSH to NODE1
b. sudo mkdir -p /data/1/journal
c. sudo chown hdfs:hadoop /data/1/journal
d. sudo rsync -ahvAX /hadoop/hdfs/journal/* /data/1/journal/
Start the journal node on
NODE1
4. Repeat step 3 for remaining two journal nodes NODE2 and NODE3.
5. Restart the required services accordingly (Rolling or All at once)
Section 2: ( Service Downtime is required )
1. Create the new directory on all Journal nodes.
Example:
# mkdir -p /data/1/journal/
2. Change ownership to "hdfs:hadoop" on all Journal nodes.
Example:
# chown hdfs:hadoop /data/1/journal/
3. Backup an exiting "Fsimage and Edits" from "
dfs.namenode.name.dir", "dfs.journalnode.edits.dir" directory on both Namenode and all Journal nodes.
4. Active Namenode - Perform the
saveNamespace which will Merge "FSImage and Edits".
# su - hdfs
$ hdfs dfsadmin -safemode enter
$ hdfs dfsadmin -saveNamespace
5. Stop Active Namenode and Standby Namenode.
6. Stop all Journal nodes.
6. Update, latest Journal node directory.
Ambari -> HDFS -> Configs -> Advanced -> Advanced hdfs-site -> dfs.journalnode.edits.dir
7. Start all Journal nodes.
8. Format JN directory.
# su - hdfs
$ hdfs namenode -initializeSharedEdits
Note: The command should be run on one of the
Namenode hosts.
9. Start Active Namenode.
Note:
Don't proceed next step until Namenode out of Safemode.
10. Bootstrap Standby Namenode.
# su - hdfs
$ hdfs namenode -bootstrapStandby
Note:
The command should be run on
Standby Namenode.
During, You will get the option to format the Storage directory. Press Y and hit enter.
Re-format filesystem in Storage Directory /hadoop/hdfs/namenode ? (Y or N) Y
11. Start Standby Namenode.
... View more
- Find more articles tagged with:
- Hadoop Core
- HDFS
- How-ToTutorial
- journalnode
- namenode
Labels:
11-23-2017
02:43 PM
@Maryem Mary It appears some logs got generated. Your directory size is 4KB now. #ls -lrth /var/log/zookeeper ??
... View more
11-23-2017
02:07 PM
Ok, ensure you have right permission & ownership of the directory. /var/log/zookeeper #ls -lrth /var/log/ | grep zookeeper drwxr-xr-x. 3 zookeeper hadoop 4.0K Nov 23 13:32 zookeeper Did you check? after the pattern update. Ambari -> Zookeeper -> Configs -> Advanced zookeeper-log4j, .. log4j.appender.ROLLINGFILE.MaxBackupIndex=10 log4j.appender.ROLLINGFILE.layout=org.apache.log4j.PatternLayout log4j.appender.ROLLINGFILE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n
... View more
11-23-2017
01:42 PM
It could be your configuration problem if you don't find any logs even after this setting. or you may be configured different log directory location. Share the output of "/etc/zookeeper/conf/zookeeper-env.sh" ? Also, you are missing the log patterns. Please append to log4j.properties as highlighted here. Ambari -> Zookeeper -> Configs -> Advanced zookeeper-log4j, .. log4j.appender.ROLLINGFILE.MaxBackupIndex=10 log4j.appender.ROLLINGFILE.layout=org.apache.log4j.PatternLayout log4j.appender.ROLLINGFILE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%C{1}@%L] - %m%n
... View more
11-23-2017
01:04 PM
@Maryem Mary Share the zookeeper log or error trace to find a cause, as above configuration for enabling the logs.
... View more
11-23-2017
12:23 PM
So you miss the zkServer.sh script??? can you share the output? #ls -lrth /usr/hdp/current/zookeeper-server/bin/
... View more