Member since
09-18-2015
3274
Posts
1159
Kudos Received
426
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2747 | 11-01-2016 05:43 PM | |
| 9412 | 11-01-2016 05:36 PM | |
| 5113 | 07-01-2016 03:20 PM | |
| 8577 | 05-25-2016 11:36 AM | |
| 4723 | 05-24-2016 05:27 PM |
01-30-2016
01:16 PM
See this explanation. When the NameNode starts up, it reads the FsImage and EditLog from disk, applies all the transactions from the EditLog to the in-memory representation of the FsImage, and flushes out this new version into a new FsImage on disk. It can then truncate the old EditLog because its transactions have been applied to the persistent FsImage. This process is called a checkpoint
... View more
01-30-2016
01:15 PM
@Avinash C Yes , new fsimage old fsimage + edit logs = new fsimage
... View more
01-30-2016
01:09 PM
@Avinash C See this line "The Checkpoint node stores the latest checkpoint in a directory that is structured the same as the NameNode’s directory. This allows the checkpointed image to be always available for reading by the NameNode if necessary. See Import checkpoint." Exact read on check point https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Checkpoint_Node Please do read the HDFS user guide - 20 -30 minutes read but it will save you lot of time.
... View more
01-30-2016
01:00 PM
@vishnu rao That sounds like a good work around. I am focusing on this message "java.net.SocketTimeoutException: Read timed out" Red time out can occur when there is network latency. Nice explanation For example: Let's say everything is working fine and load in the node is very high , connection is trying to read the data for metric and it cant because there is no more room to do the processing.
... View more
01-30-2016
12:55 PM
@Adrian Savory You can use ssh or browser http://127.0.0.1:4200 or ssh -p 2222 root@127.0.0.1
... View more
01-30-2016
12:54 PM
@Avinash C Details link
... View more
01-30-2016
12:51 PM
@Avinash C
See this description "The secondary NameNode merges the fsimage and the edits log files periodically and keeps edits log size within a limit. It is usually run on a different machine than the primary NameNode since its memory requirements are on the same order as the primary NameNode." The Checkpoint node periodically creates checkpoints of the namespace. It downloads fsimage and edits from the active NameNode, merges them locally, and uploads the new image back to the active NameNode. The Checkpoint node usually runs on a different machine than the NameNode since its memory requirements are on the same order as the NameNode. The Checkpoint node is started by bin/hdfs namenode -checkpoint on the node specified in the configuration file. The location of the Checkpoint (or Backup) node and its accompanying web interface are configured via the dfs.namenode.backup.address and dfs.namenode.backup.http-addressconfiguration variables. The start of the checkpoint process on the Checkpoint node is controlled by two configuration parameters. dfs.namenode.checkpoint.period, set to 1 hour by default, specifies the maximum delay between two consecutive checkpoints dfs.namenode.checkpoint.txns, set to 1 million by default, defines the number of uncheckpointed transactions on the NameNode which will force an urgent checkpoint, even if the checkpoint period has not been reached. The Checkpoint node stores the latest checkpoint in a directory that is structured the same as the NameNode's directory. This allows the checkpointed image to be always available for reading by the NameNode if necessary. See Import checkpoint.
... View more
01-30-2016
12:19 AM
@Michael Young Thanks for testing. I believe we have the answer 😉 Thanks @Josh Elser for chiming in ...Really appreciate it!!!! All comments upvote
... View more
01-30-2016
12:00 AM
1 Kudo
@Michael Young @Josh Elser I think it has to do with zk shutdown hbase and RS and run this hbase clean --cleanZk
... View more
01-29-2016
11:39 PM
That link has example of using sqlline.py ..Let me reproduce this again. Can you share your connecting string ? @Michael Young
... View more