Member since
08-08-2017
1652
Posts
30
Kudos Received
11
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1923 | 06-15-2020 05:23 AM | |
| 15500 | 01-30-2020 08:04 PM | |
| 2076 | 07-07-2019 09:06 PM | |
| 8122 | 01-27-2018 10:17 PM | |
| 4575 | 12-31-2017 10:12 PM |
11-15-2019
02:40 AM
ok do you can summarize all options to recover the namenode ( also option out from the box )
... View more
11-15-2019
12:45 AM
hi all
I want to ask this important question
lets say we have the following:
HDP cluster with :
3 masters machine ( active/standby name-node ) , ( active/standby resource manager )
3 datanode machines
- each data-node machine have 4 disks for HDFS ( not include the OS )
3 kafka machines
- each kafka machine have one disk of 10T ( not include the OS )
now we want to install from scratch all cluster include HDP and ambari
but save the data on datanode machines and kafka topics data machine by the following:
we umount the disks on datanode machines and kafka machines
example
on datanode machine ( note - /etc/stab is already configured )
umount /grid/data1
umount /grid/data2
.
.
.
so the second scratch installation we install all the cluster ( by blueprint ) , but without data-node HDFS disks , and kafka topic disk ( scratch installation means fresh new linux OS )
after installation we mount all data-node machines disks and kafka disks machines ( where we are store all topics )
example
on datanode machine ( note - /etc/stab is already configured )
mount /grid/data1
mount /grid/data2
.
.
.
in order to complete the picture , need to restart HDFS and YARN and kafka
so - is this scenario could to work ?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Kafka
11-14-2019
01:01 PM
hi lets say on the faulty namenode we see only the following: under - /hadoop/hdfs/journal/hdfsha/current example: -rw-r--r-- 1 hdfs hadoop 155 Nov 13 00:50 VERSION -rw-r--r-- 1 hdfs hadoop 3 Nov 14 21:24 last-promised-epoch drwxr-xr-x 2 hdfs hadoop 4096 Nov 14 21:24 paxos -rw-r--r-- 1 hdfs hadoop 3 Nov 14 21:24 last-writer-epoch -rw-r--r-- 1 hdfs hadoop 8 Nov 14 23:16 committed-txid as you know usually we have also the files like - edits_0000000000000066195-0000000000000066242 is this scenario change the picture?
... View more
11-13-2019
02:03 PM
you means the files /hadoop/hdfs/namenode/current/fsimage_* should be the same on both nodes exactly ? in this case its easy to copy from the good namenode to the bad namenode and restart the bad namenode let me know if this is the procedure?
... View more
11-13-2019
01:47 PM
sure we can backup both folders in tar file but what is your suggestion on the node without the fsimage files?
... View more
11-13-2019
01:06 PM
hi the second name node have the fsimage files , but from ambari the second namenode not appears as standby/active is just up
... View more
11-13-2019
05:16 AM
we have two namenode machines ( are part of HDP cluster in ambari )
because electricity failure , we notices about the following
on one name node we see that fsimage_xxxx files are missing while on the second namenode they are exists
is it possible to re-create them on the faulty name node
example on the bad node
ls /hadoop/hdfs/namenode/current | grep fsimage_ no output
on the good namenode
ls /hadoop/hdfs/namenode/current | grep fsimage_ fsimage_0000000000044556627 fsimage_0000000000044556627.md5 fsimage_0000000000044577059 fsimage_0000000000044577059.md5
the status for now is that name-node service not startup successfully from ambari
and the logs from the faulty name-node say like this:
ERROR namenode.NameNode (NameNode.java:main(1783)) - Failed to start namenode. java.io.FileNotFoundException: No valid image files found
... View more
Labels:
11-12-2019
07:08 AM
by the following hdfs fsck / -files -blocks -locations | grep blk_xxxxxx_xxxxxx as: su hdfs hdfs fsck / -files -blocks -locations | grep blk_1081495827_7755233 we not get any results so I guess its mean that blk_xxxxx_xxxx isnt exist in HDFS file-system what next ?
... View more
11-12-2019
01:55 AM
please send me the fsck cli that you want me to run
... View more
11-12-2019
01:52 AM
we also do the following su hdfs hadoop fsck / -files -blocks >/tmp/file and we bot found the block - blk_1081495827_7755233 in the file - /tmp/file so what is the reason that block removed?
... View more