Member since
08-08-2017
1604
Posts
18
Kudos Received
11
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
234 | 06-15-2020 05:23 AM | |
1018 | 01-30-2020 08:04 PM | |
393 | 07-07-2019 09:06 PM | |
2396 | 01-27-2018 10:17 PM | |
1243 | 12-31-2017 10:12 PM |
03-21-2021
10:37 PM
about the API Ambari command cli , can you show me the full syntax that replace the disable to enable
... View more
03-21-2021
08:30 AM
in Ambari we have the following feature that is disable , under yarn Configs what is the relevant ambari rest api in order to change the state of CPU Scheduling from disable to enable ?
... View more
Labels:
02-16-2021
06:47 AM
hi all we are trying to download the ambari version 2.6.1 but without success ( according to https://docs.hortonworks.com.s3.amazonaws.com/HDPDocuments/HDF3/HDF-3.1.1/bk_installing-hdf-on-hdp-ppc/content/ambari_repositories.html ) wget http://public-repo-1.hortonworks.com/ambari/centos7-ppc/2.x/updates/2.6.1.0 --2021-02-16 14:44:08-- http://public-repo-1.hortonworks.com/ambari/centos7-ppc/2.x/updates/2.6.1.0 Resolving public-repo-1.hortonworks.com (public-repo-1.hortonworks.com)... 13.225.255.100, 13.225.255.128, 13.225.255.124, ... Connecting to public-repo-1.hortonworks.com (public-repo-1.hortonworks.com)|13.225.255.100|:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2021-02-16 14:44:09 ERROR 403: Forbidden. any idea why this version cant be download? but we can download from other site as wget http://archive.apache.org/dist/ambari/ambari-2.6.1/apache-ambari-2.6.1-src.tar.gz
... View more
Labels:
02-11-2021
06:22 AM
can you described more about - "T he rebalance (by Blockpool" we have HDP cluster with ambari so not sure what we need to do
... View more
01-20-2021
10:27 PM
We have ambari cluster , HDP version 2.6.5 Cluster include management of two name-node ( one is active and the secondary is standby ) And 65 datanode machines We have problem with the standby name-node that not started and from the namenode logs we Can see the following 2021-01-01 15:19:43,269 ERROR namenode.NameNode (NameNode.java:main(1783)) - Failed to start namenode. java.io.IOException: There appears to be a gap in the edit log. We expected txid 90247527115, but got txid 90247903412. from ambari we can see For now the active namenode is up but the standby name node is down , and the root cause for This issue is because **namenode matadata is damaged/corrupted.** So we have two solution - A or B A) run the following recover on standby namenode su hadoop namenode -recover B) Put Active NN in safemode su hdfs hdfs dfsadmin -safemode enter Do a savenamespace operation on Active NN su hdfs hdfs dfsadmin -saveNamespace Leave Safemode su hdfs hdfs dfsadmin -safemode leave Login to Standby NN Run below command on Standby namenode to get latest fsimage that we saved in above steps. su hdfs hdfs namenode -bootstrapStandby -force what is the preferred solution ( solution A or Solution B ) for our problem?
... View more
Labels:
01-19-2021
11:07 AM
we have Hadoop cluster with 2 name-nodes ( active standby ) and 12 data nodes all 12 data-nodes machines have disks for HDFS we are before the action of `hadoop namenode -recover` , and that because we suspect about corrupted files as fsimage_0000000000001253918 or edits_0000000000001203337-0000000000001214475 etc so to recover the hdfs meta data we can do the following $ hadoop namenode -recover DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it. 21/01/19 17:56:35 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: user = hdfs STARTUP_MSG: host = master1.sys67.com/17.2.12.78 STARTUP_MSG: args = [-recover] STARTUP_MSG: version = 2.7.3.2.6.5.0-292 21/01/19 17:56:35 INFO namenode.NameNode: createNameNode [-recover] You have selected Metadata Recovery mode. This mode is intended to recover lost metadata on a corrupt filesystem. Metadata recovery mode often permanently deletes data from your HDFS filesystem. Please back up your edit log and fsimage before trying this! Are you ready to proceed? (Y/N) (Y or N) y the question is: dose this action could also affected the data itself on the data-nodes machines ? or only the meta data on namenode machines?
... View more
Labels:
01-19-2021
09:10 AM
we have ambari cluster , HDP version `2.6.5` cluster include management of two name-node ( one is active and the secondary is standby ) and 65 datanode machines we have problem with the standby name-node that not started and from the namenode logs we can see the following 2021-01-01 15:19:43,269 ERROR namenode.NameNode (NameNode.java:main(1783)) - Failed to start namenode. java.io.IOException: There appears to be a gap in the edit log. We expected txid 90247527115, but got txid 90247903412. at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:94) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:215) for now the active namenode is up but the standby name node is down regarding to java.io.IOException: There appears to be a gap in the edit log. We expected txid 90247527115, but got txid 90247903412. what is the preferred solution to fix this problem?
... View more
Labels:
01-17-2021
09:21 AM
we have HDP cluster version `2.6.5` with `8` data nodes , all machines are installed on rhel 7.6 version HDP cluster is based amabri platform version - `2.6.1` each data-node ( worker machine ) include two disks and each disk size is 1.8T when we access the data-node machines we can see differences between the size of the disks for example on the first data-node the size is : ( by `df -h` ) /dev/sdb 1.8T 839G 996G 46% /grid/sdc /dev/sda 1.8T 1014G 821G 56% /grid/sdb on the second data-node the size is: /dev/sdb 1.8T 1.5T 390G 79% /grid/sdc /dev/sda 1.8T 1.5T 400G 79% /grid/sdb on the third data-node th size is: /dev/sdb 1.8T 1.7T 170G 91% /grid/sdc /dev/sda 1.8T 1.7T 169G 91% /grid/sdb and so on the big question is why HDFS not perform the re-balance on the HDFS disks? *for example expected results on all disks should be with the same size on all datanodes machines* why is the used size differences between `datanode1` to `datanode2` to `datanode3` etc ? any advice about the tune parameters in HDFS that can help us? *because its very critical when one disk is reached `100%` size and the other are more small as `50%`*
... View more
Labels:
01-17-2021
09:19 AM
we installed small HDP cluster with one data-node machine HDP version is `2.6.5` and ambari version is `2.6.1` so this is new cluster that contain two name-node with only one data-node ( worker machine ) the interesting behavior that we see is that increasing of `under replica` on ambari dashboard , for now the number is `15000` under replica blocks as we know the most root cause of this problem is network issues between name node to data-node but this isn't the case in our hadoop cluster we can also decrease the under replica by the following procedure su - <$hdfs_user> bash-4.1$ hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files -bash-4.1$ for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ; hadoop fs -setrep 3 $hdfsfile; done but we not want to do it because under replica problem should not happens from beginning and maybe need to tune some HDFS parameters , but we not sure about this please let us know about any advice that can help us
... View more
Labels:
10-20-2020
01:45 PM
hi all
we have ambari cluster with HDP version 2.6.5
from ambari dashboard we see the following
we can see that YARN show 111% - this isn't logical value
instead to show X from 100%
what could be the problem here ?
... View more
10-14-2020
10:16 AM
hi all
the following post in you-tube described how to add the hbase in ambari with all component on specific hosts
https://www.youtube.com/watch?v=BsvLoPE0XOk
what we want to do is exactly the same but with ambari rest API
since the target is to use script and create by REST API the Hbase service with Hbase component as the following example
we will happy to get examples of REST API
... View more
Labels:
10-14-2020
08:48 AM
now I want to add the component as HBASE master and RegionServers and Phoenix Query Servers
... View more
10-14-2020
08:47 AM
this is what I have now in ambari
... View more
10-14-2020
08:44 AM
actually what I want to do is to use API to install HBASE , with all hbase component so when you wrote the OOZIE procedure , what is the API to add HBASE insted OOZIE?
... View more
10-14-2020
08:42 AM
I have already the files ls /etc/hbase/conf/ hadoop-metrics2-hbase.properties hbase-env.cmd hbase-env.sh hbase-policy.xml hbase-site.xml log4j.properties regionservers
... View more
10-14-2020
08:24 AM
I have another issue that is related but it happens when I install component from amabri from host I do --> Add --> install hbase master but I get the following Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/hook.py", line 37, in <module>
BeforeInstallHook().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/hook.py", line 28, in hook
import params
File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/params.py", line 110, in <module>
hbase_user_dirs = format("/home/{hbase_user},/tmp/{hbase_user},/usr/bin/{hbase_user},/var/log/{hbase_user},{hbase_tmp_dir}")
File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/format.py", line 95, in format
return ConfigurationFormatter().format(format_string, args, **result)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/format.py", line 59, in format
result_protected = self.vformat(format_string, args, all_params)
File "/usr/lib64/python2.7/string.py", line 549, in vformat
result = self._vformat(format_string, args, kwargs, used_args, 2)
File "/usr/lib64/python2.7/string.py", line 582, in _vformat
result.append(self.format_field(obj, format_spec))
File "/usr/lib64/python2.7/string.py", line 599, in format_field
return format(value, format_spec)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'hbase-env' was not found in configurations dictionary!
... View more
10-14-2020
06:39 AM
we have presto cluster with 35 presto Linux workers nodes we want to add new jar - `phoenix-server-client-4.7.0-HBase-1.1.jar` to `plugin/phoenix` folder while the current jars that related to phoenix in this folder are phoenix-client-4.14.1-HBase-1.4-embedded.jar presto-phoenix-317.jar while all the other jars in folder comes with scratch presto installation we try to add only the jar - `phoenix-server-client-4.7.0-HBase-1.1.jar` to folder - `plugin/phoenix` but when we send new presto query we get the error about Newer Phoenix clients can't communicate with older Phoenix servers. The following servers require an updated phoenix-[version]-server.jar to be put in the class path of HBase:
... View more
Labels:
10-13-2020
02:13 PM
we have ambari cluster and HDP version 2.6.5
we want to add the service OOZIE on specific nodes in our cluster
for example - we have 10 workers machines in the HDP cluster
so we want to add he service OOZIE on each worker machine
from documentation we found the following API that really add the service to ambari
curl --user admin:admin -H "X-Requested-By: ambari" -i -X POST -d '{"ServiceInfo":{"service_name":"OOZIE"}}' http://localhost:8080/api/v1/clusters/HDP/services
but we not succeeded to improve the API in order to install the service OOZIE on each worker node
the final target is to add the service to amabri while each worker node in amabri should have the service OOZIE
any ideas how to continue from this stage?
... View more
10-12-2020
11:23 AM
hi all we have HDP cluster ( version 2.6.4 ) , with 98 DataNode machines we want to add the HBase service on each of DataNode machines can we add the HBase services by REST API on all DataNode machines ? example from ambari
... View more
Labels:
10-12-2020
02:47 AM
little question why just not stop the service - HDFS on each new data node and set it to maintenance mode ?
... View more
10-09-2020
12:59 AM
you said that not need to run it , but the post that I mentioned say to run it , so what is right
... View more
10-09-2020
12:56 AM
hi all we have HDP 2.6.4 cluster with 245 workers machines each worker have ( datanode and resource manager ) we want to add 10 new workers machines to the cluster but we want to disable the datanode machines so no data will transfer from the old datanodes to the new datanodes I thinking to do maintenance mode on the new datanode , but not sure if this action is enough in order to disable the datanodes machine on the new workers
... View more
Labels:
09-15-2020
01:57 AM
hi all
recently we configured our small HDP clusters according to this post
https://community.cloudera.com/t5/Community-Articles/Write-or-Append-failures-in-very-small-Clusters-under-heavy/ta-p/245446
our problem is that many small HDP clusters have under replica problem or corrupted blocs
after implementation of the post - https://community.cloudera.com/t5/Community-Articles/Write-or-Append-failures-in-very-small-Clusters-under-heavy/ta-p/245446
we see improvement , and after fix under replica , the problem about under replica not returned
since this post is talking about small clusters
then I must to ask if we can do the configuration as mention in the post also on large clusters?
... View more
Labels:
09-13-2020
09:17 AM
hi all We are performing now the change hostname configuration on production cluster according to the document - https://docs.cloudera.com/HDPDocuments/Ambari-2.7.0.0/administering-ambari/content/amb_changing_host_names.html The last stage is talking about – "in case NameNode HA enabled , then need to run the following command on one of the name node" hdfs zkfc -formatZK -force since we have active name node and standby name node we assume that our namenode is HA enable ? but we want to understand what are the risks when doing the following cli on one of the namenode hdfs zkfc -formatZK -force is the below command is safety to run without risks ?
... View more
Labels:
09-13-2020
09:08 AM
thank you for the post but another question - according to the document - https://docs.cloudera.com/HDPDocuments/Ambari-2.7.0.0/administering-ambari/content/amb_changing_host_names.html The last stage is talking about – in case NameNode HA enabled , then need to run the following command on one of the name node hdfs zkfc -formatZK -force thank you for the post but since we have active name node and standby name node we assume that our namenode is HA enable example from our cluster but we want to understand what are the risks when doing the following cli on one of the namenode hdfs zkfc -formatZK -force is the below command is safety to run without risks ?
... View more
09-08-2020
01:42 PM
We have HDP cluster version `2.6.5` and ambari `2.6.1` version
Cluster include 3 masters machines , and 211 data-nodes machines ( workers machines ) , all machines are `rhel 7.2` version
Example
master1.sys77.com , master2.sys77.com , master3.sys77.com …
And data nodes machines as
worker01.sys77.com , worker02.sys77.com ----> worker211.sys77.com
Now we want to change the domain name to `bigdata.com` instead of `sys77.com`
What is the procedure to replace the `domain name` (`sys77.com`) for Hadoop cluster ? ( HDP cluster with ambari )
... View more
08-13-2020
02:04 PM
another question - lets say the last snapshot is corrupted , then how zookeeper know to take the good snapshot before the last ?
... View more
08-13-2020
01:59 PM
can you also explain the differences between snapshot to log in zookeeper under Version-2 ?
... View more
08-13-2020
01:57 PM
so if you not recommended on 3 backup ( I feel you recommended more then 3 ) , then what is the count of backup that we can sleep well -:)
... View more
08-13-2020
12:45 PM
ZooKeeper server creates snapshot and log files, but never deletes them. So we need to care about the retention policy. How to decide on the right amount of remaining Zookeeper snapshot files? Need to say that ZooKeeper server itself only needs the latest complete fuzzy snapshot and the log files from the start of that snapshot. But since ZooKeeper creates a backup of snapshot file, how many ZooKeeper snapshot backups do we need to retain? Sometimes snapshots can be corrupted, so the backup of snapshot files should take this into consideration. In our ZooKeeper server we saw that snapshot backup is created each day. Example of snapshot file from my ZooKeeper server: -rw-r--r-- 1 ZooKeeper hadoop 458138861 Aug 10 07:12 snapshot.19000329d1 -rw-r--r-- 1 ZooKeeper hadoop 458138266 Aug 10 07:13 snapshot.19000329de -rw-r--r-- 1 ZooKeeper hadoop 458143590 Aug 10 09:24 snapshot.1900032d7a -rw-r--r-- 1 ZooKeeper hadoop 458142983 Aug 10 09:25 snapshot.1900032d84 -rw-r--r-- 1 ZooKeeper hadoop 458138686 Aug 11 03:42 snapshot.1900034b74 -rw-r--r-- 1 ZooKeeper hadoop 458138686 Aug 12 01:51 snapshot.1900036fa3 -rw-r--r-- 1 ZooKeeper hadoop 458138079 Aug 12 03:03 snapshot.1900037196 -rw-r--r-- 1 ZooKeeper hadoop 458138686 Aug 12 03:08 snapshot.19000371c8 -rw-r--r-- 1 ZooKeeper hadoop 458138432 Aug 12 03:09 snapshot.19000371de -rw-r--r-- 1 ZooKeeper hadoop 458138091 Aug 12 12:02 snapshot.1900038053 -rw-r--r-- 1 ZooKeeper hadoop 458138091 Aug 12 18:04 snapshot.1900038a39 -rw-r--r-- 1 ZooKeeper hadoop 458138091 Aug 13 13:01 snapshot.190003a923 -rw-r--r-- 1 ZooKeeper hadoop 2 Aug 13 13:01 currentEpoch -rw-r--r-- 1 ZooKeeper hadoop 67108880 Aug 13 21:17 log.190002d2ce
... View more
Labels: