Member since
10-04-2017
13
Posts
0
Kudos Received
0
Solutions
12-19-2018
01:34 PM
I have installed Ambari with stand-alone database. I have invested a lot of time in setting the cluster (Spark dependencies for many projects, Yarn configurations, etc). I run ambari-server as root, HDP version is 2.6.3.0-235, OS is Ubuntu 16.04
I had experienced problems in stating the ambari-server (service ambari-server start), specifically, there were connection problems related to postgres. I found postgres-9.5 installed on that system and removed it (apt-get remove postgres-9.5).
Now, all the problems have started, after I have done that, ambari-server service was no longer available. The problem I got: Failed to start ambari-server.service: Unit ambari-server.service not found. Now, I have tried running ambari-server install again and hope it would repair the problem (I saw ambari-server files at many locations on the machine but the service definition was missing). I have stumbled upon the problems: apt-get install ambari-server
# resulted in the following error: Some packages could not be installed. This may mean that you haverequested an impossible situation or if you are using the unstabledistribution that some required packages have not yet been createdor been moved out of Incoming.The following information may help to resolve the situation:The following packages have unmet dependencies: ambari-server : Depends: postgresql (>= 8.1) but it is not going to be installedE: Unable to correct problems, you have held broken packages. apt-get upgrade ambari-server
# resulted in the following error: Some packages could not be installed. This may mean that you haverequested an impossible situation or if you are using the unstabledistribution that some required packages have not yet been createdor been moved out of Incoming.The following information may help to resolve the situation:The following packages have unmet dependencies: postgresql : Depends: postgresql-11 but it is not going to be installedE: Broken packages I tried manually installing postgres 9.5 as before and I still get the error: Depends: postgresql (>= 8.1) I was not able to install postgres-11 due to missing dependencies (libicu52) not found. What can I do to fix it?
... View more
Labels:
- Labels:
-
Apache Ambari
08-02-2018
06:29 AM
Hi @Felix Albani and thx for the answer, I have double checked and no executor instances configuration is set. More than that, no memory, cores or any such configuration is set for executor or driver. I do want to repeat that I run the calculations from Jupyter Notebook, and it feels like the allocation is the default configuration. I have tried running a program with spark-submit, and the results are the same
... View more
08-01-2018
06:45 AM
Hi everyone, have a cluster managed with Yarn and runs Spark jobs, the components
were installed using Ambari (2.6.3.0-235). I have 6 hosts each with 6
cores. I use Fair scheduler I want Yarn to automatically add/remove executor cores, but no matter what I do it doesn't work Relevant Spark configuration (configured in Ambari): <code>spark.dynamicAllocation.schedu<wbr>lerBacklogTimeout 10s
spark.dynamicAllocation.sustai<wbr>nedSchedulerBacklogTimeout 5s
spark.driver.memory 4G
spark.dynamicAllocation.enable<wbr>d true
spark.dynamicAllocation.initia<wbr>lExecutors 6 (has no effect - starts with 2)
spark.dynamicAllocation.maxExe<wbr>cutors 10
spark.dynamicAllocation.minExe<wbr>cutors 1
spark.scheduler.mode FAIR
spark.shuffle.service.enabled true
SPARK_EXECUTOR_MEMORY="36G" Relevant Yarn configuration (configured in Ambari): <code>yarn.nodemanager.aux-services mapreduce_shuffle,spark_shuffl<wbr>e,spark2_shuffle
YARN Java heap size 4096
yarn.resourcemanager.scheduler<wbr>.class org.apache.hadoop.yarn.server.<wbr>resourcemanager.scheduler.fair<wbr>.FairScheduler
yarn.scheduler.fair.preemption true
yarn.nodemanager.aux-services.<wbr>spark2_shuffle.class org.apache.spark.network.yarn.<wbr>YarnShuffleService
yarn.nodemanager.aux-services.<wbr>spark2_shuffle.classpath {{stack_root}}/${hdp.version}/<wbr>spark2/aux/*
yarn.nodemanager.aux-services.<wbr>spark_shuffle.class org.apache.spark.network.yarn.<wbr>YarnShuffleService
yarn.nodemanager.aux-services.<wbr>spark_shuffle.classpath {{stack_root}}/${hdp.version}/<wbr>spark/aux/*
Minimum Container Size (VCores) 0
Maximum Container Size (VCores) 12
Number of virtual cores 12 Also I followed the manual in and passed all the steps to configure external shuffle service, I copied the yarn-shuffle jar: cp /usr/hdp/2.6.3.0-235/spark/aux/spark-2.2.0.2.6.3.0-235-yarn-shuffle.jar /usr/hdp/2.6.3.0-235/hadoop-yarn/lib/ I see only 3 cores are allocated to the application (default executors is 2 so I guess its 2+driver) screenshot from the queue is attached, although many tasks are pending (screenshot added). I want to get to a point where Yarn starts with 3 cpu for every application, but when there are pending tasks more resources are allocated. If it it relevant, I use Jupyter Notebook and findspark to connect to the cluster: import findspark findspark.init() spark = SparkSession.builder.appName("internal-external2").getOrCreate() I would really appreciate any suggestion/help, there is no manual on that topic I didn't try. thx a lot, Anton
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN
01-23-2018
01:03 PM
@gnovak thx for the answer, I was able to make it work according to your explanation with one small addition. I did have to allow excessive queue usage with: yarn.scheduler.capacity.root.test1.user-limit-factor=2 I have another question, I tried to define test2 to be fair and submit two application to it, but no preemption has happened and one app simply took 100% of the queue. I tried posting from different users and everything but couldn't make it happen. Is there another configuration I need to set for it to work?
... View more
01-22-2018
02:46 PM
I try to run two Yarn queues where if only one queue is active it will consume all the resources and once a job will arrive to the second queue Yarn will preempt some of the resources of the first queue to start the second job. For that I follow the manual written YARN Preemption with Spark using a Fair Policy but for some reason only the leaf queues has the option of fair ordering policy. In fact the selection for ordering policy doesn't exist at all on non leaf queue. I am running on hdp version: 2.6.3.0-235 I have followed the instruction perfectly on the manual. I have also tried manually set the fair policy in 'Scheduler' tab of Yarn, when I do that Yarn crashes and would not start. What can I do? Screenshot are attached
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN
10-10-2017
12:06 PM
It was exactly the issue, only the version I had installed is 9.2. I followed the instructions for the pre-installed database and it worked perfectly. Thanks alot.
... View more
10-05-2017
07:41 AM
i am installing ambari-server with RHEL7.3 but unfortunately it fails, when I run it with verbose: `ambari-server setup -v` I am getting the following error: Traceback (most recent call last): File "/usr/sbin/ambari-server.py", line 951, in <module> mainBody() File "/usr/sbin/ambari-server.py", line 921, in mainBody main(options, args, parser) File "/usr/sbin/ambari-server.py", line 873, in main action_obj.execute() File "/usr/sbin/ambari-server.py", line 78, in execute self.fn(*self.args, **self.kwargs) File "/usr/lib/python2.6/site-packages/ambari_server/serverSetup.py", line 1144, in setup _setup_database(options) File "/usr/lib/python2.6/site-packages/ambari_server/serverSetup.py", line 960, in _setup_database dbmsAmbari.setup_database() File "/usr/lib/python2.6/site-packages/ambari_server/dbConfiguration.py", line 150, in setup_database self._setup_local_database() File "/usr/lib/python2.6/site-packages/ambari_server/dbConfiguration_linux.py", line 479, in _setup_local_database retcode, out, err = self._configure_postgres() File "/usr/lib/python2.6/site-packages/ambari_server/dbConfiguration_linux.py", line 748, in _configure_postgres PGConfig._configure_pg_hba_postgres_user() File "/usr/lib/python2.6/site-packages/ambari_server/dbConfiguration_linux.py", line 727, in _configure_pg_hba_postgres_user pg_hba_conf_file_content_in = sudo.read_file(PGConfig.PG_HBA_CONF_FILE) File "/usr/lib/python2.6/site-packages/resource_management/core/sudo.py", line 144, in read_file with open(filename, "rb") as fp: IOError: [Errno 2] No such file or directory: '/var/lib/pgsql/data/pg_hba.conf' As you can see Ambari searches the conf file in '/var/lib/pgsql/data/pg_hba.conf' where its actual location is: '/var/lib/pgsql/9.2/data/pg_hba.conf' I couldn't find any Ambari config for specifying that path, any help would be really appreciated.
... View more
Labels:
- Labels:
-
Apache Ambari
10-04-2017
09:15 PM
Thank you for the quick and accurate response! this solved the issue.
... View more
10-04-2017
07:00 PM
I try to install Ambari-server under RedHat 7.3, python 2.7.5 I got the following Ambari and HDP repositories and ran `yum clean all` just to be sure: http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.5.2.0/ambari.repo http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.2.0/hdp.repo After I run yum `install ambari-server` i try to setup it using `ambari-server setup` and I am getting the following issue: Traceback (most recent call last): File "/usr/sbin/ambari-server.py", line 40, in <module> from ambari_server.serverUpgrade import upgrade, upgrade_stack, set_current File "/usr/lib/python2.6/site-packages/ambari_server/serverUpgrade.py", line 51, in <module> from ambari_server.setupMpacks import replay_mpack_logs File "/usr/lib/python2.6/site-packages/ambari_server/setupMpacks.py", line 30, in <module> from ambari_commons.inet_utils import download_file, download_file_anyway ImportError: cannot import name download_file_anyway It is weird to me that the ambari-server packages are installed under python2.6 (which is not installed) since the RHEL7 version should work with python2.7. What is the issue? how can it be solved?
... View more
Labels:
- Labels:
-
Apache Ambari