Member since
11-17-2017
6
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1471 | 11-23-2017 02:30 AM |
11-23-2017
02:30 AM
At last, I solved this problem: I suspected the HDP version stored in postgres is wrong. So I modified these items: delete from servicedesiredstate where service_name='SPARK2';
delete from clusterservices where service_name='SPARK2';
delete from alert_grouping where definition_id = 201;
delete from alert_grouping where definition_id = 202;
delete from alert_grouping where definition_id = 203;
delete from alert_definition where service_name = 'SPARK2';
select * from stack;
stack_id | stack_name | stack_version
----------+-------------+-----------------
1 | HDP | 2.0.6.GlusterFS
2 | HDP | 2.5
3 | HDP | 2.4
4 | HDP | 2.3.ECS
5 | HDP | 2.0.6
6 | HDP | 2.1.GlusterFS
7 | HDP | 2.3
8 | HDP | 2.3.GlusterFS
9 | HDP | 2.2
10 | HDP | 2.1
11 | HDP | 2.0
51 | HDP | 2.6
101 | BigInsights | 4.2.5
102 | BigInsights | 4.1
103 | BigInsights | 4.0
104 | BigInsights | 4.2
update clusters set desired_stack_id = 51 where cluster_id = 2; Note that the digits above are referred to table alert_grouping and stack. So in my case, the desired_stack_id in table clusters was 3 (means HDP2.4). I updated it to 51(means HDP2.6) and corrected the undefined alert definition. At last the ambari server startup and recognize service Spark2 in HDP2.6. If someone can offer more details in ambari-server metadata in database, it will be easier to locate a problem like this.
... View more
11-22-2017
07:11 AM
Additional progress: I use ambari rest api to add spark2 service. The http request return 200 and I can figure it out spark2 by get method. But after I restart ambari-server, it failed to startup.The log: 22 Nov 2017 14:57:25,693 ERROR [main] ClusterImpl:431 - Can not get service info: stackName=HDP, stackVersion=2.4, serviceName=SPARK2
...
org.apache.ambari.server.ServiceNotFoundException: Service not found, clusterName=solar, serviceName=SPARK2
at org.apache.ambari.server.state.cluster.ClusterImpl.getService(ClusterImpl.java:864)
at org.apache.ambari.server.api.services.AmbariMetaInfo.reconcileAlertDefinitions(AmbariMetaInfo.java:1240)
at org.apache.ambari.server.controller.AmbariServer.run(AmbariServer.java:532)
at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1067) The log show the HDP stackVersion=2.4, But I am sure that I install HDP2.6 in all nodes and hdp-select prove it. Maybe I can modify this HDP version check mistake?
... View more
11-21-2017
06:52 AM
After upgrading Ambari from 2.2.2 to 2.6.0.0 and HDP from 2.4.3 to 2.6.3.0, I tried adding service spark2. I followed this guide , tried to add new service but could not find spark2. I checked the services in Admin -> Stack and Version and No spark2 there but I found Spark2 in Admin -> Manage Ambari -> Versions -> HDP2.6.3.0. And hdp-select also shows spark2 is in hdp repo By the way, I also tried install spark2 through yum install spark2-master spark2-python( I use python as main develop lang). But pyspark shows: File "/usr/hdp/current/spark2-client/python/pyspark/shell.py", line 33, in <module>
from pyspark.sql import SparkSession, SQLContext
ImportError: cannot import name SparkSession
If I can not install spark2 in Ambari, I want to use spark2-client at lest. The screenshots following:
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Spark
11-17-2017
11:58 AM
@Jay Kumar SenSharma It works! I downgrade Ambari to 2.5.1 and run 'ambari-server upgrade'. After successful completion, I upgrade Ambari to 2.6.0 and run 'ambari-server upgrade' again. This time the server startup without problem. Thank you for the solution.
... View more
11-17-2017
08:20 AM
Not exactly, I'm using HDP 2.4.3. Maybe I shoud also upgrade the Ambari to 2.4 at first?
... View more
11-17-2017
07:50 AM
I'm trying upgrade ambari-server and ambari-agent from 2.2.2 to 2.6 in Centos 7. Refered to official document , I progressed smoothly until the command "ambari-server upgrade",And the /var/log/ambari-server/ambari-server.log shows: 17 Nov 2017 14:03:37,055 INFO [main] LockFactory:53 - Lock profiling is disabled
17 Nov 2017 14:03:37,062 INFO [main] SchemaUpgradeHelper:394 - Upgrading schema to target version = 2.6.0.0
17 Nov 2017 14:03:37,082 INFO [main] SchemaUpgradeHelper:403 - Upgrading schema from source version = 2.2.2
17 Nov 2017 14:03:37,084 ERROR [main] SchemaUpgradeHelper:437 - Exception occurred during upgrade, failed
org.apache.ambari.server.AmbariException: Database version does not meet minimal upgrade requirements. Expected version should be not less than 2.4.0, current version is 2.2.2
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:408)
17 Nov 2017 14:05:09,089 INFO [main] AmbariServer:207 - Found org/apache/ambari/server/controller/AmbariServer.class class in file:/usr/lib/ambari-server/ambari-server-2.6.0.0.267.jar!/org/apache/ambari/server/controller/AmbariServer.class
Ambari-server 2.2.2
HDP 2.4.3.0-227
Postgres 9.2.18
It looks like that the least version before upgrading to 2.6 is 2.4. But I notice this page imply the version after 2.2.0 is valid for upgrading. Someone have an idea?
... View more
Labels:
- Labels:
-
Apache Ambari