Member since
01-18-2016
5
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
909 | 02-01-2016 10:07 AM |
09-12-2017
09:23 AM
Hi, I'm answering on behalf of myself and @Elias Abacioglu, (we work together). We noticed that each time we retried the installation, the "Duplicate entry '1' for key 'PRIMARY'" would increase, so next time it said "2". We figured that there was some inconsistent state in the 'ID' field, which we saw when looking at the mariadb general_log from the installation attempt. The mysql query that ambari issued was: INSERT INTO servicecomponentdesiredstate (id, component_name, desired_state, desired_version, recovery_enabled, service_name, cluster_id, desired_stack_id) VALUES (6, 'LIVY2_SERVER', 'INIT', 'UNKNOWN', 0, 'SPARK2', 2, 151)
Notice that 'id' get value '6' this time, and this of course failed with error-message: Duplicate entry '6'for key 'PRIMARY' The relevant columns of the servicecomponentdesiredstate table looked like this: MariaDB [ambari]> select component_name,id from servicecomponentdesiredstate order by id asc;
+-------------------------+----+
| component_name | id |
+-------------------------+----+
| APP_TIMELINE_SERVER | 1 |
| DATANODE | 2 |
| HBASE_CLIENT | 3 |
| HBASE_MASTER | 4 |
| HBASE_REGIONSERVER | 5 |
| HCAT | 6 |
| HDFS_CLIENT | 7 |
| HISTORYSERVER | 8 |
| HIVE_CLIENT | 9 |
| HIVE_METASTORE | 10 |
| HIVE_SERVER | 11 |
| JOURNALNODE | 12 |
| MAPREDUCE2_CLIENT | 13 |
| METRICS_COLLECTOR | 14 |
| METRICS_GRAFANA | 15 |
| METRICS_MONITOR | 16 |
| MYSQL_SERVER | 17 |
| NAMENODE | 18 |
| NFS_GATEWAY | 19 |
| NODEMANAGER | 20 |
| OOZIE_CLIENT | 21 |
| OOZIE_SERVER | 22 |
| PHOENIX_QUERY_SERVER | 23 |
| PIG | 24 |
| RESOURCEMANAGER | 25 |
| SECONDARY_NAMENODE | 26 |
| SPARK_CLIENT | 27 |
| SPARK_JOBHISTORYSERVER | 28 |
| SPARK_THRIFTSERVER | 29 |
| TEZ_CLIENT | 30 |
| WEBHCAT_SERVER | 31 |
| YARN_CLIENT | 32 |
| ZKFC | 33 |
| ZOOKEEPER_CLIENT | 34 |
| ZOOKEEPER_SERVER | 35 |
+-------------------------+----+
We then figured that if we get the pointer for 'id', where ever it came from, to get to '35', installation would succeed. So Elias ran the following command: CURL -X POST http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/services/SPARK2/components/LIVY2_SERVER until pointer was right: Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '33' for key 'PRIMARY'
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '34' for key 'PRIMARY'
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '35' for key 'PRIMARY' And at this point, we retried the full installation of Spark2 from Ambari UI, and it worked as expected! We have no clue at this point where or why the counter for 'id', at the insert into the servicecomponentdesiredstate table came from. If someone with deeper understanding of the ambari database-connections care to give us a hint, we can check it out.
... View more
02-01-2016
01:14 PM
Hi Jonas, do you have any update for when your collegues can have fixed the repositories? I checked on Friday, and the ubuntu14 download tar.gz still contain ubuntu12 path/files and i'm having problem getting this repository to "validate" in ambari for use with ubuntu14, so effectively I can't use it, and ambari will default to download from your public repository which is very slow for me. (But maybe this is an Ambari issue, and not repository issue)
... View more
02-01-2016
10:07 AM
1 Kudo
I'm not sure if we've been doing it wrong, but after changing our yarn jobs the problem went away, could have been an issue at our end. Although I still have this feeling in the back of my head that the "yarn.resourcemanager.scheduler.address" should be something else than a specific node, it should be the "yarn-cluster" address. But this might be an ambari-problem. Thanks for your time!
... View more
01-25-2016
01:42 PM
Thanks for a swift reply, however there are no pending restarts due to stale configs, and the service checks completes without any issues detected. None the less i restarted all yarn-services and re-ran the service-check, but no change in behaviour or config content. I had similar issues with the namenode rpc-address after enabling NN-HA, that dissapeared after updating from HDP-2.3.2.0 to HDP-2.3.4.0, hence i suspected this to be a bug. So, problem remains.
... View more
01-25-2016
01:29 PM
I've got a problem with my resourcemanager in yarn-site.cml config after enabling RM high availability. First I had a resource manager at Server1, then when I installed Server2, and enabled RM HA, Server2 became the "active" resourcemanager. My problem is that Ambari leaves the "old" configs of "yarn.resourcemanager.scheduler.address" set to "Server1" when adding the "yarn.resourcemanager.cluster-id" and other "yarn.resourcemanager.hostname.rm1" ..rm2 etc entries under the Custom yarn-site field in Ambari to contain both Server1 and Server2. When I try to run my yarn apps they try to connect to Server1 port 8030, which it's not listening on, since it is "Standby Resourcemanager". I'd expect it to connect to the "yarn-cluster" name as defined in "yarn.resourcemanager.cluster-id", not specific hostname of the old Resource manager. Is there something i'm overlooking in ambari configs, or is this a (known) bug?
... View more
Labels:
- Labels:
-
Apache YARN
-
Cloudera Manager