Hi, I'm answering on behalf of myself and @Elias Abacioglu, (we work together). We noticed that each time we retried the installation, the "Duplicate entry '1' for key 'PRIMARY'" would increase, so next time it said "2". We figured that there was some inconsistent state in the 'ID' field, which we saw when looking at the mariadb general_log from the installation attempt. The mysql query that ambari issued was: INSERT INTO servicecomponentdesiredstate (id, component_name, desired_state, desired_version, recovery_enabled, service_name, cluster_id, desired_stack_id) VALUES (6, 'LIVY2_SERVER', 'INIT', 'UNKNOWN', 0, 'SPARK2', 2, 151)
Notice that 'id' get value '6' this time, and this of course failed with error-message: Duplicate entry '6'for key 'PRIMARY' The relevant columns of the servicecomponentdesiredstate table looked like this: MariaDB [ambari]> select component_name,id from servicecomponentdesiredstate order by id asc;
+-------------------------+----+
| component_name | id |
+-------------------------+----+
| APP_TIMELINE_SERVER | 1 |
| DATANODE | 2 |
| HBASE_CLIENT | 3 |
| HBASE_MASTER | 4 |
| HBASE_REGIONSERVER | 5 |
| HCAT | 6 |
| HDFS_CLIENT | 7 |
| HISTORYSERVER | 8 |
| HIVE_CLIENT | 9 |
| HIVE_METASTORE | 10 |
| HIVE_SERVER | 11 |
| JOURNALNODE | 12 |
| MAPREDUCE2_CLIENT | 13 |
| METRICS_COLLECTOR | 14 |
| METRICS_GRAFANA | 15 |
| METRICS_MONITOR | 16 |
| MYSQL_SERVER | 17 |
| NAMENODE | 18 |
| NFS_GATEWAY | 19 |
| NODEMANAGER | 20 |
| OOZIE_CLIENT | 21 |
| OOZIE_SERVER | 22 |
| PHOENIX_QUERY_SERVER | 23 |
| PIG | 24 |
| RESOURCEMANAGER | 25 |
| SECONDARY_NAMENODE | 26 |
| SPARK_CLIENT | 27 |
| SPARK_JOBHISTORYSERVER | 28 |
| SPARK_THRIFTSERVER | 29 |
| TEZ_CLIENT | 30 |
| WEBHCAT_SERVER | 31 |
| YARN_CLIENT | 32 |
| ZKFC | 33 |
| ZOOKEEPER_CLIENT | 34 |
| ZOOKEEPER_SERVER | 35 |
+-------------------------+----+
We then figured that if we get the pointer for 'id', where ever it came from, to get to '35', installation would succeed. So Elias ran the following command: CURL -X POST http://$AMBARI_HOST:8080/api/v1/clusters/$CLUSTER_NAME/services/SPARK2/components/LIVY2_SERVER until pointer was right: Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '33' for key 'PRIMARY'
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '34' for key 'PRIMARY'
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '35' for key 'PRIMARY' And at this point, we retried the full installation of Spark2 from Ambari UI, and it worked as expected! We have no clue at this point where or why the counter for 'id', at the insert into the servicecomponentdesiredstate table came from. If someone with deeper understanding of the ambari database-connections care to give us a hint, we can check it out.
... View more
