Ambari Upgrade 2.6.1.5 => 2.7.3

[#1]

Setup Info:

Cluster behind Firewall, no internet access

mysql = 5.7.23-1 (Engine=InnoDB)

Using local repo (Followed https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-installation/content/setting_up_a...) and now I have /etc/yum.repos.d/ambari.repo and /etc/yum.repos.d/hdp.rep pointing to local yum repo. This works fine as far as yum is concerned and I was able to successfully complete steps # 1 through 13 from https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-upgrade-major/content/upgrade_amb...

Now I am facing problems in step # 14

ambari-server upgrade

Error messages on screen:

=======================================

# ambari-server upgrade -v

Using python /usr/bin/python

Upgrading ambari-server

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Upgrade Ambari Server

INFO: Updating Ambari Server properties in ambari.properties ...

INFO: Updating Ambari Server properties in ambari-env.sh ...

INFO: Original file ambari-env.sh kept

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: No mpack replay logs found. Skipping replaying mpack commands

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Fixing database objects owner

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

Ambari Server configured for MySQL. Confirm you have made a backup of the Ambari Server database [y/n] (n)? y

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Upgrading database schema

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: AMBARI_SERVER_LIB is not set, using default /usr/lib/ambari-server

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: Loading properties from /etc/ambari-server/conf/ambari.properties

INFO: about to run command: /usr/jdk64/jdk1.8.0_112/bin/java -cp '/etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/mysql-connector-java.jar' org.apache.ambari.server.upgrade.SchemaUpgradeHelper > /var/log/ambari-server/ambari-server.out 2>&1

INFO:

process_pid=10015

Traceback (most recent call last):

File "/usr/sbin/ambari-server.py", line 1060, in <module>

mainBody()

File "/usr/sbin/ambari-server.py", line 1030, in mainBody

main(options, args, parser)

File "/usr/sbin/ambari-server.py", line 980, in main

action_obj.execute()

File "/usr/sbin/ambari-server.py", line 77, in execute

self.fn(*self.args, **self.kwargs)

File "/usr/lib/ambari-server/lib/ambari_server/serverUpgrade.py", line 262, in upgrade

retcode = run_schema_upgrade(args)

File "/usr/lib/ambari-server/lib/ambari_server/serverUpgrade.py", line 162, in run_schema_upgrade

upgrade_response = json.loads(stdout)

File "/usr/lib/ambari-server/lib/ambari_simplejson/__init__.py", line 307, in loads

return _default_decoder.decode(s)

File "/usr/lib/ambari-server/lib/ambari_simplejson/decoder.py", line 335, in decode

obj, end = self.raw_decode(s, idx=_w(s, 0).end())

File "/usr/lib/ambari-server/lib/ambari_simplejson/decoder.py", line 353, in raw_decode

raise ValueError("No JSON object could be decoded")

ValueError: No JSON object could be decoded

=======================================

No logs in ambari-server.out, ambari-server-check-database.log, ambari-audit.log, ambari-alerts.log

/var/log/ambari-server/ambari-server-command.log

=======================================

INFO 2019-04-30 22:00:25,282 ambari-server.py:893 - loglevel=logging.INFO

=======================================

/var/log/ambari-server/ambari-eclipselink.log

=======================================

[EL Warning]: 2019-04-30 22:00:33.046--ServerSession(525558913)-- You have specified multiple ids for the entity class [org.apache.ambari.server.orm.entities.KerberosKeytabServiceMappingEntity] without specifying an @IdClass. By doing this you may lose the ability to find by identity, distributed cache support etc. Note: You may however use EntityManager find operations by passing a list of primary key fields. Else, you will have to use JPQL queries to read your entities. For other id options see @PrimaryKey.

[EL Warning]: 2019-04-30 22:00:33.089--ServerSession(525558913)-- The reference column name [resource_type_id] mapped on the element [field permissions] does not correspond to a valid id or basic field/column on the mapping reference. Will use referenced column name as provided.

[EL Info]: 2019-04-30 22:00:33.595--ServerSession(525558913)-- EclipseLink, version: Eclipse Persistence Services - 2.6.2.v20151217-774c696

[EL Info]: 2019-04-30 22:00:33.903--ServerSession(525558913)-- /file:/usr/lib/ambari-server/ambari-server-2.7.3.0.139.jar_ambari-server_nonJtaDataSource=1508038883_url=jdbc:mysql://xxxx-hostxxx-xxx.xxx.xxx.com:3306/ambaridb_user=ambari login successful

=======================================

/var/log/ambari-server/ambari-server.log

=======================================

:

:

2019-04-30 22:00:36,182 INFO [main] ServiceModule:204 - Display name service/parent: null/Pig

2019-04-30 22:00:36,210 INFO [Stack Version Loading Thread] RepoUrlInfoCallable:94 - Loading latest URL info from http://public-repo-1.hortonworks.com/HDP/hdp

_urlinfo.json for stacks HDP-2.3.GlusterFS,HDP-2.3.ECS,HDP-2.5,HDP-2.6,HDP-2.3,HDP-2.4,HDP-2.1.GlusterFS,HDP-2.1,HDP-3.0,HDP-3.1,HDP-2.2,HDP-2.0

2019-04-30 22:00:38,294 INFO [Stack Version Loading Thread] RepoUrlInfoCallable:107 - Could not load the URI from http://public-repo-1.hortonworks.com/HDP/hdp

_urlinfo.json, stack defaults will be used

2019-04-30 22:00:38,294 INFO [Stack Version Loading Thread] RepoUrlInfoCallable:110 - Loaded URI http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json for

stacks HDP-2.3.GlusterFS,HDP-2.3.ECS,HDP-2.5,HDP-2.6,HDP-2.3,HDP-2.4,HDP-2.1.GlusterFS,HDP-2.1,HDP-3.0,HDP-3.1,HDP-2.2,HDP-2.0 in 2086ms

2019-04-30 22:00:38,294 INFO [main] StackContext:180 - Loaded urlinfo in 2088ms

2019-04-30 22:00:38,295 ERROR [main] StackContext:189 - Could not load repo results

java.net.SocketTimeoutException: connect timed out

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:589)

at sun.net.NetworkClient.doConnect(NetworkClient.java:175)

at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)

at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)

at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)

at sun.net.www.http.HttpClient.New(HttpClient.java:308)

at sun.net.www.http.HttpClient.New(HttpClient.java:326)

at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202)

at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)

at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)

at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966)

at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)

at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)

at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)

at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:218)

at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:142)

at org.apache.ambari.server.controller.internal.URLStreamProvider.readFrom(URLStreamProvider.java:116)

at org.apache.ambari.server.controller.internal.URLStreamProvider.readFrom(URLStreamProvider.java:121)

at org.apache.ambari.server.state.stack.RepoUrlInfoCallable.call(RepoUrlInfoCallable.java:97)

at org.apache.ambari.server.state.stack.RepoUrlInfoCallable.call(RepoUrlInfoCallable.java:50)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

2019-04-30 22:00:38,297 INFO [main] StackContext:222 - Loaded all VDF in 0ms

2019-04-30 22:00:38,337 INFO [main] StackManager:208 - Adding stack HDP-3.0 to the database

2019-04-30 22:00:38,358 INFO [main] StackManager:208 - Adding stack HDP-3.1 to the database

:

:

2019-04-30 22:00:38,664 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_hostcomponentdesiredstate_component_name ON hostcomponentdesiredstate, errorCode = 1553, message = Cannot drop index 'FK_hostcomponentdesiredstate_component_name': needed in a foreign key constraint

2019-04-30 22:00:38,665 WARN [main] DBAccessorImpl:382 - FK fk_hostcomponentstate_component_name not found for table hostcomponentstate

2019-04-30 22:00:38,665 WARN [main] DBAccessorImpl:997 - Foreign key fk_hostcomponentstate_component_name from hostcomponentstate table does not exist and will not be dropped

2019-04-30 22:00:38,666 INFO [main] DBAccessorImpl:869 - Executing query: DROP INDEX fk_hostcomponentstate_component_name ON hostcomponentstate

2019-04-30 22:00:38,667 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_hostcomponentstate_component_name ON hostcomponentstate, errorCode = 1553, message = Cannot drop index 'FK_hostcomponentstate_component_name': needed in a foreign key constraint

2019-04-30 22:00:38,668 WARN [main] DBAccessorImpl:382 - FK fk_servicecomponentdesiredstate_service_name not found for table servicecomponentdesiredstate

2019-04-30 22:00:38,668 WARN [main] DBAccessorImpl:997 - Foreign key fk_servicecomponentdesiredstate_service_name from servicecomponentdesiredstate table does not exist and will not be dropped

2019-04-30 22:00:38,670 INFO [main] DBAccessorImpl:869 - Executing query: DROP INDEX fk_servicecomponentdesiredstate_service_name ON servicecomponentdesiredstate

2019-04-30 22:00:38,671 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_servicecomponentdesiredstate_service_name ON servicecomponentdesiredstate, errorCode = 1553, message = Cannot drop index 'FK_servicecomponentdesiredstate_service_name': needed in a foreign key constraint

2019-04-30 22:00:38,672 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE stage ADD status VARCHAR(255) NOT NULL DEFAULT 'PENDING'

2019-04-30 22:00:38,727 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE stage ADD display_status VARCHAR(255) NOT NULL DEFAULT 'PENDING'

2019-04-30 22:00:38,779 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE request ADD display_status VARCHAR(255) NOT NULL DEFAULT 'PENDING'

2019-04-30 22:00:38,848 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE request ADD user_name VARCHAR(255)

2019-04-30 22:00:38,914 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE host_role_command ADD ops_display_name VARCHAR(255)

2019-04-30 22:00:39,885 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE hostcomponentdesiredstate DROP COLUMN security_state

2019-04-30 22:00:39,917 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE hostcomponentstate DROP COLUMN security_state

2019-04-30 22:00:39,947 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE servicedesiredstate DROP COLUMN security_state

2019-04-30 22:00:39,977 WARN [main] DBAccessorImpl:581 - Primary constraint PK_ambari_configuration not altered to table ambari_configuration as column [category_name, property_name] not present or constraint already exists

2019-04-30 22:00:39,979 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE hostcomponentstate ADD last_live_state VARCHAR(255) DEFAULT 'UNKNOWN'

2019-04-30 22:00:40,008 INFO [main] UpgradeCatalog270:615 - users.create_time's type = 93

2019-04-30 22:00:40,008 INFO [main] UpgradeCatalog270:588 - Converting user creation times...

2019-04-30 22:00:40,011 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE users ADD create_time_numeric BIGINT

2019-04-30 22:00:40,049 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE users DROP COLUMN create_time

2019-04-30 22:00:40,077 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE users CHANGE create_time_numeric create_time BIGINT

2019-04-30 22:00:40,078 INFO [main] UpgradeCatalog270:607 - Converted user creation times

2019-04-30 22:00:40,079 INFO [main] DBAccessorImpl:869 - Executing query: DROP TABLE user_authentication_tmp

2019-04-30 22:00:40,084 INFO [main] DBAccessorImpl:869 - Executing query: CREATE TABLE user_authentication_tmp (user_authentication_id BIGINT NOT NULL, user_id INTEGER NOT NULL, authentication_type VARCHAR(50) NOT NULL, authentication_key VARCHAR(2048), create_time BIGINT, update_time BIGINT) ENGINE=INNODB

2019-04-30 22:00:40,088 WARN [main] DBAccessorImpl:581 - Primary constraint PK_user_authentication not altered to table user_authentication as column [user_authentication_id] not present or constraint already exists

2019-04-30 22:00:40,121 INFO [main] DBAccessorImpl:522 - Foreign Key constraint FK_user_authentication_users already exists, skipping

2019-04-30 22:00:40,123 WARN [main] DBAccessorImpl:842 - Error executing query: insert into user_authentication(user_authentication_id, user_id, authentication_type, authentication_key, create_time, update_time) select user_authentication_id, user_id, authentication_type, authentication_key, create_time, update_time from user_authentication_tmp, errorCode = 1062, message = Duplicate entry '1' for key 'PRIMARY'

2019-04-30 22:00:40,123 ERROR [main] SchemaUpgradeHelper:209 - Upgrade failed.

com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '1' for key 'PRIMARY'

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)

at com.mysql.jdbc.Util.getInstance(Util.java:386)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1041)

at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4187)

at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4119)

at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2570)

at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2731)

at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2814)

at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1813)

at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1727)

at org.apache.ambari.server.orm.DBAccessorImpl.executeUpdate(DBAccessorImpl.java:840)

at org.apache.ambari.server.orm.DBAccessorImpl.executeUpdate(DBAccessorImpl.java:833)

at org.apache.ambari.server.upgrade.UpgradeCatalog270.createUserAuthenticationTable(UpgradeCatalog270.java:689)

at org.apache.ambari.server.upgrade.UpgradeCatalog270.upgradeUserTables(UpgradeCatalog270.java:346)

at org.apache.ambari.server.upgrade.UpgradeCatalog270.executeDDLUpdates(UpgradeCatalog270.java:321)

at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:970)

at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:207)

at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:450)

2019-04-30 22:00:40,123 ERROR [main] SchemaUpgradeHelper:475 - Exception occurred during upgrade, failed

org.apache.ambari.server.AmbariException: Duplicate entry '1' for key 'PRIMARY'

at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:210)

at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:450)

Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '1' for key 'PRIMARY'

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)

at com.mysql.jdbc.Util.getInstance(Util.java:386)

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1041)

at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4187)

at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4119)

at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2570)

at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2731)

at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2814)

at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1813)

at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1727)

at org.apache.ambari.server.orm.DBAccessorImpl.executeUpdate(DBAccessorImpl.java:840)

at org.apache.ambari.server.orm.DBAccessorImpl.executeUpdate(DBAccessorImpl.java:833)

at org.apache.ambari.server.upgrade.UpgradeCatalog270.createUserAuthenticationTable(UpgradeCatalog270.java:689)

at org.apache.ambari.server.upgrade.UpgradeCatalog270.upgradeUserTables(UpgradeCatalog270.java:346)

at org.apache.ambari.server.upgrade.UpgradeCatalog270.executeDDLUpdates(UpgradeCatalog270.java:321)

at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:970)

at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:207)

... 1 more

========================================

Is this issue because I am using local repo without internet? or is it something to do with DB schema. My Cluster is working fine, no alerts, my DB seems ok as I have tried this process 3 times now and everytime I hit the same issue but I am able to successfully rollback and restore it back to Ambari 2.6.1.5 successfully.

What if instead of trying ambari-server upgrade I reinstall 2.7.3? Will I loose my hdfs data?

I will really appreciate if you can point me in right direction.

Thanks