Created 05-01-2019 08:32 PM
Setup Info:
Cluster behind Firewall, no internet access
mysql = 5.7.23-1 (Engine=InnoDB)
Using local repo (Followed https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-installation/content/setting_up_a...) and now I have /etc/yum.repos.d/ambari.repo and /etc/yum.repos.d/hdp.rep pointing to local yum repo. This works fine as far as yum is concerned and I was able to successfully complete steps # 1 through 13 from https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-upgrade-major/content/upgrade_amb...
Now I am facing problems in step # 14
ambari-server upgrade
Error messages on screen:
=======================================
# ambari-server upgrade -v
Using python /usr/bin/python
Upgrading ambari-server
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Upgrade Ambari Server
INFO: Updating Ambari Server properties in ambari.properties ...
INFO: Updating Ambari Server properties in ambari-env.sh ...
INFO: Original file ambari-env.sh kept
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: No mpack replay logs found. Skipping replaying mpack commands
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Fixing database objects owner
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
Ambari Server configured for MySQL. Confirm you have made a backup of the Ambari Server database [y/n] (n)? y
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Upgrading database schema
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: AMBARI_SERVER_LIB is not set, using default /usr/lib/ambari-server
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: about to run command: /usr/jdk64/jdk1.8.0_112/bin/java -cp '/etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/share/java/mysql-connector-java.jar' org.apache.ambari.server.upgrade.SchemaUpgradeHelper > /var/log/ambari-server/ambari-server.out 2>&1
INFO:
process_pid=10015
Traceback (most recent call last):
File "/usr/sbin/ambari-server.py", line 1060, in <module>
mainBody()
File "/usr/sbin/ambari-server.py", line 1030, in mainBody
main(options, args, parser)
File "/usr/sbin/ambari-server.py", line 980, in main
action_obj.execute()
File "/usr/sbin/ambari-server.py", line 77, in execute
self.fn(*self.args, **self.kwargs)
File "/usr/lib/ambari-server/lib/ambari_server/serverUpgrade.py", line 262, in upgrade
retcode = run_schema_upgrade(args)
File "/usr/lib/ambari-server/lib/ambari_server/serverUpgrade.py", line 162, in run_schema_upgrade
upgrade_response = json.loads(stdout)
File "/usr/lib/ambari-server/lib/ambari_simplejson/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/usr/lib/ambari-server/lib/ambari_simplejson/decoder.py", line 335, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/ambari-server/lib/ambari_simplejson/decoder.py", line 353, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
=======================================
No logs in ambari-server.out, ambari-server-check-database.log, ambari-audit.log, ambari-alerts.log
/var/log/ambari-server/ambari-server-command.log
=======================================
INFO 2019-04-30 22:00:25,282 ambari-server.py:893 - loglevel=logging.INFO
=======================================
/var/log/ambari-server/ambari-eclipselink.log
=======================================
[EL Warning]: 2019-04-30 22:00:33.046--ServerSession(525558913)-- You have specified multiple ids for the entity class [org.apache.ambari.server.orm.entities.KerberosKeytabServiceMappingEntity] without specifying an @IdClass. By doing this you may lose the ability to find by identity, distributed cache support etc. Note: You may however use EntityManager find operations by passing a list of primary key fields. Else, you will have to use JPQL queries to read your entities. For other id options see @PrimaryKey.
[EL Warning]: 2019-04-30 22:00:33.089--ServerSession(525558913)-- The reference column name [resource_type_id] mapped on the element [field permissions] does not correspond to a valid id or basic field/column on the mapping reference. Will use referenced column name as provided.
[EL Info]: 2019-04-30 22:00:33.595--ServerSession(525558913)-- EclipseLink, version: Eclipse Persistence Services - 2.6.2.v20151217-774c696
[EL Info]: 2019-04-30 22:00:33.903--ServerSession(525558913)-- /file:/usr/lib/ambari-server/ambari-server-2.7.3.0.139.jar_ambari-server_nonJtaDataSource=1508038883_url=jdbc:mysql://xxxx-hostxxx-xxx.xxx.xxx.com:3306/ambaridb_user=ambari login successful
=======================================
/var/log/ambari-server/ambari-server.log
=======================================
:
:
2019-04-30 22:00:36,182 INFO [main] ServiceModule:204 - Display name service/parent: null/Pig
2019-04-30 22:00:36,210 INFO [Stack Version Loading Thread] RepoUrlInfoCallable:94 - Loading latest URL info from http://public-repo-1.hortonworks.com/HDP/hdp
_urlinfo.json for stacks HDP-2.3.GlusterFS,HDP-2.3.ECS,HDP-2.5,HDP-2.6,HDP-2.3,HDP-2.4,HDP-2.1.GlusterFS,HDP-2.1,HDP-3.0,HDP-3.1,HDP-2.2,HDP-2.0
2019-04-30 22:00:38,294 INFO [Stack Version Loading Thread] RepoUrlInfoCallable:107 - Could not load the URI from http://public-repo-1.hortonworks.com/HDP/hdp
_urlinfo.json, stack defaults will be used
2019-04-30 22:00:38,294 INFO [Stack Version Loading Thread] RepoUrlInfoCallable:110 - Loaded URI http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json for
stacks HDP-2.3.GlusterFS,HDP-2.3.ECS,HDP-2.5,HDP-2.6,HDP-2.3,HDP-2.4,HDP-2.1.GlusterFS,HDP-2.1,HDP-3.0,HDP-3.1,HDP-2.2,HDP-2.0 in 2086ms
2019-04-30 22:00:38,294 INFO [main] StackContext:180 - Loaded urlinfo in 2088ms
2019-04-30 22:00:38,295 ERROR [main] StackContext:189 - Could not load repo results
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)
at sun.net.www.http.HttpClient.New(HttpClient.java:326)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:218)
at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:142)
at org.apache.ambari.server.controller.internal.URLStreamProvider.readFrom(URLStreamProvider.java:116)
at org.apache.ambari.server.controller.internal.URLStreamProvider.readFrom(URLStreamProvider.java:121)
at org.apache.ambari.server.state.stack.RepoUrlInfoCallable.call(RepoUrlInfoCallable.java:97)
at org.apache.ambari.server.state.stack.RepoUrlInfoCallable.call(RepoUrlInfoCallable.java:50)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2019-04-30 22:00:38,297 INFO [main] StackContext:222 - Loaded all VDF in 0ms
2019-04-30 22:00:38,337 INFO [main] StackManager:208 - Adding stack HDP-3.0 to the database
2019-04-30 22:00:38,358 INFO [main] StackManager:208 - Adding stack HDP-3.1 to the database
:
:
2019-04-30 22:00:38,664 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_hostcomponentdesiredstate_component_name ON hostcomponentdesiredstate, errorCode = 1553, message = Cannot drop index 'FK_hostcomponentdesiredstate_component_name': needed in a foreign key constraint
2019-04-30 22:00:38,665 WARN [main] DBAccessorImpl:382 - FK fk_hostcomponentstate_component_name not found for table hostcomponentstate
2019-04-30 22:00:38,665 WARN [main] DBAccessorImpl:997 - Foreign key fk_hostcomponentstate_component_name from hostcomponentstate table does not exist and will not be dropped
2019-04-30 22:00:38,666 INFO [main] DBAccessorImpl:869 - Executing query: DROP INDEX fk_hostcomponentstate_component_name ON hostcomponentstate
2019-04-30 22:00:38,667 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_hostcomponentstate_component_name ON hostcomponentstate, errorCode = 1553, message = Cannot drop index 'FK_hostcomponentstate_component_name': needed in a foreign key constraint
2019-04-30 22:00:38,668 WARN [main] DBAccessorImpl:382 - FK fk_servicecomponentdesiredstate_service_name not found for table servicecomponentdesiredstate
2019-04-30 22:00:38,668 WARN [main] DBAccessorImpl:997 - Foreign key fk_servicecomponentdesiredstate_service_name from servicecomponentdesiredstate table does not exist and will not be dropped
2019-04-30 22:00:38,670 INFO [main] DBAccessorImpl:869 - Executing query: DROP INDEX fk_servicecomponentdesiredstate_service_name ON servicecomponentdesiredstate
2019-04-30 22:00:38,671 WARN [main] DBAccessorImpl:878 - Error executing query: DROP INDEX fk_servicecomponentdesiredstate_service_name ON servicecomponentdesiredstate, errorCode = 1553, message = Cannot drop index 'FK_servicecomponentdesiredstate_service_name': needed in a foreign key constraint
2019-04-30 22:00:38,672 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE stage ADD status VARCHAR(255) NOT NULL DEFAULT 'PENDING'
2019-04-30 22:00:38,727 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE stage ADD display_status VARCHAR(255) NOT NULL DEFAULT 'PENDING'
2019-04-30 22:00:38,779 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE request ADD display_status VARCHAR(255) NOT NULL DEFAULT 'PENDING'
2019-04-30 22:00:38,848 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE request ADD user_name VARCHAR(255)
2019-04-30 22:00:38,914 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE host_role_command ADD ops_display_name VARCHAR(255)
2019-04-30 22:00:39,885 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE hostcomponentdesiredstate DROP COLUMN security_state
2019-04-30 22:00:39,917 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE hostcomponentstate DROP COLUMN security_state
2019-04-30 22:00:39,947 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE servicedesiredstate DROP COLUMN security_state
2019-04-30 22:00:39,977 WARN [main] DBAccessorImpl:581 - Primary constraint PK_ambari_configuration not altered to table ambari_configuration as column [category_name, property_name] not present or constraint already exists
2019-04-30 22:00:39,979 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE hostcomponentstate ADD last_live_state VARCHAR(255) DEFAULT 'UNKNOWN'
2019-04-30 22:00:40,008 INFO [main] UpgradeCatalog270:615 - users.create_time's type = 93
2019-04-30 22:00:40,008 INFO [main] UpgradeCatalog270:588 - Converting user creation times...
2019-04-30 22:00:40,011 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE users ADD create_time_numeric BIGINT
2019-04-30 22:00:40,049 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE users DROP COLUMN create_time
2019-04-30 22:00:40,077 INFO [main] DBAccessorImpl:869 - Executing query: ALTER TABLE users CHANGE create_time_numeric create_time BIGINT
2019-04-30 22:00:40,078 INFO [main] UpgradeCatalog270:607 - Converted user creation times
2019-04-30 22:00:40,079 INFO [main] DBAccessorImpl:869 - Executing query: DROP TABLE user_authentication_tmp
2019-04-30 22:00:40,084 INFO [main] DBAccessorImpl:869 - Executing query: CREATE TABLE user_authentication_tmp (user_authentication_id BIGINT NOT NULL, user_id INTEGER NOT NULL, authentication_type VARCHAR(50) NOT NULL, authentication_key VARCHAR(2048), create_time BIGINT, update_time BIGINT) ENGINE=INNODB
2019-04-30 22:00:40,088 WARN [main] DBAccessorImpl:581 - Primary constraint PK_user_authentication not altered to table user_authentication as column [user_authentication_id] not present or constraint already exists
2019-04-30 22:00:40,121 INFO [main] DBAccessorImpl:522 - Foreign Key constraint FK_user_authentication_users already exists, skipping
2019-04-30 22:00:40,123 WARN [main] DBAccessorImpl:842 - Error executing query: insert into user_authentication(user_authentication_id, user_id, authentication_type, authentication_key, create_time, update_time) select user_authentication_id, user_id, authentication_type, authentication_key, create_time, update_time from user_authentication_tmp, errorCode = 1062, message = Duplicate entry '1' for key 'PRIMARY'
2019-04-30 22:00:40,123 ERROR [main] SchemaUpgradeHelper:209 - Upgrade failed.
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '1' for key 'PRIMARY'
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1041)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4187)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4119)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2570)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2731)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2814)
at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1813)
at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1727)
at org.apache.ambari.server.orm.DBAccessorImpl.executeUpdate(DBAccessorImpl.java:840)
at org.apache.ambari.server.orm.DBAccessorImpl.executeUpdate(DBAccessorImpl.java:833)
at org.apache.ambari.server.upgrade.UpgradeCatalog270.createUserAuthenticationTable(UpgradeCatalog270.java:689)
at org.apache.ambari.server.upgrade.UpgradeCatalog270.upgradeUserTables(UpgradeCatalog270.java:346)
at org.apache.ambari.server.upgrade.UpgradeCatalog270.executeDDLUpdates(UpgradeCatalog270.java:321)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:970)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:207)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:450)
2019-04-30 22:00:40,123 ERROR [main] SchemaUpgradeHelper:475 - Exception occurred during upgrade, failed
org.apache.ambari.server.AmbariException: Duplicate entry '1' for key 'PRIMARY'
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:210)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:450)
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '1' for key 'PRIMARY'
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1041)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4187)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4119)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2570)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2731)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2814)
at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1813)
at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1727)
at org.apache.ambari.server.orm.DBAccessorImpl.executeUpdate(DBAccessorImpl.java:840)
at org.apache.ambari.server.orm.DBAccessorImpl.executeUpdate(DBAccessorImpl.java:833)
at org.apache.ambari.server.upgrade.UpgradeCatalog270.createUserAuthenticationTable(UpgradeCatalog270.java:689)
at org.apache.ambari.server.upgrade.UpgradeCatalog270.upgradeUserTables(UpgradeCatalog270.java:346)
at org.apache.ambari.server.upgrade.UpgradeCatalog270.executeDDLUpdates(UpgradeCatalog270.java:321)
at org.apache.ambari.server.upgrade.AbstractUpgradeCatalog.upgradeSchema(AbstractUpgradeCatalog.java:970)
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executeUpgrade(SchemaUpgradeHelper.java:207)
... 1 more
========================================
Is this issue because I am using local repo without internet? or is it something to do with DB schema. My Cluster is working fine, no alerts, my DB seems ok as I have tried this process 3 times now and everytime I hit the same issue but I am able to successfully rollback and restore it back to Ambari 2.6.1.5 successfully.
What if instead of trying ambari-server upgrade I reinstall 2.7.3? Will I loose my hdfs data?
I will really appreciate if you can point me in right direction.
Thanks