Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

After 2.3.4 rolling upgrade Restart Ambari Server is getting me a 500 Status (server Error) on GET

After 2.3.4 rolling upgrade Restart Ambari Server is getting me a 500 Status (server Error) on GET

New Contributor

We have performed a rolling upgrade from 2.3 to 2.3.4 and everything went fine and looked alright in ambari. To set the version we had to perform a final restart of ambari after which none of the services are being accessible in ambari. I can see them all green after logging in but I keep continuously getting a popup of 500 Server Error Statuses.

500 status codereceived on GET method for API: /api/v1/clusters/****/host_components?HostRoles/component_name=NAMENODE&metrics/dfs/FSNamesystem/HAState=active&fields=metrics/jvm/gcCount[1461270184,146

500 status codereceived on GET method for API: /api/v1/clusters/****/hosts?fields=Hosts/rack_info,Hosts/host_name,Hosts/maintenance_state,Hosts/public_host_name,Hosts/cpu_count,Hosts/ph_cpu_count,alerts_summary,Hosts/host_status,Hosts/last_heartbeat_time,Hosts/ip,host_components/HostRoles/state,host_components/HostRoles/maintenance_state,host_components/HostRoles/stale_configs,host_components/HostRoles/service_name,host_components/HostRoles/desired_admin_state,metrics/disk,metrics/load/load_one,Hosts/total_mem,Hosts/os_arch,Hosts/os_type,metrics/cpu/cpu_system,metrics/cpu/cpu_user,metrics/memory/mem_total,metrics/memory/mem_free,stack_versions/HostStackVersions,stack_versions/repository_versions/RepositoryVersions/repository_version,stack_versions/repository_versions/RepositoryVersions/id,stack_versions/repository_versions/RepositoryVersions/display_name&minimal_response=true&page_size=25&from=0

Ambari Server logs are showing nothing, and agent logs are complaining that there is a 500 server error.

Even the following on command line is returning

[**** ~]$ curl -u **** -H "X-Requested-By: ambari" -X GET http://*:8080/api/v1/clusters/***/hosts/abc.abc.com { "status": 500, "message": "Server Error" }

Any help would be greatly appreciated

Thanks

Prasad

3 REPLIES 3
Highlighted

Re: After 2.3.4 rolling upgrade Restart Ambari Server is getting me a 500 Status (server Error) on GET

New Contributor

Anything in ambari-server.out file? Is there any API that works (/api/v1/clusters/***, /api/v1/stacks, etc.) ? Also, when you restart ambari-server, is there anything new being written to ambari-server.log?

Re: After 2.3.4 rolling upgrade Restart Ambari Server is getting me a 500 Status (server Error) on GET

Is your Ambari database running? If you use Postgres try "service postgresql status". You can also try to restart both ambari-server and DB: stop ambari-server, run "service postgresql restart", and then start ambari-server. By the way, have you finalized your upgrade before restarting Ambari? There is actually no need to restart Ambari after RU of HDP.

Re: After 2.3.4 rolling upgrade Restart Ambari Server is getting me a 500 Status (server Error) on GET

New Contributor

Yes the database is running, and ambari-server.out file was just showing as Server Error. ambari_server.log just was spewing errors that it cant connect to some nodes.

Tried all kinds of restarts and infact tried restoring database. Had to completely reset ambari database and rebuilt it. Did not like the way it had to be done.

Kept a back up of current corrupted database so that i can test it on an R&D cluster.