Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

App Timeline server duplicated (sort of) after failed move

avatar
Explorer

I'm running ambari 2.1.0. I tried to move the App Timeline Server (ATS) and the process failed (for various reasons).

I was able to bring ambari back up, but now it seems to think it has two ATS masters. In order to get things operational, I've put one of them (the one I was trying to move the service to) in maintenance mode and started the original one. Now I have a permanent alert on the first host that it failed to connect to the ATS on the second host.

From the dashboard, YARN appears to be up/operational but I'm not sure if it is or not.

Any suggestions on how I might be able to untangle this?

1 ACCEPTED SOLUTION

avatar
Rising Star

Thanks for sharing the output. Yes, that's exactly what I meant (REST API call to get the ATS instances registered with Ambari).

To delete the bad ATS instance from Ambari, you can issue the following API call:

curl -u admin:admin -k -H "X-Requested-By: ambari" -X DELETE https://localhost:8443/api/v1/clusters/ROGERGPFS/hosts/<hostname-with-bad-ATS>/host_components/APP_T...

View solution in original post

17 REPLIES 17

avatar
Rising Star

Hi Wayne.

What's the output of /api/v1/clusters/<cluster_name>/host_components?HostRoles/component_name=APP_TIMELINE_SERVER

Do you see two ATS instances in the output?

avatar
Explorer

I cannot seem to find that command.

avatar
Explorer

However, I might have gotten what you want in a different manner:

[root@cg-hm08 ~]# curl -i -uadmin:<> -k -H "X-Requested-By: ambari" -d '{"HostRoles": { "state": "STARTED"}}' -X GET 'https://localhost:8443/api/v1/clusters/ROGERGPFS/host_components?HostRoles/component_name=APP_TIMELINE_SERVER'

HTTP/1.1 200 OK

User: admin

Set-Cookie: AMBARISESSIONID=1jmupkh1wyuoo8n3d84vc8zsc;Path=/;Secure;HttpOnly

Expires: Thu, 01 Jan 1970 00:00:00 GMT

Content-Type: text/plain

Vary: Accept-Encoding, User-Agent

Content-Length: 1028

Server: Jetty(8.1.17.v20150415)

{

"href" : "https://localhost:8443/api/v1/clusters/ROGERGPFS/host_components?HostRoles/component_name=APP_TIMELINE_SERVER",

"items" : [

{

"href" : "https://localhost:8443/api/v1/clusters/ROGERGPFS/hosts/cg-hm09.ncsa.illinois.edu/host_components/APP_TIMELINE_SERVER",

"HostRoles" : {

"cluster_name" : "ROGERGPFS",

"component_name" : "APP_TIMELINE_SERVER",

"host_name" : "cg-hm09.ncsa.illinois.edu"

},

"host" : {

"href" : "https://localhost:8443/api/v1/clusters/ROGERGPFS/hosts/cg-hm09.ncsa.illinois.edu"

}

},

{

"href" : "https://localhost:8443/api/v1/clusters/ROGERGPFS/hosts/cg-hm11.ncsa.illinois.edu/host_components/APP_TIMELINE_SERVER",

"HostRoles" : {

"cluster_name" : "ROGERGPFS",

"component_name" : "APP_TIMELINE_SERVER",

"host_name" : "cg-hm11.ncsa.illinois.edu"

},

"host" : {

"href" : "https://localhost:8443/api/v1/clusters/ROGERGPFS/hosts/cg-hm11.ncsa.illinois.edu"

}

}

]

}

avatar
Rising Star

Thanks for sharing the output. Yes, that's exactly what I meant (REST API call to get the ATS instances registered with Ambari).

To delete the bad ATS instance from Ambari, you can issue the following API call:

curl -u admin:admin -k -H "X-Requested-By: ambari" -X DELETE https://localhost:8443/api/v1/clusters/ROGERGPFS/hosts/<hostname-with-bad-ATS>/host_components/APP_T...

avatar
Explorer

Yeah. When I screw something up, I don't seem to do it by half measures.

avatar
Explorer

#curl -u admin:<> -k -H "X-Requested-By: ambari" -X DELETE https://localhost:8443/api/v1/clusters/ROGERGPFS/hosts/cg-hm09.ncsa.illinois.edu/host_components/APP...

{

"status" : 500,

"message" : "org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Host Component cannot be removed, clusterName=ROGERGPFS, serviceName=YARN, componentName=APP_TIMELINE_SERVER, hostname=cg-hm09.ncsa.illinois.edu, request={ clusterName=ROGERGPFS, serviceName=YARN, componentName=APP_TIMELINE_SERVER, hostname=cg-hm09.ncsa.illinois.edu, desiredState=null, state=null, desiredStackId=null, staleConfig=null, adminState=null}"

}

avatar
Rising Star

🙂 Can you get the corresponding stack trace from the server log? It's at /var/log/ambari-server/ambari-server.log

avatar
Explorer

sorry if you saw the rest of the activity on this response. First, I posted the resultant output, but I did it as an answer. Given that it wasn't, I deleted it noticing only too late that I could have converted it to a comment. The next time I attempted to add the output, I managed to grab the wrong copy buffer (and wound up duplicating the original informational output from the GET command. So, I deleted that. Sigh.

avatar
Rising Star

No worries. If you can provide the stack trace from the ambari-server.log file I should be able to help you further.