Support Questions
Find answers, ask questions, and share your expertise

App Timeline server duplicated (sort of) after failed move

Solved Go to solution
Highlighted

App Timeline server duplicated (sort of) after failed move

Explorer

I'm running ambari 2.1.0. I tried to move the App Timeline Server (ATS) and the process failed (for various reasons).

I was able to bring ambari back up, but now it seems to think it has two ATS masters. In order to get things operational, I've put one of them (the one I was trying to move the service to) in maintenance mode and started the original one. Now I have a permanent alert on the first host that it failed to connect to the ATS on the second host.

From the dashboard, YARN appears to be up/operational but I'm not sure if it is or not.

Any suggestions on how I might be able to untangle this?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: App Timeline server duplicated (sort of) after failed move

Contributor

Thanks for sharing the output. Yes, that's exactly what I meant (REST API call to get the ATS instances registered with Ambari).

To delete the bad ATS instance from Ambari, you can issue the following API call:

curl -u admin:admin -k -H "X-Requested-By: ambari" -X DELETE https://localhost:8443/api/v1/clusters/ROGERGPFS/hosts/<hostname-with-bad-ATS>/host_components/APP_T...

View solution in original post

17 REPLIES 17

Re: App Timeline server duplicated (sort of) after failed move

Contributor

Hi Wayne.

What's the output of /api/v1/clusters/<cluster_name>/host_components?HostRoles/component_name=APP_TIMELINE_SERVER

Do you see two ATS instances in the output?

Highlighted

Re: App Timeline server duplicated (sort of) after failed move

Explorer

I cannot seem to find that command.

Highlighted

Re: App Timeline server duplicated (sort of) after failed move

Explorer

However, I might have gotten what you want in a different manner:

[root@cg-hm08 ~]# curl -i -uadmin:<> -k -H "X-Requested-By: ambari" -d '{"HostRoles": { "state": "STARTED"}}' -X GET 'https://localhost:8443/api/v1/clusters/ROGERGPFS/host_components?HostRoles/component_name=APP_TIMELINE_SERVER'

HTTP/1.1 200 OK

User: admin

Set-Cookie: AMBARISESSIONID=1jmupkh1wyuoo8n3d84vc8zsc;Path=/;Secure;HttpOnly

Expires: Thu, 01 Jan 1970 00:00:00 GMT

Content-Type: text/plain

Vary: Accept-Encoding, User-Agent

Content-Length: 1028

Server: Jetty(8.1.17.v20150415)

{

"href" : "https://localhost:8443/api/v1/clusters/ROGERGPFS/host_components?HostRoles/component_name=APP_TIMELINE_SERVER",

"items" : [

{

"href" : "https://localhost:8443/api/v1/clusters/ROGERGPFS/hosts/cg-hm09.ncsa.illinois.edu/host_components/APP_TIMELINE_SERVER",

"HostRoles" : {

"cluster_name" : "ROGERGPFS",

"component_name" : "APP_TIMELINE_SERVER",

"host_name" : "cg-hm09.ncsa.illinois.edu"

},

"host" : {

"href" : "https://localhost:8443/api/v1/clusters/ROGERGPFS/hosts/cg-hm09.ncsa.illinois.edu"

}

},

{

"href" : "https://localhost:8443/api/v1/clusters/ROGERGPFS/hosts/cg-hm11.ncsa.illinois.edu/host_components/APP_TIMELINE_SERVER",

"HostRoles" : {

"cluster_name" : "ROGERGPFS",

"component_name" : "APP_TIMELINE_SERVER",

"host_name" : "cg-hm11.ncsa.illinois.edu"

},

"host" : {

"href" : "https://localhost:8443/api/v1/clusters/ROGERGPFS/hosts/cg-hm11.ncsa.illinois.edu"

}

}

]

}

Highlighted

Re: App Timeline server duplicated (sort of) after failed move

Contributor

Thanks for sharing the output. Yes, that's exactly what I meant (REST API call to get the ATS instances registered with Ambari).

To delete the bad ATS instance from Ambari, you can issue the following API call:

curl -u admin:admin -k -H "X-Requested-By: ambari" -X DELETE https://localhost:8443/api/v1/clusters/ROGERGPFS/hosts/<hostname-with-bad-ATS>/host_components/APP_T...

View solution in original post

Highlighted

Re: App Timeline server duplicated (sort of) after failed move

Explorer

Yeah. When I screw something up, I don't seem to do it by half measures.

Highlighted

Re: App Timeline server duplicated (sort of) after failed move

Explorer

#curl -u admin:<> -k -H "X-Requested-By: ambari" -X DELETE https://localhost:8443/api/v1/clusters/ROGERGPFS/hosts/cg-hm09.ncsa.illinois.edu/host_components/APP...

{

"status" : 500,

"message" : "org.apache.ambari.server.controller.spi.SystemException: An internal system exception occurred: Host Component cannot be removed, clusterName=ROGERGPFS, serviceName=YARN, componentName=APP_TIMELINE_SERVER, hostname=cg-hm09.ncsa.illinois.edu, request={ clusterName=ROGERGPFS, serviceName=YARN, componentName=APP_TIMELINE_SERVER, hostname=cg-hm09.ncsa.illinois.edu, desiredState=null, state=null, desiredStackId=null, staleConfig=null, adminState=null}"

}

Highlighted

Re: App Timeline server duplicated (sort of) after failed move

Contributor

:) Can you get the corresponding stack trace from the server log? It's at /var/log/ambari-server/ambari-server.log

Highlighted

Re: App Timeline server duplicated (sort of) after failed move

Explorer

sorry if you saw the rest of the activity on this response. First, I posted the resultant output, but I did it as an answer. Given that it wasn't, I deleted it noticing only too late that I could have converted it to a comment. The next time I attempted to add the output, I managed to grab the wrong copy buffer (and wound up duplicating the original informational output from the GET command. So, I deleted that. Sigh.

Highlighted

Re: App Timeline server duplicated (sort of) after failed move

Contributor

No worries. If you can provide the stack trace from the ambari-server.log file I should be able to help you further.