question Re: How to export all the output/error logs for a Bulk operation e.g. Start all Service in Support Questions

How to export all the output/error logs for a Bulk operation e.g. Start all Service

pankaj_singh — Tue, 01 Dec 2015 15:09:57 GMT

Suppose executed a bulk operation through Amabri UI e.g. "Start All Services" .

The requirement is -

I want to export the complete log of this operation for further analysis

One way is to find all hosts impacted by this operation , get the sub tasks executed on each hosts and then loop through them and get the logs and merge them

Is there a more simple way of getting all logs for a bulk operation done through ambari

Re: How to export all the output/error logs for a Bulk operation e.g. Start all Service

abajwa — Tue, 01 Dec 2015 16:24:11 GMT

+ @Paul Codding and @bdurai

Once Logsearch is available you should be able to do this pretty easily from Logsearch UI because you can view all log messages across all components for the specific time window you are interested in (and filter by error level, include/exclude substrings etc). You can try a development version of it using steps here

Re: How to export all the output/error logs for a Bulk operation e.g. Start all Service

jstraub — Tue, 01 Dec 2015 16:58:03 GMT

Awesome project 🙂

Re: How to export all the output/error logs for a Bulk operation e.g. Start all Service

jstraub — Tue, 01 Dec 2015 17:15:28 GMT

@pankaj singh Another option would be to get the log files directly from the individual nodes.

For example, use the following request to get all the task information for a request:

http://horton01.example.com:8080/api/v1/clusters/bigdata/requests/111?fields=*,tasks/Tasks/request_id,tasks/Tasks/command,tasks/Tasks/command_detail,tasks/Tasks/host_name,tasks/Tasks/id,tasks/Tasks/role,tasks/Tasks/status&minimal_response=true

The response looks something like this =>

"tasks" : [
    {
      "Tasks" : {
        "command" : "CUSTOM_COMMAND",
        "command_detail" : "RESTART MAPREDUCE2/MAPREDUCE2_CLIENT",
        "host_name" : "horton01.example.com",
        "id" : 1157,
        "request_id" : 111,
        "role" : "MAPREDUCE2_CLIENT",
        "status" : "COMPLETED"
      }
    },
    {
      "Tasks" : {
        "command" : "CUSTOM_COMMAND",
        "command_detail" : "RESTART MAPREDUCE2/HISTORYSERVER",
        "host_name" : "horton02.example.com",
        "id" : 1158,
        "request_id" : 111,
        "role" : "HISTORYSERVER",
        "status" : "COMPLETED"
      }
    },

The import parts are the id and the host_name

Now you can log into the host and retrieve the error and output logfile from:

error: /var/lib/ambari-agent/data/errors-<id>.txt
output: /var/lib/ambari-agent/data/output-<id>.txt

For Example on host horton02.example.com:

error log: /var/lib/ambari-agent/data/errors-1158.txt
output log: /var/lib/ambari-agent/data/output-1158.txt

Note: You might also be able to get the request and tasks ids from the Ambari Database.

Re: How to export all the output/error logs for a Bulk operation e.g. Start all Service

pankaj_singh — Tue, 01 Dec 2015 18:12:27 GMT

It is helpful.

Re: How to export all the output/error logs for a Bulk operation e.g. Start all Service

abajwa — Wed, 02 Dec 2015 02:20:46 GMT

Thanks for sharing. So this would allow users to access the same output/errors available via ambari when you start/stop a service...but its possible that the actual root cause would only be available in the service's log, right?

Re: How to export all the output/error logs for a Bulk operation e.g. Start all Service

jstraub — Mon, 19 Aug 2019 12:45:03 GMT

exactly, its basically the content of these two fields from ambari UI =>

As you pointed out, the actual root cause is usually available in the individual service log -> /var/log/...