I'm attempting to write a shell script to capture the error output from a "kudu cluster kcsk <masters>" command, but it appears that the error output that is shown after the following text is possibly executed by a second command:
Even if I write the output from this command to a text file, this portion of the output isn't included. Does anyone know of a way to capture this using shell scripting? Basically, if there is any output, I simply want to create a custom alert that tells me kudu has some sort of problem and I need to investigate.
Any assistance is greatly appreciated.
As far as I know, kudu cluster ksck <kudu_master_rpc_endpoint> follows the standard convention and exits with non-zero status if it detects a problem with the cluster or detected other run-time issue. You could rely on the exit code of the kudu CLI tool to understand whether it detected any problem or not. In any case, the tool outputs information both into stdout and stderr in case of an error, so maybe make sure you capture both streams. However, the part after 'Errors:' is output into stdout. When running the tool, make sure it runs under credentials of the 'kudu' user (e.g., sudo -u kudu kudu cluster ksck ...) or whatever OS user kudu processes are run with: that's to pass authz checks.
I'm not sure what you mean by 'second command': the only command what is run is the kudu CLI tool itself when you execute 'kudu cluster ksck ...'. Maybe, try first to experiment with the tool in a plain interactive shell environement and see what you get. As of now with kudu CLI binary from 1.9.0 release, I'm able to capture all the output of 'kudu cluster ksck' with no issues when running it in a standard bash session.
However, you might be hitting this bug: https://issues.apache.org/jira/browse/KUDU-2819 The bug rarely manifested itself and has been found and fixed just recently. So, make sure the kudu CLI tool doesn't crash when you run it against your cluster. If it does, get or build a binary with the fix for KUDU-2819 included and use the one.
Thanks very much for your response.
What I mean by "runs a second command" is that if you execute something like this:
kudu cluster ksck masters.domain.com > output.txt
You will notice that the "Errors:" section at the very bottom of the command output isn't actually written to the file. This holds true if you attempt to store the output in a variable. This leads me to believe that part of the output isn't actually part of the original command execution.
I've found a way to work around this in shell programatically (if you could call it that). I found the Linux "script" command, which captures output to screen, rather than just the output from a command itself. I had to use the "-e -c" parameters to get it to function as a cron job for automated monitoring / alerting, but it appears to work as I desired.
Hopefully anyone else looking to perform automated monitoring/alerting of things not covered by Cloudera Manager for Kudu find this post helpful.