Using SaltStack to run commands on HDCloud and Cloudbreak

[#1]

Using SaltStack to run commands on HDCloud and Cloudbreak

HDCloud and Cloudbreak make it easy to provision, configure and elastically grow HDP clusters on cloud infrastructure. During provisioning time Cloudbreak can execute recipes on the nodes participating in the cluster. Recipes are simple script written in bash or python or any other scripting language available on the nodes.

It is common that users would like to execute ad-hoc commands (e.g. collecting logs, installing extra packages, executing scripts) on nodes not only during provisioning time, but when the cluster is used.

Infrastructure management of Cloudbreak

Under the hood Cloudbreak uses SaltStack to manage nodes of the cluster, install packages, change configuration files and execute recipes. After provisioning phase, users can take advantage of this infrastructure management tool and execute their own scripts and create their own Salt States.

Connectivity check

By default Salt master is installed on one of the master nodes, more specifically on the same node where Ambari server is available. In order to run simple salt connectivity command, just ssh to the Ambari server machine and execute the following command:

sudo salt '*' test.ping

Remote command execution

Running commands on remote systems is the core function of Salt, it can execute arbitrary commands across your cluster completely parallel. Execute a command on all nodes of your cluster:

sudo salt '*' cmd.run 'echo hello'

Targeting commands

If you would like to execute commands only on specific nodes the you can use the targeting mechanism of Salt. E.g execute the first command on master node(s) and the 2nd command on worker node(s):

sudo salt -G 'hostgroup:master' cmd.run 'yarn application -list'
sudo salt -G 'hostgroup:worker' cmd.run 'free -h'

Targeting is very flexible in Salt, you can read about this in the Salt documentation. Probably one of the most common targeting option is to use Salt Grains. There are additional Salt Grains defined by Cloudbreak beyond the standard Grains supported by SaltStack such as hostgroup and roles. You can list all supported grains with:

sudo salt '*' grains.items

Creating your own Salt State

If you would like to create more complex things than executing a simple command you can create your own Salt State. E.g you can create a state which installs multiple packages, by saving the following file under /srv/salt/nettools/init.sls:

install_network_packages:
  pkg.installed:
    - pkgs:
      - rsync
      - lftp
      - curl

You can execute this new state on every node with:

salt '*' state.apply nettools

There is a much more elegant way to include your pre-written Salt States, for that you can take a look at Salt Formulas.