Frequently, administrators need to take care of several servers and clients that have a similar configuration, such as in a company, an Internet café, or the IT laboratory for a school. Although tools such as Puppet and Cfengine make the work easy, they also have steep learning curves. Therefore, it can be easier to rely on built-in tools when dealing with smaller installations.

SSH is a standard tool for administering remote computers. The secure shell makes secure login over a network possible so that necessary modifications can be done on the remote server followed by restarts for the affected services. This works well with just one remote computer, but when several are involved, the process becomes tedious.

Entering a password is the first issue to address in the multiple computer scenario. If you have to log in on multiple computers to make modifications, then is there a way to do this without typing in the password for each computer? Regular and automatic backups also have to run even though the administrator has taken a vacation or has gone to bed for the night. What are the possible solutions for these and other concerns?

SSH Login Without a Password

SSH normally expects password entry via a keyboard. There is no command-line option for this action. Actually, this is a positive state of affairs. Otherwise, any user could look at all current processes including the command-line parameters via ps ax . This would in turn allow a user to read the password belonging to other users in plain text.

SSH offers the public key encryption method for logins without a password. The user then needs to know the login name along with the corresponding password, plus have a private key that matches the public key on the server.

Listing 1 shows how to create a key pair. The -N "" option lets you indicate that you do not want to use a passphrase. This will save a keyboard entry. SSH keygen creates two files. These include the private key, which should remain secret, in the mysshkey file, and the public key in the mysshkey.pub file. The latter has to be on the server(s) where you will log on with your private key (see the "Password Tips" box for more).

$ cd ~/.ssh
$ ssh-keygen -N "" -f mysshkey
Generating public/private rsa key pair.
Your identification has been saved in mysshkey.
Your public key has been saved in mysshkey.pub.
The key fingerprint is:
a6:d2:c5:e9:5b:80:10:a0:a6:ef:d5:6a:3d:03:df:d0 user@hostname
The key's randomart image is:
+--[ RSA 2048]----+
|  ...            |
| .   .           |
|..  .            |
|o    . o .       |
|.     ..S        |
| .  .o.=E.       |
|  . o++o. .      |
| . ..o= .o       |
|  ...  o.        |
+-----------------+
$ ssh-copy-id -i mysshkey.pub user@myserver
Password:
Now try logging into the machine, with "ssh 'user@myserver'", and check in:
 .ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.

The ssh-copy-id (line 20) command performs the key transfer once you log in one more time using a password. For the first log in, you will also need to respond when the system asks whether the fingerprint for the remote computer is trusted (see the "Identity Check" box). The login will work from then on without a password when the following command is issued:

$ ssh -i ~/.ssh/mysshkey user@myserver

$ ssh www.example.com
The authenticity of host 'www.example.com (192.0.2.1)' can't be established.
ECDSA key fingerprint is ad:57:60:2b:53:c5:08:07:8b:b3:26:87:1d:2d:5a:b5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'www.example.com' (ECDSA) to the list of known hosts.
Password:
Last login: Sat Sep 12 14:41:29 2015 from otherhost.example.com
Have a lot of fun...

What is the best way to automate tasks? The easiest option is to use a simple for loop in the shell as shown in Listing 3. In this listing, the COMMAND variable, when executed, installs the pssh package on computers ranging from host1 to host4 by using the Debian/Ubuntu apt tool. In the process, the apt option -y avoids queries.

HOSTS="host1 host2 host3 host4"
COMMAND="apt-get -y install pssh"
for i in $HOSTS ; do
 ssh -o ConnectTimeout=10 -i ~/.ssh/mysshkey $i -c "$COMMAND"
done

SSH itself interrupts the unproductive connection attempts after 10 seconds (-o ConnectTimeout=10 ), when it cannot reach an opposite computer. It then continues on to connecting with the next computer. In the absence of this precautionary measure, the for loop would get hung up and not proceed to work on the other computers. This type of loop offers a very practical solution, but there are more sophisticated methods available. One of these is hidden inside the pssh package that was installed in the example.

Parallel SSH

The pssh program makes it possible to execute ssh commands on several computers in parallel [1]. To do this, you will need to have a running SSH server on the clients, and pssh needs to be installed on the control server . In effect, the example above, in which pssh was installed on all servers, was unnecessarily complicated.

Figure 1 shows a parallel ssh session initiated by pssh. First, you need to create a file, which contains a list of computers to be used. I have called this file hostfile in the example. It contains a host name on each line, optionally with accompanying user name and port in the form of <User>@<Host>:<Port> .

Figure 1: Parallel SSH sessions can be started via pssh.

Then, you start the parallel login on the indicated host computers via the following command:

$ pssh -i -x "-i ~/.ssh/mysshkey"   -t 10 -h hostfile command

The -i option indicates that output should be sent to the terminal. Alternatively, you could pipe output into files, one per target computer. The -t 10 option indicates a timeout of 10 seconds. The -x parameter in turn has as its value an option that will be forwarded to ssh. In the example, this is "-i ~/.ssh/mysshkey" , which indicates usage of the ssh key.

The login on localhost worked fine in Figure 1. In this example, I did not accept the SSH hostkey nor an account for http://www.linuxuser.de. Therefore, the login there won't work. The timeout interrupts the login to http://www.google.at. As expected, there is not publicly available SSH server there.

The pssh package includes pscp and prsync in addition to the pssh tool, which is used to execute SSH commands on multiple computers simultaneously. All of these tools are used to copy files onto multiple computers with scp and rsync .

Cluster SSH

Cluster SSH [2] also offers the possibility of working on multiple computers simultaneously. The invocation is made via cssh <host1> <host2> … . A small text entry window then opens (Figure 2) where you can enter the desired command. A terminal window will also open for each computer. Cssh will transfer the entry character by character from the entry window to the individual devices, even when you have mistyped and then used the backspace to correct the error. You will be able to see how this works on the individual terminals. Then, you can follow the output of the commands on the individual hosts once the return key is pressed.

Figure 2: The text entry window for Cluster SSH.

To execute a separate command on one of the computers, you should shift the focus to its terminal. There you should modify the command as necessary before you press the return key in the Cssh window to execute the command on all computers.

If desired, you can define the computers in the cluster in a configuration file (/etc/clusters or ~/.clusterssh/clusters ). This is similar to what you can do in Pssh. The definition here proceeds by assigning a name and then lists the computers that belong to this class. The computer name assigned in Listing 4 is webserver . When you call cssh, you can call this class instead of the names of the individual computers [3]. The Cluster SSH package includes both cssh and crsh for RSH and also ctel for Telnet. Neither protocol offers encryption, and they are therefore considered irrelevant nowadays.

webserver root@www1.example.com root@www2.example.com root@www3.example.com

Limitations

When logins are made without a password, you should figure out for security reasons how to restrict command execution to only those tasks which are the most important. This would include things like triggering backups, indicating currently running processes, or restarting the server. SSH makes these types of restrictions possible via so-called forced commands. It then always executes a particular command at login.

When the key gets copied onto the server, it lands in the ~/.ssh/authorized_keys file. The corresponding line begins with ssh-rsa . RSA stands for encryption method being used. Next comes a jumble of characters that form your public key, and an optional comment. If you do not specify anything else, then this comment consists of the <User>@<Host> for the computer on which you have created the key. You can enter a forced command as needed at the beginning of the line.

command="ps -ef" ssh-rsa AAB3NzaC1yc..

In this example, the ps -ef command for indicating running processes always starts after login with the key. This is true even if you have given a different command to ssh. This could be changed to something like initiating a backup, restarting a service, or rebooting the server. Because you can create arbitrarily many keys, it is possible to use a different key for each task together with the accompanying forced command.

All of this can be simplified. SSH assigns to the variable $SSH_ORIGINAL_COMMAND the command that you indicated. This is called whitelisting a command. It is done by applying the shell script in Listing 5, which you save on the server for use as a forced command. Then, the available disk memory can be determined via ssh -i ~/.ssh/mysshkey server.example.com df . In the same vein, the currently executing processes can be determined via … ps , and the available main memory via … free .

#!/bin/sh
case $SSH_ORIGINAL_COMMAND in
 "df")
 df -h
 ;;
 "ps")
 ps -ef
 ;;
 "free")
 free
 ;;
 *)
 echo "Illegal command."
 exit 1
 ;;
esac

Although this is a good idea, it can be abused. (See the "Tricky Aspects of Forced Commands" box.) It is possible to restrict the list of computers for which the key can be used as a login via the from="…" option.

from="*.example.org,!chef.  example.org" ssh-rsa ...

This key can be used to log into each computer under example.org , but not from the chef computer thanks to the negation indicated by the exclamation mark.

One final step may be to turn off various capabilities of ssh. The protocol offers exciting options such as X11 and port forwarding. You should deactivate these curiosities if they are not absolutely necessary. Otherwise, an interloper can gain access to certain services via port forwarding. The corresponding options are summarized in Table 1. More detailed information is available on the manpage of the ssh daemon (man sshd ).

Further Automation

A login without password is typically appropriate for routine tasks which do not require the assistance of an administrator. In the case of night-time backups, for example, you would create a cronjob [4] that executes the backup when the computer is not being used. The simplest case involves the use of scp to back up data from the home directory (Listing 6, line 1). The -p option ensures that access rights and times are preserved, although this does not hold true for file owners.

$ scp -r -p -i ~/.ssh/mysshkey root@myserver:/home backupdirectory
$ ssh -i ~/.ssh/mysshkey root@myserver "tar cvzf - /home" >myserverbackup.tar.gz
$ rsync -az -e "ssh -i ~/.ssh/mysshkey" myserver:/home myserverbackup

A further possibility and one that is relatively unknown consists of transferring the data directly via ssh. This can be done for example by creating a tar backup and writing it to STDOUT, which is indicated by (- . The data is then transferred via the ssh connection and is then locally copied via the redirect operator > to the backup file (Listing 6, line 2).

Rsync offers yet another possibility for making backups. It only transfers data that has changed. In the example, the command from the last line of Listing 6 writes the data of the home directory on the myserver server into the local myserverbackup/ directory.

The -e "ssh -i ~/.ssh/mysshkey" option ensures that the call to the remote shell ssh is given the -i ~/.ssh/mysshkey option. This is the mechanism for logging in with the private key. To secure Rsync with a forced command, you should enter the -v ("verbose") option twice (Listing 7). Rsync then indicates which command should be executed on the remote server.

$ rsync -avvz -e "ssh -i ~/.ssh/mysshkey" myserver:/home myserverbackup
opening connection using: ssh -i "~/.ssh/mysshkey" myserver rsync --server --sender -vvlogDtprze.iLs . /home (10 args)

You can then record "rsync --server --sender -vvlogDtprze.iLs . /home" without the verbose options as a forced command on the remote server. This makes the key usable only for the Rsync backups from /home . The options --server --sender are rsync internal, which under normal circumstances you would never set.

The remaining options in the example allow features such as symlinks, owners, groups, device files, time stamps, etc. to continue to be available. Via the -v option, scp can also issue a command, including internal options, for remote execution(sending command: … ). You can use this as a forced command provided you only use the accompanying key for copies via scp.

Conclusion

You don't always need a big, full-blown solution to manage servers. Administrators of multiple computers can also make their work easier by using simpler programs. SSH offers all of the basic requirements in this regard. With some shell scripting and small additional tools, you can definitely make your administration tasks go more smoothly.