Using Expect scripts to automate tasks

Creating an Interactive SSH Session with Autoexpect

After installing Autoexpect and all of its required packages, you're ready to create Expect scripts automatically by stepping through the procedures you want to automate.

Using the above example, SSH to a remote system, run

ps -ef |grep apache

and then log out.

Invoking Autoexpect is easy:

$ Autoexpect
Autoexpect started, file is script.exp
$

Although it looks as if nothing has happened or is happening, every keystroke you type will be recorded into script.exp . Every STDOUT response you receive will also be copied into that same file.

Your entire session is recorded – but not just recorded, it is also formatted in Expect script style. To stop recording keystrokes to your script, press Ctrl+D on your keyboard to stop Autoexpect and copy the buffer to your file. The complete transcription of this simple procedure is very long and includes a lot of commentary from the author, Don Libes (Listing 2).

Listing 2

Autoexpect-Generated Script

01 #!/usr/bin/expect -f
02 #
03 # This Expect script was generated by Autoexpect on Thu Oct 11 15:53:18 2012
04 # Expect and Autoexpect were both written by Don Libes, NIST.
05 #
06 # Note that Autoexpect does not guarantee a working script.  It
07 # necessarily has to guess about certain things.  Two reasons a script
08 # might fail are:
09 #
10 # 1) timing - A surprising number of programs (rn, ksh, zsh, telnet,
11 # etc.) and devices discard or ignore keystrokes that arrive "too
12 # quickly" after prompts.  If you find your new script hanging up at
13 # one spot, try adding a short sleep just before the previous send.
14 # Setting "force_conservative" to 1 (see below) makes Expect do this
15 # automatically - pausing briefly before sending each character.  This
16 # pacifies every program I know of.  The -c flag makes the script do
17 # this in the first place.  The -C flag allows you to define a
18 # character to toggle this mode off and on.
19
20 set force_conservative 0  ;# set to 1 to force conservative mode even if
21                           ;# script wasn't run conservatively originally
22 if {$force_conservative} {
23         set send_slow {1 .1}
24         proc send {ignore arg} {
25                 sleep .1
26                 exp_send -s -- $arg
27         }
28 }
29
30 #
31 # 2) differing output - Some programs produce different output each time
32 # they run.  The "date" command is an obvious example.  Another is
33 # ftp, if it produces throughput statistics at the end of a file
34 # transfer.  If this causes a problem, delete these patterns or replace
35 # them with wildcards.  An alternative is to use the -p flag (for
36 # "prompt") which makes Expect only look for the last line of output
37 # (i.e., the prompt).  The -P flag allows you to define a character to
38 # toggle this mode off and on.
39 #
40 # Read the man page for more info.
41 #
42 # -Don
43
44 set timeout -1
45 spawn $env(SHELL)
46 match_max 100000
47 expect -exact "]0;USER@HOST: ~USER@HOST:~\$ "
48 send -- "ssh SERVER\r"
49 expect -exact "ssh SERVER\r
50 USER@HOST's password: "
51 send -- "PASSWORD\r"
52 expect -exact "\r
53 Linux SERVER 2.6.32-43-server #97-Ubuntu SMP Wed Sep 5 16:56:41 UTC 2012 x86_64 GNU/Linux\r
54 Ubuntu 10.04.4 LTS\r
55 \r
56 Welcome to the Ubuntu Server!\r
57  * Documentation:  http://www.ubuntu.com/server/doc\r
58 \r
59   System information as of Thu Oct 11 15:55:28 CDT 2012\r
60 \r
61   System load:  1.09               Temperature:         40 C\r
62   Usage of /:   1.0% of 454.22GB   Processes:           168\r
63   Memory usage: 22%                Users logged in:     1\r
64   Swap usage:   0%                 IP address for eth0: 192.168.1.250\r
65 \r
66   Graph this data and manage this system at https://landscape.canonical.com/\r
67 \r
68 7 packages can be updated.\r
69 7 updates are security updates.\r
70 \r
71 New release 'precise' available.\r
72 Run 'do-release-upgrade' to upgrade to it.\r
73 \r
74 *** System restart required ***\r
75 Last login: Thu Oct 11 15:53:41 2012 from HOST\r\r
76 ]0;USER@HOST: ~USER@HOST:~\$ "
77 send -- "ps -ef|grep apache\r"
78 expect -exact "ps -ef|grep apache\r
79 www-data   555 23171  0 Oct07 ?        00:00:00 /usr/sbin/apache2 -k start\r
80 www-data   556 23171  0 Oct07 ?        00:00:00 /usr/sbin/apache2 -k start\r
81 www-data   557 23171  0 Oct07 ?        00:00:00 /usr/sbin/apache2 -k start\r
82 www-data   558 23171  0 Oct07 ?        00:00:00 /usr/sbin/apache2 -k start\r
83 www-data   559 23171  0 Oct07 ?        00:00:00 /usr/sbin/apache2 -k start\r
84 khess    21504 21433  0 15:55 pts/1    00:00:00 grep apache\r
85 root     23171     1  0 Sep27 ?        00:00:28 /usr/sbin/apache2 -k start\r
86 ]0;USER@HOST: ~USER@HOST:~\$ "
87 send -- "exit\r"
88 expect -exact "exit\r
89 logout\r
90 Connection to SERVER closed.\r\r
91 ]0;USER@HOST: ~USER@HOST:~\$ "
92 send -- "^D"
93 expect eof
94 USER@HOST:~$

You can see from the listing that you have a lot of cleanup to do before you distill this transcript down to its essential parts. Autoexpect also changes permissions on the script.exp file so that it is executable. The parts you needed for this script to execute correctly are shown in Listing 3 in my cleaned up version.

Listing 3

Cleaned Up Autoexpect Script

01 #!/usr/bin/expect -f
02
03 set force_conservative 0  ;# set to 1 to force conservative mode even if
04                           ;# script wasn't run conservatively originally
05 if {$force_conservative} {
06         set send_slow {1 .1}
07         proc send {ignore arg} {
08                 sleep .1
09                 exp_send -s -- $arg
10         }
11 }
12
13 set timeout -1
14 spawn $env(SHELL)
15 match_max 100000
16 expect -exact "$ "
17 send -- "ssh SERVER\r"
18 expect -exact "password: "
19 send -- "PASSWORD\r"
20 expect -exact "$ "
21 send -- "ps -ef|grep apache\r"
22 expect -exact "$ "
23 send -- "exit\r"
24 expect -exact "$ "

You can see that the complex prompts, such as

expect -exact "exit\r
logout\r
Connection to <SERVER> closed.\r\r
]0;<USER>@<HOST>: ~<USER>@<HOST>:~\$ "

have been shortened significantly to:

expect -exact "$ "

The prompt still works because Expect looks for the last few characters in an expect line and not the entire string.

You could shorten the line that expects the password prompt from:

expect -exact "password: "

expect -exact ": "

A word of caution against shortening your Expect lines too much: It makes the script more difficult, not easier, to read and interpret in the future when you try to figure out what's going on. You might not realize that ": " is a password prompt. Unless you're great at including comments in your scripts, you might spend hours debugging this shortened version.

Conclusion

To be perfectly honest, I only use Autoexpect when building an Expect draft script. To sit down and attempt writing an Expect script line by line just isn't appealing after being seduced and ruined by the ease of removing unwanted lines from an Autoexpect-created script [3].

Autoexpect makes using Expect fun and more intuitive by letting you perform a procedure one time instead of many. After discovering and using Autoexpect, my Expect scripting creation time and debug time has been cut by at least two-thirds. I suspect you'll have much the same return on your time as well.

Infos