Downloads and analysis with HTTPie

Scripting

This is not a tool for homemade Internet surveillance. However, you can access device controls via a web interface or even pages with defined values, and so on. Often these have no mail interface, so you would have to access the web page at irregular intervals.

The shell script in Listing 2 monitors the contents of a simple web page (like what you can see in Listing 3). If the keyword "empty" appears, everything is fine. A mail for this is only sent when the script is started or after an error has been reported and then amended. If something changes, then an error message is sent. The target mailbox is not flooded with identical error messages thanks to a flag that keeps track of when changes occur.

Listing 2

control.sh

#!/bin/bash
# Usage: control.sh <site URL> <user to email>
# Variable to accept the exit code
l=0
# Variable that memorizes the previous state
flag=1
while true; do
 http -v $1 | grep "empty"> /dev/null
 l=$(echo $?)
 if [ $l -eq 0 ]; then
   if [ $flag -eq 1 ]; then
     echo "Message: empty" | mail -s "Normal Message" $2
     # Reset $flag to "Normal"
     flag=0
   fi
 else
   if [ $flag -eq 0 ]; then
     echo "Message: error" | mail -s "ERROR MESSAGE" $2
     # Set $flag to "Error"
     flag=1
   fi
 fi
 sleep 10
done

Listing 3

testsite.html

<html>
 <body>
  <h1>Test Site</h1>
  <p>192.169.124 Test field<p>
  <p>(Keyword: empty)<p>
 </body>
</html>

The script takes two parameters: the URL of the webpage you want to monitor, and the user to whom you want to send the warning emails. Running the script would look like this:

control.sh www.somesite.com/test.html <user>

For all of this to work, you will have to install the mailutils package:

sudo apt install mailutils

Choose Internal only from the options the installer gives you, and then set up mail to work with your user. You do this by first making your user join the the mail group:

sudo usermod -a -G mail <user>

Next you have to create an email box for user by sending an internal email to the user. You can do this as root with:

sudo mail -s "First email" <user>

You can see the internal emails you receive by simply running mail .

Figure 7 shows what happens during a test run: new emails show up in you mailing list. Figure 8 (top) shows the sample page, which was altered (bottom) for demonstration purposes while the script was running.

Figure 7: Overview of the status emails.

Figure 8: Above, the unmodified monitored page; below the modified monitored page.

Conclusion

HTTPie is a compact, well-rounded tool for data download and analysis. This article covers some of the basic functions. The other notes on the project page and on the man page are always a worthwhile read.

Infos