Small shell tools for text editing

Slashdot it! Delicious Share on Facebook Tweet! Digg!

Sorting Files

You often might need the sort function to sort out multiple identical lines from a pipe. Of course, sort also serves for all other sorting tasks. Table 12 gives a quick overview of the sort functions. Note that it's not just about forward and backward sorting, but also about what is being sorted.

Table 12

The following command:

cat *.txt | sort -u

outputs all items in a folder, with multiple occurrences of identical lines suppressed.

Removing Identical Lines

In overflowing log files with tons of consecutive identical messages, the uniq command helps provide clarity. However, it can also be used when searching for duplicates. To handle duplicates, the input or piped data needs to be sorted. Table 13 includes a few of the most important options.

Table 13

In the first example that follows, you can concatenate all the text files, sort them together, and output only the duplicate lines prefixed with their occurrence count:

cat *.txt | sort |uniq -dc

Next, you can extract only the unique lines from the data sources:

cat *.txt | sort | uniq -u

And, in this example:

cat *.txt | sort | uniq

all duplicates are removed from the pipe output and both the unique and duplicate lines are displayed.

Buy this article as PDF

Express-Checkout as PDF

Pages: 1

Price $0.99
(incl. VAT)

Buy Ubuntu User

Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content