Small shell tools for text editing
|
Sorting Files
You often might need the sort function to sort out multiple identical lines from a pipe. Of course, sort also serves for all other sorting tasks. Table 12 gives a quick overview of the sort functions. Note that it's not just about forward and backward sorting, but also about what is being sorted.
Table 12
The following command:
cat *.txt | sort -u
outputs all items in a folder, with multiple occurrences of identical lines suppressed.
Removing Identical Lines
In overflowing log files with tons of consecutive identical messages, the uniq command helps provide clarity. However, it can also be used when searching for duplicates. To handle duplicates, the input or piped data needs to be sorted. Table 13 includes a few of the most important options.
Table 13
In the first example that follows, you can concatenate all the text files, sort them together, and output only the duplicate lines prefixed with their occurrence count:
cat *.txt | sort |uniq -dc
Next, you can extract only the unique lines from the data sources:
cat *.txt | sort | uniq -u
And, in this example:
cat *.txt | sort | uniq
all duplicates are removed from the pipe output and both the unique and duplicate lines are displayed.
Buy this article as PDF
Pages: 1
(incl. VAT)
Buy Ubuntu User
US / Canada
UK / Australia
Related content
-
moreutils completes the shell's toolbox
The moreutils package expands the standard tools for the shell with useful and sometimes exotic tools.
-
Accelerated command processing with GNU Parallel
With the snazzy little program GNU Parallel, you can make use of the full power of your multicore CPUs through scripts.
-
Awk as tool and scripting language
In Linux, the need for text processing programs is a big one: System configuration, system management, and data exchange all use text-based files. With Awk, you have a powerful tool at your fingertips for text editing and targeted modification.
-
Scanning and editing text with OCR
With a small script, you can convert large amounts of scanned text into PDF files that you can then browse with typical Linux tools – all thanks to OCR.
-
From shell script to compact tool
A simple script is usually enough to automate a knotty task in Linux, but with some simple shell commands and a little effort, you can convert a simple batch file into a robust shell program.