Basic Data Extraction and Text Processing in Linux
cut, sort, uniq, wc
By themselves, these old *nix tools can’t accomplish much. But when used together, the possibilities for what we can accomplish with data processing are endless. Familiarizing yourself with these tools will unlock a new level of competency. Other data filtering tools not mentioned here but are worth mentioning are
- Understand how piping commands works in linux
- Know how to use the
cut command [man cut]
Take a look at this command
cut -d " " -f 1
In this example, the cut command defines a delimiter
-d " " and selects the first field
-f 1. In other words, the cut operation will find the first instance of a space and then cut everything after that space, for every line. The way field parameter works is that each delimiter (
" " in this case) creates a new field and you can decide which fields to keep.
sort command [man sort]
The sort command does what it sounds like it does. It will take in a bunch of unsorted lines and sort them. If using the sort command without any arguments, its default sort rules are as follows:
- numbers > letters
- lowercase > uppercase
If we have a list like
The default sort rules would output this
Notice that the numbers don’t seem to be in order. The default sort compares each character individually. To sort numbers by integer order, you can use
uniq command [man uniq]
Running some text through the default
uniq command will output the text file without duplicate lines. Another useful argument,
uniq -c will output each line prepended with a count of how many times that line occurred. This is useful if you want to analyze the frequency of certain lines in your data.
wc command [man wc]
Wc stands for wordcount, but the
wc command can count more than just words. In many cases with text processing and analysis, we want a line count.
wc -l will do just that. Check out the man link above for more things wc can do.
Putting it all together
Here’s the structure of a sample
access.log file from an IIS webserver
02:49:12 127.0.0.1 GET / 200
02:49:35 127.0.0.1 GET /index.html 200
03:01:06 127.0.0.1 GET /images/sponsered.gif 304
03:52:36 127.0.0.1 GET /search.php 200
04:17:03 127.0.0.1 GET /admin/style.css 200
05:04:54 127.0.0.1 GET /favicon.ico 404
05:38:07 127.0.0.1 GET /js/ads.js 200
If we wanted to get the frequency of each http status code we would use a command like
cat access.log | cut -d " " -f 5 | sort -n | uniq -c Which would return
So there are 5 occurences of 200 status codes, 1 occurance of a 304, and 1 occurance of a 404.