Mastering awk and grep: A Practical Foundation Guide

Introduction

When processing text in Linux and Unix-like environments, awk and grep are indispensable commands.

Understanding the basic syntax and usage of awk and grep allows you to streamline log analysis and CSV processing.

This article explains everything from the basics of awk and grep to practical applications in an easy-to-understand way.

Reference: GNU awk
Reference: GNU grep

Basic Syntax and Usage of awk and grep

Creating the File

cat << 'EOF' > input.txt
apple 100
banana 200
orange 150
apple 300
grape 120
EOF

Command

grep "apple" input.txt

Output

apple 100
apple 300

Command

awk '{print $1, $2}' input.txt

Output

apple 100
banana 200
orange 150
apple 300
grape 120

Command

awk '$2 >= 150 {print $1, $2}' input.txt

Output

banana 200
orange 150
apple 300

Command

grep "apple" input.txt | awk '{sum += $2} END {print sum}'

Output

400

How It Works

Command	How It Works	Executable Command
grep	Searches for lines containing the specified string	grep "apple" input.txt
awk	Splits lines by delimiter and processes each column	awk '{print $1, $2}' input.txt
awk condition	Extracts only lines matching the specified condition	awk '$2 >= 150 {print $1, $2}' input.txt
grep + awk	Aggregates grep results using awk	grep "apple" input.txt | awk '{sum += $2} END {print sum}'

Explanation

awk excels at column-level processing and aggregation, while grep specializes in string searching.
Combining them enables efficient log analysis and data extraction.

How to Use awk and grep Appropriately

Creating the File

cat << 'EOF' > input.txt
apple 120
banana 80
orange 150
grape 90
apple 200
EOF

Command

grep 'apple' input.txt

Output

apple 120
apple 200

Command

awk '$2 >= 100 { print $1, $2 }' input.txt

Output

apple 120
orange 150
apple 200

How It Works

Command	How It Works	Executable Command
grep	Extracts lines containing the specified string	grep 'apple' input.txt
awk	Evaluates conditions by column and processes/extracts data	awk '$2 >= 100 { print $1, $2 }' input.txt

Explanation

grep specializes in string searching and can be used at high speed for log searches.

awk supports column processing and conditional branching, making it suitable for processing tabular data.

How to Use Regular Expressions with awk and grep

Creating the File

cat << 'EOF' > input.txt
error: disk full
INFO: process started
warning: cpu high
ERROR: memory leak
user login success
EOF

Command

grep -E 'error|warning' input.txt

Output

error: disk full
warning: cpu high

Command

grep -Ei 'error|warning' input.txt

Output

error: disk full
warning: cpu high
ERROR: memory leak

Command

awk '/error|warning/' input.txt

Output

error: disk full
warning: cpu high

Command

awk 'tolower($0) ~ /error|warning/' input.txt

Output

error: disk full
warning: cpu high
ERROR: memory leak

How It Works

Command	How It Works	Executable Command
grep -E	Enables extended regular expressions with -E and performs OR search using |	grep -E 'error|warning' input.txt
grep -Ei	Searches case-insensitively with -i	grep -Ei 'error|warning' input.txt
awk	Extracts matching lines using /regex/	awk '/error|warning/' input.txt
awk tolower	Converts the entire line to lowercase with tolower($0) before comparing	awk 'tolower($0) ~ /error|warning/' input.txt

Explanation

grep is well-suited for fast string searching and is commonly used for log monitoring and filtering.

awk supports conditional branching and column processing in addition to regular expressions, making it useful for complex text analysis.

How to Perform Exact Match and Partial Match Searches with awk and grep

Creating the File

cat << 'EOF' > input.txt
apple
apple pie
banana
banana split
grape
pineapple
EOF

Command

grep -x "apple" input.txt

Output

apple

Command

grep "apple" input.txt

Output

apple
apple pie
pineapple

Command

awk '$0=="apple"' input.txt

Output

apple

Command

awk '/apple/' input.txt

Output

apple
apple pie
pineapple

How It Works

Command	Match Type	Executable Command	How It Works
grep	Exact match	grep -x "apple" input.txt	Using -x displays only lines where the entire line matches exactly
grep	Partial match	grep "apple" input.txt	Searches for lines containing the specified string
awk	Exact match	awk '$0=="apple"' input.txt	$0 represents the entire line; == performs exact match comparison
awk	Partial match	awk '/apple/' input.txt	Searches for lines matching the pattern using /string/

Explanation

grep can easily switch between exact and partial matching using options.

awk allows flexible searching and advanced text processing through the use of conditional expressions.

How to Search Case-Insensitively with awk and grep

Creating the File

cat << 'EOF' > input.txt
Apple
apple
Banana
BANANA
Cherry
EOF

Command

grep -i "apple" input.txt

Output

Apple
apple

Command

awk 'tolower($0) ~ /banana/' input.txt

Output

Banana
BANANA

How It Works

Command	How It Works	Executable Command
grep	Searches case-insensitively using the -i option	grep -i "apple" input.txt
awk	Converts the string to lowercase using tolower() before comparing	awk 'tolower($0) ~ /banana/' input.txt

Explanation

grep can easily ignore case by simply adding -i.
awk enables flexible conditional searching by using tolower().

How to Perform AND Search and OR Search with awk and grep

Creating the File

cat << 'EOF' > input.txt
apple orange banana
apple grape
banana melon
orange lemon
apple orange
EOF

Command

grep 'apple' input.txt | grep 'orange'

Output

apple orange banana
apple orange

Command

awk '/apple/ && /orange/' input.txt

Output

apple orange banana
apple orange

Command

grep -E 'apple|orange' input.txt

Output

apple orange banana
apple grape
orange lemon
apple orange

Command

awk '/apple|orange/' input.txt

Output

apple orange banana
apple grape
orange lemon
apple orange

How It Works

Search Method	Command	How It Works
grep AND search	grep 'apple' input.txt | grep 'orange'	Pipes the first grep result to the second grep to extract lines containing both
awk AND search	awk '/apple/ && /orange/' input.txt	Extracts lines satisfying multiple conditions simultaneously using &&
grep OR search	grep -E 'apple|orange' input.txt	Extracts lines matching either condition using |
awk OR search	awk '/apple|orange/' input.txt	Extracts lines matching the OR condition in the regular expression

Explanation

grep achieves AND search using pipes and can concisely write OR search with -E.

awk allows flexible combinations of conditions using && and |.

How to Specify Multiple Conditions with awk and grep

Creating the File

cat << 'EOF' > input.txt
Alice Tokyo Sales
Bob Osaka Engineering
Carol Tokyo Engineering
Dave Nagoya Sales
Eve Tokyo Marketing
EOF

Command

grep 'Tokyo' input.txt | grep 'Engineering'

Output

Carol Tokyo Engineering

Command

awk '/Tokyo/ && /Engineering/' input.txt

Output

Carol Tokyo Engineering

How It Works

Command	How It Works
grep 'Tokyo' input.txt | grep 'Engineering'	Chains grep with a pipe to extract only lines containing both "Tokyo" and "Engineering"
awk '/Tokyo/ && /Engineering/' input.txt	Uses && in awk to display only lines matching both conditions

Explanation

In awk, the logical operator && allows flexible combinations of multiple conditions.
Adding conditions or including numeric comparisons is also easy to extend.

How to Search for Multiple Keywords with awk and grep

Creating the File

cat << 'EOF' > input.txt
Linux awk command
grep is useful
sed and awk together
find with grep
python script
EOF

Command

grep -E 'awk|grep' input.txt

Output

Linux awk command
grep is useful
sed and awk together
find with grep

Command

awk '/awk|grep/' input.txt

Output

Linux awk command
grep is useful
sed and awk together
find with grep

How It Works

Command	How It Works	Executable Command
grep	Enables extended regular expressions with -E and performs OR search for multiple keywords using |	grep -E 'awk|grep' input.txt
awk	Searches for multiple keywords using OR conditions within /condition/	awk '/awk|grep/' input.txt

Explanation

grep is commonly used for high-speed multi-keyword searching.
awk can also perform column processing and conditional branching at the same time as searching.

How to Extract Specific Columns with awk and grep

Creating the File

cat << 'EOF' > input.txt
id,name,department,salary
1,Alice,Sales,5000
2,Bob,Engineering,7000
3,Carol,Marketing,6000
4,David,Engineering,8000
EOF

Command

awk -F',' '{print $2, $4}' input.txt

Output

name salary
Alice 5000
Bob 7000
Carol 6000
David 8000

Command

grep 'Engineering' input.txt

Output

2,Bob,Engineering,7000
4,David,Engineering,8000

Command

grep 'Engineering' input.txt | awk -F',' '{print $2, $4}'

Output

Bob 7000
David 8000

How It Works

Command	How It Works
awk -F',' '{print $2, $4}' input.txt	Sets the delimiter to , with -F',' and extracts the 2nd and 4th columns using $2 and $4
grep 'Engineering' input.txt	Extracts only lines containing Engineering
grep 'Engineering' input.txt | awk -F',' '{print $2, $4}'	Pipes the grep results to awk to display only specific columns

Explanation

grep is a command for searching lines matching a condition.
Combining it with awk allows you to efficiently extract only the columns you need.

How to Process CSV Files with awk and grep

Creating the File

cat << 'EOF' > input.txt
id,name,score
1,Alice,85
2,Bob,92
3,Charlie,78
4,David,90
EOF

Command

grep "Bob" input.txt

Output

2,Bob,92

Command

awk -F',' '{print $2, $3}' input.txt

Output

name score
Alice 85
Bob 92
Charlie 78
David 90

Command

awk -F',' 'NR > 1 && $3 >= 90 {print $2, $3}' input.txt

Output

Bob 92
David 90

How It Works

Command	How It Works
grep "Bob" input.txt	Uses grep to search for lines containing "Bob" in the CSV
awk -F',' '{print $2, $3}' input.txt	Uses , as the delimiter in awk to display the 2nd column (name) and 3rd column (score)
awk -F',' 'NR > 1 && $3 >= 90 {print $2, $3}' input.txt	Excludes the header row with NR > 1 and extracts only rows where score is 90 or above

Explanation

grep is convenient for simple string searches.

awk is powerful for conditional branching and column operations, and is widely used for CSV and log analysis.

How to Remove Blank Lines and Duplicate Lines with awk and grep

Creating the File

cat << 'EOF' > input.txt
apple

orange
apple
banana

orange
grape
banana
EOF

Command

grep -v '^$' input.txt | awk '!seen[$0]++'

Output

apple
orange
banana
grape

How It Works

Command	Role	Executable Command
grep -v '^$'	Removes blank lines	grep -v '^$' input.txt
awk '!seen[$0]++'	Removes duplicate lines	awk '!seen[$0]++' input.txt
grep + awk	Removes blank lines and duplicate lines simultaneously	grep -v '^$' input.txt | awk '!seen[$0]++'

Explanation

grep removes blank lines, and the result is passed to awk to eliminate duplicate lines.
By using awk's associative arrays, lines that have already appeared are recorded to prevent them from being output again.

How to Perform Line-Numbered Search with awk and grep

Creating the File

cat << 'EOF' > input.txt
apple
banana
grape
pineapple
apple juice
orange
EOF

Command

grep -n "apple" input.txt

Output

1:apple
4:pineapple
5:apple juice

Command

awk '/apple/{print NR ":" $0}' input.txt

Output

1:apple
4:pineapple
5:apple juice

How It Works

Command	How It Works	Executable Command
grep	Displays the line number of matched lines using the -n option	grep -n "apple" input.txt
awk	Retrieves and displays the current line number using the NR variable	awk '/apple/{print NR ":" $0}' input.txt

Explanation

grep is suitable for simple searches and can easily display line numbers with -n.
awk can also perform flexible conditional branching and formatting in addition to displaying line numbers.

How to Count Occurrences in a File with awk and grep

Creating the File

cat << 'EOF' > input.txt
error
success
error
warning
error
success
EOF

Command

grep -c "error" input.txt

Output

3

Command

awk '/error/ {count++} END {print count}' input.txt

Output

3

How It Works

Command	How It Works	Executable Command
grep	Counts the number of lines matching the specified string	grep -c "error" input.txt
awk	Increments a variable for each matching line and outputs the total count	awk '/error/ {count++} END {print count}' input.txt

Explanation

grep is convenient for quickly checking counts.

awk can flexibly handle complex processing such as conditional branching and aggregation.

How to Search Across Multiple Files with awk and grep

Creating the File

cat << 'EOF' > users.log
2026-05-01 INFO user=alice action=login
2026-05-01 ERROR user=bob action=upload
2026-05-02 INFO user=charlie action=logout
EOF

Creating the File

cat << 'EOF' > system.log
2026-05-01 WARN disk_usage=85%
2026-05-02 ERROR service=db connection_failed
2026-05-03 INFO backup completed
EOF

Command

grep "ERROR" *.log

Output

system.log:2026-05-02 ERROR service=db connection_failed
users.log:2026-05-01 ERROR user=bob action=upload

Command

awk '/ERROR/ {print FILENAME ":" $0}' *.log

Output

system.log:2026-05-02 ERROR service=db connection_failed
users.log:2026-05-01 ERROR user=bob action=upload

Command

awk '/INFO/ {print FILENAME, $2, $3}' *.log

Output

system.log INFO backup
users.log INFO user=alice
users.log INFO user=charlie

How It Works

Command	How It Works	Executable Command
grep	Searches for matching strings across multiple files	grep "ERROR" *.log
awk	Processes and displays lines matching the condition	awk '/ERROR/ {print FILENAME ":" $0}' *.log
awk	Extracts only the filename or specific columns	awk '/INFO/ {print FILENAME, $2, $3}' *.log

Explanation

grep is suitable for simple string searching and can search multiple files at once.
awk can process search results, making it useful for log analysis and column extraction.

How to Recursively Search a Directory with awk and grep

Creating the File

cat << 'EOF' > input.txt
src/app/main.py:ERROR failed login
src/app/api.py:INFO request success
src/lib/util.py:ERROR timeout
logs/app.log:WARN retry
logs/debug.log:ERROR disk full
EOF

Command

grep -r "ERROR" .

Output

./input.txt:src/app/main.py:ERROR failed login
./input.txt:src/lib/util.py:ERROR timeout
./input.txt:logs/debug.log:ERROR disk full

Command

awk -F ':' '/ERROR/ {print $1, $2}' input.txt

Output

src/app/main.py ERROR failed login
src/lib/util.py ERROR timeout
logs/debug.log ERROR disk full

Command

grep -r "ERROR" . | awk -F ':' '{print $1, $2}'

Output

./input.txt src/app/main.py
./input.txt src/lib/util.py
./input.txt logs/debug.log

How It Works

Purpose	Executable Command	How It Works
Recursive search	grep -r "ERROR" .	Recursively searches the directory tree using -r
Conditional extraction	awk -F ':' '/ERROR/ {print $1, $2}' input.txt	Specifies the delimiter with -F ':' and extracts only ERROR lines
grep and awk combined	grep -r "ERROR" . | awk -F ':' '{print $1, $2}'	Passes grep results to awk to display only necessary fields

Explanation

grep can search a directory tree at high speed, and awk excels at column-level processing. Combining them streamlines log analysis and error checking.

Practical Examples of Pipe Processing Combining awk and grep

Creating the File

cat << 'EOF' > input.txt
Alice Sales 120
Bob Marketing 95
Charlie Sales 140
David Engineering 110
Eve Sales 98
Frank Marketing 130
EOF

Command

awk '$2 == "Sales"' input.txt | grep '[0-9][0-9]$'

Output

Alice Sales 120
Charlie Sales 140
Eve Sales 98

How It Works

Processing	Executable Command	Role
Extract Sales department only	awk '$2 == "Sales"' input.txt	Extracts rows where the 2nd column is Sales
Filter by numeric format	grep '[0-9][0-9]$'	Searches for lines ending with two or more digits
Pipe combination	awk '$2 == "Sales"' input.txt | grep '[0-9][0-9]$'	Passes awk results to grep for further filtering

Explanation

awk performs column-based conditional extraction, and grep adds string pattern searching on top.
Using pipes (|) enables flexible text processing by chaining multiple commands.

How to Extract Error Logs with awk and grep

Creating the File

cat << 'EOF' > input.txt
2026-05-13 10:00:01 INFO Application started
2026-05-13 10:01:15 ERROR Database connection failed
2026-05-13 10:02:20 WARN Memory usage high
2026-05-13 10:03:45 ERROR Disk space low
2026-05-13 10:04:50 INFO User login success
EOF

Command

grep "ERROR" input.txt

Output

2026-05-13 10:01:15 ERROR Database connection failed
2026-05-13 10:03:45 ERROR Disk space low

Command

awk '/ERROR/' input.txt

Output

2026-05-13 10:01:15 ERROR Database connection failed
2026-05-13 10:03:45 ERROR Disk space low

Command

awk '/ERROR/ {print $1, $2, $3}' input.txt

Output

2026-05-13 10:01:15 ERROR
2026-05-13 10:03:45 ERROR

How It Works

Command	How It Works	Executable Command
grep	Extracts lines containing the specified string	grep "ERROR" input.txt
awk	Extracts lines matching the pattern	awk '/ERROR/' input.txt
awk print	Displays only specific columns	awk '/ERROR/ {print $1, $2, $3}' input.txt

Explanation

grep is suitable for simple string searching and is commonly used in log analysis.

awk can process data by column, making it useful for detailed log analysis.

How to Aggregate Access Logs with awk and grep

Creating the File

cat << 'EOF' > access.log
192.168.1.10 - - [13/May/2026:10:00:01 +0900] "GET /index.html HTTP/1.1" 200 512
192.168.1.11 - - [13/May/2026:10:00:03 +0900] "POST /login HTTP/1.1" 302 128
192.168.1.10 - - [13/May/2026:10:00:05 +0900] "GET /dashboard HTTP/1.1" 200 1024
192.168.1.12 - - [13/May/2026:10:00:10 +0900] "GET /index.html HTTP/1.1" 404 256
192.168.1.11 - - [13/May/2026:10:00:15 +0900] "GET /dashboard HTTP/1.1" 200 2048
EOF

Command

grep ' 200 ' access.log

Output

192.168.1.10 - - [13/May/2026:10:00:01 +0900] "GET /index.html HTTP/1.1" 200 512
192.168.1.10 - - [13/May/2026:10:00:05 +0900] "GET /dashboard HTTP/1.1" 200 1024
192.168.1.11 - - [13/May/2026:10:00:15 +0900] "GET /dashboard HTTP/1.1" 200 2048

Command

awk '{print $1}' access.log | sort | uniq -c

Output

      2 192.168.1.10
      2 192.168.1.11
      1 192.168.1.12

Command

grep 'GET' access.log | awk '{print $7}' | sort | uniq -c

Output

      2 /dashboard
      2 /index.html

How It Works

Purpose	Executable Command	How It Works
Extract HTTP status 200 only	grep ' 200 ' access.log	Uses grep to search for only successful responses
Count accesses per IP	awk '{print $1}' access.log | sort | uniq -c	Extracts IPs with awk and counts occurrences
Aggregate accessed URLs	grep 'GET' access.log | awk '{print $7}' | sort | uniq -c	Extracts URLs from GET requests and aggregates them

Explanation

Combining awk and grep enables fast and flexible aggregation of access logs.

Using shell pipes allows complex analysis to be achieved with simple commands.

How to Process Large Volumes of Data at High Speed with awk and grep

Creating the File

cat << 'EOF' > input.txt
2026-01-01 INFO login success userA
2026-01-01 ERROR timeout userB
2026-01-01 INFO logout userA
2026-01-01 ERROR disk_full userC
2026-01-01 INFO login success userD
2026-01-01 WARN retry userE
2026-01-01 ERROR timeout userF
EOF

Command

grep "ERROR" input.txt

Output

2026-01-01 ERROR timeout userB
2026-01-01 ERROR disk_full userC
2026-01-01 ERROR timeout userF

Command

awk '{print $2, $3}' input.txt

Output

INFO login
ERROR timeout
INFO logout
ERROR disk_full
INFO login
WARN retry
ERROR timeout

Command

grep "ERROR" input.txt | awk '{print $3}' | sort | uniq -c

Output

      1 disk_full
      2 timeout

Command

awk '$2=="ERROR"{count[$3]++} END{for(i in count) print i, count[i]}' input.txt

Output

disk_full 1
timeout 2

How It Works

How It Works	Executable Command	Description
Extract with grep	grep "ERROR" input.txt	High-speed search for ERROR lines only
Extract columns with awk	awk '{print $2, $3}' input.txt	Retrieves only the necessary columns
grep + awk combination	grep "ERROR" input.txt | awk '{print $3}'	Processes only necessary fields after extraction
awk aggregation	awk '$2=="ERROR"{count[$3]++} END{for(i in count) print i, count[i]}' input.txt	Counts occurrences by condition
sort + uniq aggregation	grep "ERROR" input.txt | awk '{print $3}' | sort | uniq -c	Counts identical data entries

Explanation

Combining awk and grep allows you to quickly extract and aggregate the information you need from large volumes of logs. Structuring your workflow so that grep narrows down the data first and awk then processes it results in higher processing efficiency.

Tips for Making the Most of awk and grep

awk and grep are useful on their own, but combining them allows you to build a powerful text processing environment.

The pattern of using grep to extract only the necessary lines and then processing and aggregating the results with awk is widely used in many development environments.

Once you become proficient with awk and grep, the efficiency of log analysis, server operations, and automation tasks will improve significantly.

It is important to grasp the fundamentals and gradually build familiarity through hands-on file operations.