Introduction
When processing text in Linux and Unix-like environments, awk and grep are indispensable commands.
Understanding the basic syntax and usage of awk and grep allows you to streamline log analysis and CSV processing.
This article explains everything from the basics of awk and grep to practical applications in an easy-to-understand way.
Reference: GNU awk
Reference: GNU grep
Basic Syntax and Usage of awk and grep
Creating the File
cat << 'EOF' > input.txt
apple 100
banana 200
orange 150
apple 300
grape 120
EOFCommand
grep "apple" input.txtOutput
apple 100
apple 300Command
awk '{print $1, $2}' input.txtOutput
apple 100
banana 200
orange 150
apple 300
grape 120Command
awk '$2 >= 150 {print $1, $2}' input.txtOutput
banana 200
orange 150
apple 300Command
grep "apple" input.txt | awk '{sum += $2} END {print sum}'Output
400How It Works
| Command | How It Works | Executable Command |
|---|---|---|
| grep | Searches for lines containing the specified string | grep "apple" input.txt |
| awk | Splits lines by delimiter and processes each column | awk '{print $1, $2}' input.txt |
| awk condition | Extracts only lines matching the specified condition | awk '$2 >= 150 {print $1, $2}' input.txt |
| grep + awk | Aggregates grep results using awk | grep "apple" input.txt | awk '{sum += $2} END {print sum}' |
Explanation
awk excels at column-level processing and aggregation, while grep specializes in string searching.
Combining them enables efficient log analysis and data extraction.
How to Use awk and grep Appropriately
Creating the File
cat << 'EOF' > input.txt
apple 120
banana 80
orange 150
grape 90
apple 200
EOFCommand
grep 'apple' input.txtOutput
apple 120
apple 200Command
awk '$2 >= 100 { print $1, $2 }' input.txtOutput
apple 120
orange 150
apple 200How It Works
| Command | How It Works | Executable Command |
|---|---|---|
| grep | Extracts lines containing the specified string | grep 'apple' input.txt |
| awk | Evaluates conditions by column and processes/extracts data | awk '$2 >= 100 { print $1, $2 }' input.txt |
Explanation
grep specializes in string searching and can be used at high speed for log searches.
awk supports column processing and conditional branching, making it suitable for processing tabular data.
How to Use Regular Expressions with awk and grep
Creating the File
cat << 'EOF' > input.txt
error: disk full
INFO: process started
warning: cpu high
ERROR: memory leak
user login success
EOFCommand
grep -E 'error|warning' input.txtOutput
error: disk full
warning: cpu highCommand
grep -Ei 'error|warning' input.txtOutput
error: disk full
warning: cpu high
ERROR: memory leakCommand
awk '/error|warning/' input.txtOutput
error: disk full
warning: cpu highCommand
awk 'tolower($0) ~ /error|warning/' input.txtOutput
error: disk full
warning: cpu high
ERROR: memory leakHow It Works
| Command | How It Works | Executable Command |
|---|---|---|
| grep -E | Enables extended regular expressions with -E and performs OR search using | | grep -E 'error|warning' input.txt |
| grep -Ei | Searches case-insensitively with -i | grep -Ei 'error|warning' input.txt |
| awk | Extracts matching lines using /regex/ | awk '/error|warning/' input.txt |
| awk tolower | Converts the entire line to lowercase with tolower($0) before comparing | awk 'tolower($0) ~ /error|warning/' input.txt |
Explanation
grep is well-suited for fast string searching and is commonly used for log monitoring and filtering.
awk supports conditional branching and column processing in addition to regular expressions, making it useful for complex text analysis.
How to Perform Exact Match and Partial Match Searches with awk and grep
Creating the File
cat << 'EOF' > input.txt
apple
apple pie
banana
banana split
grape
pineapple
EOFCommand
grep -x "apple" input.txtOutput
appleCommand
grep "apple" input.txtOutput
apple
apple pie
pineappleCommand
awk '$0=="apple"' input.txtOutput
appleCommand
awk '/apple/' input.txtOutput
apple
apple pie
pineappleHow It Works
| Command | Match Type | Executable Command | How It Works |
|---|---|---|---|
| grep | Exact match | grep -x "apple" input.txt | Using -x displays only lines where the entire line matches exactly |
| grep | Partial match | grep "apple" input.txt | Searches for lines containing the specified string |
| awk | Exact match | awk '$0=="apple"' input.txt | $0 represents the entire line; == performs exact match comparison |
| awk | Partial match | awk '/apple/' input.txt | Searches for lines matching the pattern using /string/ |
Explanation
grep can easily switch between exact and partial matching using options.
awk allows flexible searching and advanced text processing through the use of conditional expressions.
How to Search Case-Insensitively with awk and grep
Creating the File
cat << 'EOF' > input.txt
Apple
apple
Banana
BANANA
Cherry
EOFCommand
grep -i "apple" input.txtOutput
Apple
appleCommand
awk 'tolower($0) ~ /banana/' input.txtOutput
Banana
BANANAHow It Works
| Command | How It Works | Executable Command |
|---|---|---|
| grep | Searches case-insensitively using the -i option | grep -i "apple" input.txt |
| awk | Converts the string to lowercase using tolower() before comparing | awk 'tolower($0) ~ /banana/' input.txt |
Explanation
grep can easily ignore case by simply adding -i.
awk enables flexible conditional searching by using tolower().
How to Perform AND Search and OR Search with awk and grep
Creating the File
cat << 'EOF' > input.txt
apple orange banana
apple grape
banana melon
orange lemon
apple orange
EOFCommand
grep 'apple' input.txt | grep 'orange'Output
apple orange banana
apple orangeCommand
awk '/apple/ && /orange/' input.txtOutput
apple orange banana
apple orangeCommand
grep -E 'apple|orange' input.txtOutput
apple orange banana
apple grape
orange lemon
apple orangeCommand
awk '/apple|orange/' input.txtOutput
apple orange banana
apple grape
orange lemon
apple orangeHow It Works
| Search Method | Command | How It Works |
|---|---|---|
| grep AND search | grep 'apple' input.txt | grep 'orange' | Pipes the first grep result to the second grep to extract lines containing both |
| awk AND search | awk '/apple/ && /orange/' input.txt | Extracts lines satisfying multiple conditions simultaneously using && |
| grep OR search | grep -E 'apple|orange' input.txt | Extracts lines matching either condition using | |
| awk OR search | awk '/apple|orange/' input.txt | Extracts lines matching the OR condition in the regular expression |
Explanation
grep achieves AND search using pipes and can concisely write OR search with -E.
awk allows flexible combinations of conditions using && and |.
How to Specify Multiple Conditions with awk and grep
Creating the File
cat << 'EOF' > input.txt
Alice Tokyo Sales
Bob Osaka Engineering
Carol Tokyo Engineering
Dave Nagoya Sales
Eve Tokyo Marketing
EOFCommand
grep 'Tokyo' input.txt | grep 'Engineering'Output
Carol Tokyo EngineeringCommand
awk '/Tokyo/ && /Engineering/' input.txtOutput
Carol Tokyo EngineeringHow It Works
| Command | How It Works |
|---|---|
| grep 'Tokyo' input.txt | grep 'Engineering' | Chains grep with a pipe to extract only lines containing both "Tokyo" and "Engineering" |
| awk '/Tokyo/ && /Engineering/' input.txt | Uses && in awk to display only lines matching both conditions |
Explanation
In awk, the logical operator && allows flexible combinations of multiple conditions.
Adding conditions or including numeric comparisons is also easy to extend.
How to Search for Multiple Keywords with awk and grep
Creating the File
cat << 'EOF' > input.txt
Linux awk command
grep is useful
sed and awk together
find with grep
python script
EOFCommand
grep -E 'awk|grep' input.txtOutput
Linux awk command
grep is useful
sed and awk together
find with grepCommand
awk '/awk|grep/' input.txtOutput
Linux awk command
grep is useful
sed and awk together
find with grepHow It Works
| Command | How It Works | Executable Command |
|---|---|---|
| grep | Enables extended regular expressions with -E and performs OR search for multiple keywords using | | grep -E 'awk|grep' input.txt |
| awk | Searches for multiple keywords using OR conditions within /condition/ | awk '/awk|grep/' input.txt |
Explanation
grep is commonly used for high-speed multi-keyword searching.
awk can also perform column processing and conditional branching at the same time as searching.
How to Extract Specific Columns with awk and grep
Creating the File
cat << 'EOF' > input.txt
id,name,department,salary
1,Alice,Sales,5000
2,Bob,Engineering,7000
3,Carol,Marketing,6000
4,David,Engineering,8000
EOFCommand
awk -F',' '{print $2, $4}' input.txtOutput
name salary
Alice 5000
Bob 7000
Carol 6000
David 8000Command
grep 'Engineering' input.txtOutput
2,Bob,Engineering,7000
4,David,Engineering,8000Command
grep 'Engineering' input.txt | awk -F',' '{print $2, $4}'Output
Bob 7000
David 8000How It Works
| Command | How It Works |
|---|---|
| awk -F',' '{print $2, $4}' input.txt | Sets the delimiter to , with -F',' and extracts the 2nd and 4th columns using $2 and $4 |
| grep 'Engineering' input.txt | Extracts only lines containing Engineering |
| grep 'Engineering' input.txt | awk -F',' '{print $2, $4}' | Pipes the grep results to awk to display only specific columns |
Explanation
grep is a command for searching lines matching a condition.
Combining it with awk allows you to efficiently extract only the columns you need.
How to Process CSV Files with awk and grep
Creating the File
cat << 'EOF' > input.txt
id,name,score
1,Alice,85
2,Bob,92
3,Charlie,78
4,David,90
EOFCommand
grep "Bob" input.txtOutput
2,Bob,92Command
awk -F',' '{print $2, $3}' input.txtOutput
name score
Alice 85
Bob 92
Charlie 78
David 90Command
awk -F',' 'NR > 1 && $3 >= 90 {print $2, $3}' input.txtOutput
Bob 92
David 90How It Works
| Command | How It Works |
|---|---|
| grep "Bob" input.txt | Uses grep to search for lines containing "Bob" in the CSV |
| awk -F',' '{print $2, $3}' input.txt | Uses , as the delimiter in awk to display the 2nd column (name) and 3rd column (score) |
| awk -F',' 'NR > 1 && $3 >= 90 {print $2, $3}' input.txt | Excludes the header row with NR > 1 and extracts only rows where score is 90 or above |
Explanation
grep is convenient for simple string searches.
awk is powerful for conditional branching and column operations, and is widely used for CSV and log analysis.
How to Remove Blank Lines and Duplicate Lines with awk and grep
Creating the File
cat << 'EOF' > input.txt
apple
orange
apple
banana
orange
grape
banana
EOFCommand
grep -v '^$' input.txt | awk '!seen[$0]++'Output
apple
orange
banana
grapeHow It Works
| Command | Role | Executable Command |
|---|---|---|
| grep -v '^$' | Removes blank lines | grep -v '^$' input.txt |
| awk '!seen[$0]++' | Removes duplicate lines | awk '!seen[$0]++' input.txt |
| grep + awk | Removes blank lines and duplicate lines simultaneously | grep -v '^$' input.txt | awk '!seen[$0]++' |
Explanation
grep removes blank lines, and the result is passed to awk to eliminate duplicate lines.
By using awk's associative arrays, lines that have already appeared are recorded to prevent them from being output again.
How to Perform Line-Numbered Search with awk and grep
Creating the File
cat << 'EOF' > input.txt
apple
banana
grape
pineapple
apple juice
orange
EOFCommand
grep -n "apple" input.txtOutput
1:apple
4:pineapple
5:apple juiceCommand
awk '/apple/{print NR ":" $0}' input.txtOutput
1:apple
4:pineapple
5:apple juiceHow It Works
| Command | How It Works | Executable Command |
|---|---|---|
| grep | Displays the line number of matched lines using the -n option | grep -n "apple" input.txt |
| awk | Retrieves and displays the current line number using the NR variable | awk '/apple/{print NR ":" $0}' input.txt |
Explanation
grep is suitable for simple searches and can easily display line numbers with -n.
awk can also perform flexible conditional branching and formatting in addition to displaying line numbers.
How to Count Occurrences in a File with awk and grep
Creating the File
cat << 'EOF' > input.txt
error
success
error
warning
error
success
EOFCommand
grep -c "error" input.txtOutput
3Command
awk '/error/ {count++} END {print count}' input.txtOutput
3How It Works
| Command | How It Works | Executable Command |
|---|---|---|
| grep | Counts the number of lines matching the specified string | grep -c "error" input.txt |
| awk | Increments a variable for each matching line and outputs the total count | awk '/error/ {count++} END {print count}' input.txt |
Explanation
grep is convenient for quickly checking counts.
awk can flexibly handle complex processing such as conditional branching and aggregation.
How to Search Across Multiple Files with awk and grep
Creating the File
cat << 'EOF' > users.log
2026-05-01 INFO user=alice action=login
2026-05-01 ERROR user=bob action=upload
2026-05-02 INFO user=charlie action=logout
EOFCreating the File
cat << 'EOF' > system.log
2026-05-01 WARN disk_usage=85%
2026-05-02 ERROR service=db connection_failed
2026-05-03 INFO backup completed
EOFCommand
grep "ERROR" *.logOutput
system.log:2026-05-02 ERROR service=db connection_failed
users.log:2026-05-01 ERROR user=bob action=uploadCommand
awk '/ERROR/ {print FILENAME ":" $0}' *.logOutput
system.log:2026-05-02 ERROR service=db connection_failed
users.log:2026-05-01 ERROR user=bob action=uploadCommand
awk '/INFO/ {print FILENAME, $2, $3}' *.logOutput
system.log INFO backup
users.log INFO user=alice
users.log INFO user=charlieHow It Works
| Command | How It Works | Executable Command |
|---|---|---|
| grep | Searches for matching strings across multiple files | grep "ERROR" *.log |
| awk | Processes and displays lines matching the condition | awk '/ERROR/ {print FILENAME ":" $0}' *.log |
| awk | Extracts only the filename or specific columns | awk '/INFO/ {print FILENAME, $2, $3}' *.log |
Explanation
grep is suitable for simple string searching and can search multiple files at once.
awk can process search results, making it useful for log analysis and column extraction.
How to Recursively Search a Directory with awk and grep
Creating the File
cat << 'EOF' > input.txt
src/app/main.py:ERROR failed login
src/app/api.py:INFO request success
src/lib/util.py:ERROR timeout
logs/app.log:WARN retry
logs/debug.log:ERROR disk full
EOFCommand
grep -r "ERROR" .Output
./input.txt:src/app/main.py:ERROR failed login
./input.txt:src/lib/util.py:ERROR timeout
./input.txt:logs/debug.log:ERROR disk fullCommand
awk -F ':' '/ERROR/ {print $1, $2}' input.txtOutput
src/app/main.py ERROR failed login
src/lib/util.py ERROR timeout
logs/debug.log ERROR disk fullCommand
grep -r "ERROR" . | awk -F ':' '{print $1, $2}'Output
./input.txt src/app/main.py
./input.txt src/lib/util.py
./input.txt logs/debug.logHow It Works
| Purpose | Executable Command | How It Works |
|---|---|---|
| Recursive search | grep -r "ERROR" . | Recursively searches the directory tree using -r |
| Conditional extraction | awk -F ':' '/ERROR/ {print $1, $2}' input.txt | Specifies the delimiter with -F ':' and extracts only ERROR lines |
| grep and awk combined | grep -r "ERROR" . | awk -F ':' '{print $1, $2}' | Passes grep results to awk to display only necessary fields |
Explanation
grep can search a directory tree at high speed, and awk excels at column-level processing. Combining them streamlines log analysis and error checking.
Practical Examples of Pipe Processing Combining awk and grep
Creating the File
cat << 'EOF' > input.txt
Alice Sales 120
Bob Marketing 95
Charlie Sales 140
David Engineering 110
Eve Sales 98
Frank Marketing 130
EOFCommand
awk '$2 == "Sales"' input.txt | grep '[0-9][0-9]$'Output
Alice Sales 120
Charlie Sales 140
Eve Sales 98How It Works
| Processing | Executable Command | Role |
|---|---|---|
| Extract Sales department only | awk '$2 == "Sales"' input.txt | Extracts rows where the 2nd column is Sales |
| Filter by numeric format | grep '[0-9][0-9]$' | Searches for lines ending with two or more digits |
| Pipe combination | awk '$2 == "Sales"' input.txt | grep '[0-9][0-9]$' | Passes awk results to grep for further filtering |
Explanation
awk performs column-based conditional extraction, and grep adds string pattern searching on top.
Using pipes (|) enables flexible text processing by chaining multiple commands.
How to Extract Error Logs with awk and grep
Creating the File
cat << 'EOF' > input.txt
2026-05-13 10:00:01 INFO Application started
2026-05-13 10:01:15 ERROR Database connection failed
2026-05-13 10:02:20 WARN Memory usage high
2026-05-13 10:03:45 ERROR Disk space low
2026-05-13 10:04:50 INFO User login success
EOFCommand
grep "ERROR" input.txtOutput
2026-05-13 10:01:15 ERROR Database connection failed
2026-05-13 10:03:45 ERROR Disk space lowCommand
awk '/ERROR/' input.txtOutput
2026-05-13 10:01:15 ERROR Database connection failed
2026-05-13 10:03:45 ERROR Disk space lowCommand
awk '/ERROR/ {print $1, $2, $3}' input.txtOutput
2026-05-13 10:01:15 ERROR
2026-05-13 10:03:45 ERRORHow It Works
| Command | How It Works | Executable Command |
|---|---|---|
| grep | Extracts lines containing the specified string | grep "ERROR" input.txt |
| awk | Extracts lines matching the pattern | awk '/ERROR/' input.txt |
| awk print | Displays only specific columns | awk '/ERROR/ {print $1, $2, $3}' input.txt |
Explanation
grep is suitable for simple string searching and is commonly used in log analysis.
awk can process data by column, making it useful for detailed log analysis.
How to Aggregate Access Logs with awk and grep
Creating the File
cat << 'EOF' > access.log
192.168.1.10 - - [13/May/2026:10:00:01 +0900] "GET /index.html HTTP/1.1" 200 512
192.168.1.11 - - [13/May/2026:10:00:03 +0900] "POST /login HTTP/1.1" 302 128
192.168.1.10 - - [13/May/2026:10:00:05 +0900] "GET /dashboard HTTP/1.1" 200 1024
192.168.1.12 - - [13/May/2026:10:00:10 +0900] "GET /index.html HTTP/1.1" 404 256
192.168.1.11 - - [13/May/2026:10:00:15 +0900] "GET /dashboard HTTP/1.1" 200 2048
EOFCommand
grep ' 200 ' access.logOutput
192.168.1.10 - - [13/May/2026:10:00:01 +0900] "GET /index.html HTTP/1.1" 200 512
192.168.1.10 - - [13/May/2026:10:00:05 +0900] "GET /dashboard HTTP/1.1" 200 1024
192.168.1.11 - - [13/May/2026:10:00:15 +0900] "GET /dashboard HTTP/1.1" 200 2048Command
awk '{print $1}' access.log | sort | uniq -cOutput
2 192.168.1.10
2 192.168.1.11
1 192.168.1.12Command
grep 'GET' access.log | awk '{print $7}' | sort | uniq -cOutput
2 /dashboard
2 /index.htmlHow It Works
| Purpose | Executable Command | How It Works |
|---|---|---|
| Extract HTTP status 200 only | grep ' 200 ' access.log | Uses grep to search for only successful responses |
| Count accesses per IP | awk '{print $1}' access.log | sort | uniq -c | Extracts IPs with awk and counts occurrences |
| Aggregate accessed URLs | grep 'GET' access.log | awk '{print $7}' | sort | uniq -c | Extracts URLs from GET requests and aggregates them |
Explanation
Combining awk and grep enables fast and flexible aggregation of access logs.
Using shell pipes allows complex analysis to be achieved with simple commands.
How to Process Large Volumes of Data at High Speed with awk and grep
Creating the File
cat << 'EOF' > input.txt
2026-01-01 INFO login success userA
2026-01-01 ERROR timeout userB
2026-01-01 INFO logout userA
2026-01-01 ERROR disk_full userC
2026-01-01 INFO login success userD
2026-01-01 WARN retry userE
2026-01-01 ERROR timeout userF
EOFCommand
grep "ERROR" input.txtOutput
2026-01-01 ERROR timeout userB
2026-01-01 ERROR disk_full userC
2026-01-01 ERROR timeout userFCommand
awk '{print $2, $3}' input.txtOutput
INFO login
ERROR timeout
INFO logout
ERROR disk_full
INFO login
WARN retry
ERROR timeoutCommand
grep "ERROR" input.txt | awk '{print $3}' | sort | uniq -cOutput
1 disk_full
2 timeoutCommand
awk '$2=="ERROR"{count[$3]++} END{for(i in count) print i, count[i]}' input.txtOutput
disk_full 1
timeout 2How It Works
| How It Works | Executable Command | Description |
|---|---|---|
| Extract with grep | grep "ERROR" input.txt | High-speed search for ERROR lines only |
| Extract columns with awk | awk '{print $2, $3}' input.txt | Retrieves only the necessary columns |
| grep + awk combination | grep "ERROR" input.txt | awk '{print $3}' | Processes only necessary fields after extraction |
| awk aggregation | awk '$2=="ERROR"{count[$3]++} END{for(i in count) print i, count[i]}' input.txt | Counts occurrences by condition |
| sort + uniq aggregation | grep "ERROR" input.txt | awk '{print $3}' | sort | uniq -c | Counts identical data entries |
Explanation
Combining awk and grep allows you to quickly extract and aggregate the information you need from large volumes of logs. Structuring your workflow so that grep narrows down the data first and awk then processes it results in higher processing efficiency.
Tips for Making the Most of awk and grep
awk and grep are useful on their own, but combining them allows you to build a powerful text processing environment.
The pattern of using grep to extract only the necessary lines and then processing and aggregating the results with awk is widely used in many development environments.
Once you become proficient with awk and grep, the efficiency of log analysis, server operations, and automation tasks will improve significantly.
It is important to grasp the fundamentals and gradually build familiarity through hands-on file operations.
