Regular Expressions (RegEx)

Intermediate ~30 min read

Regular expressions are a powerful language for pattern matching in text. With Python's re module, you can search for patterns, validate input, extract data, and transform text. From validating email addresses to parsing log files - regex is an essential skill for text processing, data cleaning, and form validation. Once you master regex, you'll wonder how you ever lived without it!

Basic RegEx Functions

Python's re module provides several functions for pattern matching: search() finds the first match, match() checks only at the start, findall() returns all matches, split() divides text at matches, and sub() replaces matches. These are your bread-and-butter regex operations.

# Basic RegEx Functions

import re

text = "The quick brown fox jumps over the lazy dog"

# 1. search() - Find first match anywhere in string
print("=== search() ===")
match = re.search(r"fox", text)
if match:
    print(f"Found: '{match.group()}' at position {match.start()}-{match.end()}")
else:
    print("Not found")
print()

# 2. match() - Match only at the beginning
print("=== match() ===")
result = re.match(r"The", text)
print(f"'The' at start: {result is not None}")

result = re.match(r"fox", text)
print(f"'fox' at start: {result is not None}")  # False!
print()

# 3. findall() - Find all matches, return list
print("=== findall() ===")
vowels = re.findall(r"[aeiou]", text)
print(f"All vowels: {vowels}")
print(f"Count: {len(vowels)}")

words = re.findall(r"\b\w{4}\b", text)  # 4-letter words
print(f"4-letter words: {words}")
print()

# 4. split() - Split string at pattern matches
print("=== split() ===")
parts = re.split(r"\s+", text)  # Split on whitespace
print(f"Words: {parts}")

parts = re.split(r"\s+", text, maxsplit=3)
print(f"First 3 splits: {parts}")
print()

# 5. sub() - Replace matches
print("=== sub() ===")
result = re.sub(r"fox", "cat", text)
print(f"Replace 'fox': {result}")

result = re.sub(r"\s+", "_", text)
print(f"Spaces to underscore: {result}")

Output

Click Run to execute your code

Raw Strings (r"..."): Always use raw strings for regex patterns by prefixing with r. This prevents Python from interpreting backslashes. Without it, "\d" might be misinterpreted, but r"\d" is always safe. Get in the habit of using r"pattern" for all regex.

Pattern Syntax and Metacharacters

Regex patterns use special characters (metacharacters) to match types of characters and define repetition. \d matches digits, \w matches word characters, + means "one or more", and * means "zero or more". These building blocks combine to create powerful patterns.

# RegEx Patterns and Metacharacters

import re

# 1. Character classes
print("=== Character Classes ===")
text = "Hello World 123"
print(f"Text: '{text}'")
# Extract patterns to variables (f-strings can't have backslashes in expressions)
digits = re.findall(r'\d', text)
non_digits = re.findall(r'\D', text)
word_chars = re.findall(r'\w', text)
non_word = re.findall(r'\W', text)
whitespace = re.findall(r'\s', text)
print(f"\\d (digits): {digits}")
print(f"\\D (non-digits): {non_digits}")
print(f"\\w (word chars): {word_chars}")
print(f"\\W (non-word): {non_word}")
print(f"\\s (whitespace): {whitespace}")
print()

# 2. Custom character sets
print("=== Character Sets [...] ===")
text = "The cat sat on the mat"
print(f"[cm]at: {re.findall(r'[cm]at', text)}")  # cat or mat
print(f"[a-z]: {re.findall(r'[a-z]', 'A1b2C3')}")  # lowercase
print(f"[^aeiou]: {re.findall(r'[^aeiou ]', text)}")  # not vowels
print()

# 3. Quantifiers
print("=== Quantifiers ===")
text = "goood morning, gd"
print(f"go+d: {re.findall(r'go+d', text)}")   # 1 or more o
print(f"go*d: {re.findall(r'go*d', text)}")   # 0 or more o
print(f"go?d: {re.findall(r'go?d', text)}")   # 0 or 1 o

text = "12 345 6789"
two_to_three_digits = re.findall(r'\d{2,3}', text)
print(f"\\d{{2,3}}: {two_to_three_digits}")  # 2-3 digits
print()

# 4. Anchors
print("=== Anchors ===")
text = "hello world hello"
start_hello = re.findall(r'^hello', text)
end_hello = re.findall(r'hello$', text)
word_boundary_hello = re.findall(r'\bhello\b', text)
print(f"^hello: {start_hello}")  # Start
print(f"hello$: {end_hello}")  # End
print(f"\\bhello\\b: {word_boundary_hello}")  # Word boundary
print()

# 5. Alternation and grouping
print("=== Alternation (|) ===")
text = "I have a cat and a dog"
print(f"cat|dog: {re.findall(r'cat|dog', text)}")
print(f"(cat|dog)s?: {re.findall(r'(cat|dog)s?', 'cats and dogs')}")

Output

Click Run to execute your code

Essential Metacharacters:
\d digit, \D non-digit
\w word char [a-zA-Z0-9_], \W non-word
\s whitespace, \S non-whitespace
. any char (except newline)
^ start, $ end
\b word boundary

Groups and Capturing

Parentheses create groups that capture matched text. You can extract parts of a match, use named groups for clarity, and reference captured groups in replacements. Groups are essential for extracting structured data from text - like pulling apart names, dates, or URLs.

# RegEx Groups and Capturing

import re

# 1. Basic groups with parentheses
print("=== Basic Groups ===")
text = "John Smith, Jane Doe"
matches = re.findall(r"(\w+) (\w+)", text)
print(f"Name pairs: {matches}")

# With finditer for more details
for match in re.finditer(r"(\w+) (\w+)", text):
    print(f"Full: {match.group(0)}, First: {match.group(1)}, Last: {match.group(2)}")
print()

# 2. Named groups
print("=== Named Groups ===")
pattern = r"(?P<first>\w+) (?P<last>\w+)"
match = re.search(pattern, "John Smith")
if match:
 print(f"First name: {match.group('first')}")
 print(f"Last name: {match.group('last')}")
 print(f"As dict: {match.groupdict()}")
print()

# 3. Non-capturing groups
print("=== Non-Capturing Groups (?:...) ===")
text = "I like cats and dogs"
# Without capturing
print(f"(?:cat|dog)s: {re.findall(r'(?:cat|dog)s', text)}")
# With capturing (returns tuple)
print(f"(cat|dog)s: {re.findall(r'(cat|dog)s', text)}")
print()

# 4. Backreferences
print("=== Backreferences ===")
# Find repeated words
text = "the the quick fox"
match = re.search(r"\b(\w+)\s+\1\b", text)
if match:
    print(f"Repeated word: '{match.group(1)}'")

# HTML tag matching
html = "bold and italic"
tags = re.findall(r"<(\w+)>.*?</\1>", html)
print(f"Matching tags: {tags}")
print()

# 5. Using groups in substitution
print("=== Groups in sub() ===")
text = "John Smith"
# Swap first and last name
result = re.sub(r"(\w+) (\w+)", r"\2, \1", text)
print(f"Swapped: {result}")

# Format phone numbers
phone = "5551234567"
formatted = re.sub(r"(\d{3})(\d{3})(\d{4})", r"(\1) \2-\3", phone)
print(f"Phone: {formatted}")

Output

Click Run to execute your code

Greedy vs Non-Greedy: By default, quantifiers like * and + are greedy - they match as much as possible. Add ? to make them non-greedy (match minimum). For example, .* in .* would match too much if you have multiple tags. Use .*? for non-greedy matching.

Practical Validation Examples

Regex shines for validation tasks: checking email formats, extracting phone numbers, validating passwords, cleaning messy text, and parsing structured data. These real-world patterns demonstrate how regex solves common programming challenges.

# Practical RegEx Examples

import re

# 1. Email validation
print("=== Email Validation ===")
email_pattern = r"^[\w\.-]+@[\w\.-]+\.\w+$"

emails = ["user@example.com", "invalid@", "test.user@domain.co.uk", "@bad.com"]
for email in emails:
    valid = bool(re.match(email_pattern, email))
    print(f"{email}: {'Valid' if valid else 'Invalid'}")
print()

# 2. Phone number extraction
print("=== Phone Numbers ===")
text = "Call me at 555-123-4567 or (555) 987-6543"
pattern = r"$?\d{3}$?[-.\s]?\d{3}[-.\s]?\d{4}"
phones = re.findall(pattern, text)
print(f"Found: {phones}")
print()

# 3. URL extraction
print("=== URLs ===")
text = "Visit https://example.com or http://test.org/page"
pattern = r"https?://[\w\.-]+(?:/[\w\.-]*)*"
urls = re.findall(pattern, text)
print(f"URLs: {urls}")
print()

# 4. Password validation
print("=== Password Validation ===")
def validate_password(pwd):
    checks = {
        "8+ chars": len(pwd) >= 8,
        "uppercase": bool(re.search(r"[A-Z]", pwd)),
        "lowercase": bool(re.search(r"[a-z]", pwd)),
        "digit": bool(re.search(r"\d", pwd)),
        "special": bool(re.search(r"[!@#$%^&*]", pwd))
    }
    return checks

password = "MyP@ss123"
results = validate_password(password)
print(f"Password: {password}")
for check, passed in results.items():
    print(f"  {check}: {'Pass' if passed else 'Fail'}")
print()

# 5. Text cleanup
print("=== Text Cleanup ===")
messy_text = "Hello    World!   How    are   you?"
# Multiple spaces to single
clean = re.sub(r"\s+", " ", messy_text)
print(f"Single spaces: {clean}")

# Remove HTML tags
html = "Hello World!"
text_only = re.sub(r"<[^>]+>", "", html)
print(f"No HTML: {text_only}")

Output

Click Run to execute your code

Common Mistakes

1. Forgetting raw strings

import re

# Wrong - backslash gets interpreted by Python
pattern = "\d+"  # Python sees this differently!

# Correct - use raw string
pattern = r"\d+"

# Even worse with \b (word boundary)
re.findall("\bword\b", text)  # \b is backspace in Python!
re.findall(r"\bword\b", text)  # Correct!

2. Confusing search() and match()

import re

text = "hello world"

# match() only checks the START of string
re.match(r"world", text)  # None - "world" not at start!

# search() finds anywhere in string
re.search(r"world", text)  # Match found!

# To match entire string, use anchors
re.match(r".*world$", text)  # Works
re.fullmatch(r"hello world", text)  # Better!

3. Greedy matching grabs too much

import re

html = "bold and more"

# Wrong - greedy .* matches everything between first < and last >
re.findall(r".*", html)
# Returns: ['bold and more']

# Correct - non-greedy .*? matches minimum
re.findall(r".*?", html)
# Returns: ['bold', 'more']

4. Not escaping special characters

import re

# Wrong - . matches ANY character
re.findall(r"3.14", "3.14 and 3x14")
# Returns: ['3.14', '3x14']

# Correct - escape the dot to match literal .
re.findall(r"3\.14", "3.14 and 3x14")
# Returns: ['3.14']

# Other chars to escape: . * + ? ^ $ [ ] { } | ( ) \
# Use re.escape() for user input
user_input = "price: $5.00"
pattern = re.escape(user_input)  # "price:\\ \$5\\.00"

5. Capturing when you don't need to

import re

# With capturing group - returns tuples
re.findall(r"(cat|dog)s?", "cats and dogs")
# Returns: ['cat', 'dog']  # Just the group content!

# Without capturing (non-capturing group)
re.findall(r"(?:cat|dog)s?", "cats and dogs")
# Returns: ['cats', 'dogs']  # Full matches!

# Use (?:...) when you need grouping but not capturing

Exercise: Log Parser

Task: Parse log entries to extract timestamp, level, and message.

Requirements:

Parse log format: "[TIMESTAMP] LEVEL: message"
Extract the timestamp (in brackets)
Extract the log level (INFO, ERROR, WARNING)
Extract the message after the colon

Output

Click Run to execute your code

Show Solution

import re

def parse_log(log_line):
    """Parse a log line and return components."""
    pattern = r"\[(.+?)\]\s+(INFO|ERROR|WARNING):\s+(.+)"
    match = re.match(pattern, log_line)
    if match:
        return {
            "timestamp": match.group(1),
            "level": match.group(2),
            "message": match.group(3)
        }
    return None

# Test logs
logs = [
    "[2024-03-15 10:30:45] INFO: Server started successfully",
    "[2024-03-15 10:31:02] ERROR: Connection refused",
    "[2024-03-15 10:32:15] WARNING: High memory usage detected"
]

for log in logs:
    result = parse_log(log)
    if result:
        print(f"Time: {result['timestamp']}")
        print(f"Level: {result['level']}")
        print(f"Message: {result['message']}")
        print()

Summary

Functions: search(), match(), findall(), split(), sub()
Always use: Raw strings r"pattern" for regex patterns
Character classes: \d (digit), \w (word), \s (space), . (any)
Quantifiers: * (0+), + (1+), ? (0 or 1), {n,m} (range)
Anchors: ^ (start), $ (end), \b (word boundary)
Groups: (...) capture, (?:...) non-capture, (?P<name>...) named
Non-greedy: Add ? after quantifier: .*?, +?
Escape: re.escape() for user input with special chars

What's Next?

Congratulations on completing the Modules & Packages module! You now know how to work with dates, math, JSON, and regular expressions. Next, let's learn about PIP - Python's package manager that lets you install thousands of third-party packages to extend Python's capabilities!

Previous JSON

Next PIP

Basic RegEx Functions

Pattern Syntax and Metacharacters

Groups and Capturing

Practical Validation Examples

Common Mistakes

1. Forgetting raw strings

2. Confusing search() and match()

3. Greedy matching grabs too much

4. Not escaping special characters

5. Capturing when you don't need to

Exercise: Log Parser

Summary

What's Next?

Enjoying these tutorials?