How to Flatten and Remove Keys from a Nested Dictionary in Python

When working with deeply nested dictionaries, common in JSON API responses, configuration files, and NoSQL databases, you often need to flatten the structure into a single-level dictionary with compound keys. This makes the data easier to search, filter, export to tabular formats, or store in flat key-value stores.

For example:

• Input: {'a': {'b': 1, 'c': {'d': 2}}, 'e': 3}
• Output: {'a.b': 1, 'a.c.d': 2, 'e': 3}

Each nested key is joined with its parent using a dot separator (.), producing a flat dictionary with no nesting.

This guide covers multiple approaches to flatten nested dictionaries in Python.

Using Recursion (Recommended)

The most intuitive approach uses a recursive function that traverses the nested structure and builds compound keys as it descends:

def flatten_dict(d, parent_key="", separator="."):
    """Recursively flatten a nested dictionary."""
    items = {}
    for key, value in d.items():
        new_key = f"{parent_key}{separator}{key}" if parent_key else key
        if isinstance(value, dict):
            items.update(flatten_dict(value, new_key, separator))
        else:
            items[new_key] = value
    return items

nested = {'a': {'b': 1, 'c': {'d': 2}}, 'e': 3}
flat = flatten_dict(nested)
print(flat)

Output:

{'a.b': 1, 'a.c.d': 2, 'e': 3}

How It Works

1. For each key-value pair, a compound key is built by joining the parent key with the current key using the separator.
2. If the value is a dict, the function recurses deeper.
3. If the value is a non-dict (leaf node), it's added to the result with its full compound key.

Customizable separator

You can change the separator to match your needs:

print(flatten_dict(nested, separator="/"))
# Output: {'a/b': 1, 'a/c/d': 2, 'e': 3}

print(flatten_dict(nested, separator="_"))
# Output: {'a_b': 1, 'a_c_d': 2, 'e': 3}

Using a Stack (Iterative Depth-First)

For very deeply nested dictionaries where recursion might hit Python's recursion limit, an iterative approach using a stack performs depth-first traversal without recursive calls:

def flatten_dict(d, separator="."):
    """Flatten a nested dictionary using an iterative stack."""
    flat = {}
    stack = [(d, "")]

    while stack:
        current, parent_key = stack.pop()
        for key, value in current.items():
            new_key = f"{parent_key}{separator}{key}" if parent_key else key
            if isinstance(value, dict):
                stack.append((value, new_key))
            else:
                flat[new_key] = value

    return flat

nested = {'a': {'b': 1, 'c': {'d': 2}}, 'e': 3}
print(flatten_dict(nested))

Output:

{'e': 3, 'a.b': 1, 'a.c.d': 2}

note

The output key order may differ from the recursive approach because a stack processes items in LIFO (last-in, first-out) order. The data itself is identical - only the iteration order changes. If order matters, you can use dict(sorted(flat.items())).

Using a Queue (Iterative Breadth-First)

A queue-based approach processes the dictionary level by level (breadth-first), which can be useful when you want to process shallower keys before deeper ones:

from collections import deque

def flatten_dict(d, separator="."):
    """Flatten a nested dictionary using breadth-first traversal."""
    flat = {}
    queue = deque([(d, "")])

    while queue:
        current, parent_key = queue.popleft()
        for key, value in current.items():
            new_key = f"{parent_key}{separator}{key}" if parent_key else key
            if isinstance(value, dict):
                queue.append((value, new_key))
            else:
                flat[new_key] = value

    return flat

nested = {'a': {'b': 1, 'c': {'d': 2}}, 'e': 3}
print(flatten_dict(nested))

Output:

{'e': 3, 'a.b': 1, 'a.c.d': 2}

The queue processes top-level keys ('a', 'e') before diving into nested levels ('a.b', 'a.c', then 'a.c.d').

Flattening with Specific Key Removal

Sometimes you need to flatten the dictionary and remove specific keys or values during the process. Here's how to combine flattening with filtering:

def flatten_and_remove(d, remove_keys=None, separator="."):
    """Flatten a nested dictionary and remove specified keys."""
    if remove_keys is None:
        remove_keys = set()

    flat = {}
    stack = [(d, "")]

    while stack:
        current, parent_key = stack.pop()
        for key, value in current.items():
            if key in remove_keys:
                continue  # Skip this key entirely
            new_key = f"{parent_key}{separator}{key}" if parent_key else key
            if isinstance(value, dict):
                stack.append((value, new_key))
            else:
                flat[new_key] = value

    return flat

nested = {
    'user': {
        'name': 'Alice',
        'password': 'secret123',  # Sensitive (remove)
        'profile': {
            'age': 30,
            'email': 'alice@example.com',
            'internal_id': 'xyz'  # Internal (remove)
        }
    },
    'status': 'active'
}

result = flatten_and_remove(nested, remove_keys={'password', 'internal_id'})
for key, value in sorted(result.items()):
    print(f"{key}: {value}")

Output:

status: active
user.name: Alice
user.profile.age: 30
user.profile.email: alice@example.com

The password and internal_id keys are excluded from the flattened result.

Handling Lists Inside Nested Dictionaries

Real-world JSON data often contains lists alongside nested dicts. Here's how to handle them:

def flatten_dict(d, parent_key="", separator="."):
    """Flatten a nested dictionary, including lists."""
    flat = {}
    
    if isinstance(d, dict):
        for key, value in d.items():
            new_key = f"{parent_key}{separator}{key}" if parent_key else key
            flat.update(flatten_dict(value, new_key, separator))
    elif isinstance(d, list):
        for index, item in enumerate(d):
            new_key = f"{parent_key}{separator}{index}" if parent_key else str(index)
            flat.update(flatten_dict(item, new_key, separator))
    else:
        flat[parent_key] = d
    
    return flat

nested = {
    'user': 'Alice',
    'scores': [95, 87, 92],
    'address': {
        'city': 'NYC',
        'phones': ['555-0100', '555-0200']
    }
}

result = flatten_dict(nested)
for key, value in sorted(result.items()):
    print(f"{key}: {value}")

Output:

address.city: NYC
address.phones.0: 555-0100
address.phones.1: 555-0200
scores.0: 95
scores.1: 87
scores.2: 92
user: Alice

Empty dicts and lists are dropped

With all the approaches shown above, empty dictionaries ({}) and empty lists ([]) produce no keys in the output and are silently dropped. If you need to preserve them, add explicit handling:

if isinstance(d, dict):
    if not d:
        flat[parent_key] = {}  # Preserve empty dict
    for key, value in d.items():
        # ... rest of logic

Unflattening: Converting Back to Nested

If you need to reverse the flattening and reconstruct the nested structure:

def unflatten_dict(flat, separator="."):
    """Convert a flat dictionary back to a nested dictionary."""
    nested = {}
    for compound_key, value in flat.items():
        keys = compound_key.split(separator)
        current = nested
        for key in keys[:-1]:
            current = current.setdefault(key, {})
        current[keys[-1]] = value
    return nested

flat = {'a.b': 1, 'a.c.d': 2, 'e': 3}
nested = unflatten_dict(flat)
print(nested)

Output:

{'a': {'b': 1, 'c': {'d': 2}}, 'e': 3}

Comparison of Approaches

Method	Traversal Order	Handles Deep Nesting	Best For
Recursion	Depth-first	⚠️ Limited by recursion depth	Most use cases (recommended)
Stack (iterative)	Depth-first	✅ No recursion limit	Very deeply nested structures
Queue (iterative)	Breadth-first	✅ No recursion limit	Level-by-level processing

Conclusion

Flattening nested dictionaries is essential when working with complex JSON data or configuration files.

• The recursive approach is the most readable and works well for typical nesting depths.
• For extremely deep structures, switch to the iterative stack-based approach to avoid hitting Python's recursion limit.

All methods support customizable separators and can be extended to handle lists, key filtering, and empty value preservation as needed.