How to Safely Access and Merge Nested Dictionaries in Python

Working with deeply nested dictionaries, common when handling JSON APIs, can be error-prone. One missing key in a chain like data['users'][0]['profile']['location'] causes a crash.

This guide provides professional, robust utilities for safely accessing and recursively merging complex dictionaries without external dependencies.

The Problem with Standard Access

Python's standard dictionary access raises a KeyError if a key is missing. When keys are nested, handling this requires deeply nested try-except blocks or verbose .get() chaining.

# The "Naive" Approach
try:
    city = data['user']['profile']['address']['city']
except (KeyError, TypeError):
    city = 'Unknown'

# The "Chained Get" Approach (Hard to read)
city = data.get('user', {}).get('profile', {}).get('address', {}).get('city', 'Unknown')

Both approaches reduce code readability and maintainability.

Solution 1: Safe Access with Dot Notation

Instead of writing repetitive error handling, we can create a utility function that accepts a "dot-path" string (e.g., "user.profile.name"). We use functools.reduce to traverse the dictionary dynamically.

The get_nested Helper

from functools import reduce

def get_nested(data, path, default=None):
    """
    Get value from a nested dictionary using a dot-path string.
    
    Args:
        data (dict): The dictionary to search.
        path (str): The dot-separated path (e.g., 'meta.pagination.page').
        default (any): The value to return if the path doesn't exist.
    """
    try:
        return reduce(lambda d, k: d[k], path.split('.'), data)
    except (KeyError, TypeError):
        return default

Usage Example

response = {
    'user': {
        'id': 101,
        'profile': {
            'name': 'Alice',
            'settings': None # Note: settings is not a dict
        }
    }
}

# 1. Accessing a valid path
name = get_nested(response, 'user.profile.name')
print(f"Name: {name}")   # Output: Alice

# 2. Accessing a missing key
age = get_nested(response, 'user.profile.age', 'N/A')
print(f"Age: {age}")     # Output: N/A

# 3. Handling type errors (traversing through None)
# 'user.profile.settings' is None, so 'settings.theme' would crash without our helper
theme = get_nested(response, 'user.profile.settings.theme', 'Dark Mode')
print(f"Theme: {theme}") # Output: Dark Mode

Output:

Name: Alice
Age: N/A
Theme: Dark Mode

Why TypeError?

The except (KeyError, TypeError) catch block is crucial. A KeyError happens when a key is missing. A TypeError happens if you try to access a key on something that isn't a dictionary (like trying to do None['theme'] in the third example above).

Solution 2: Deep Merging (Recursive Update)

Python's built-in dict.update() method performs a shallow merge. It completely overwrites top-level keys. If you are merging configuration files where you only want to override specific nested fields while keeping others, update() destroys your data.

The Problem with Shallow Update

config = {
    "db": {"host": "localhost", "port": 5432},
    "debug": True
}

overrides = {
    "db": {"host": "prod-db"}
}

config.update(overrides)
print(config)

Output:

{'db': {'host': 'prod-db'}, 'debug': True}

But we lost the port!

The deep_update Helper

To preserve nested data, we need a recursive function that checks if both the source and the override are dictionaries before merging.

def deep_update(source, overrides):
    """
    Recursively update a dictionary.
    
    Args:
        source (dict): The dictionary to be updated in-place.
        overrides (dict): The dictionary containing changes.
    """
    for key, value in overrides.items():
        # Check if both value and the existing key are dictionaries
        if isinstance(value, dict) and isinstance(source.get(key), dict):
            deep_update(source[key], value)
        else:
            source[key] = value
    return source

Usage Example

import json

def deep_update(source, overrides):
    """
    Recursively update a dictionary.
    
    Args:
        source (dict): The dictionary to be updated in-place.
        overrides (dict): The dictionary containing changes.
    """
    for key, value in overrides.items():
        # Check if both value and the existing key are dictionaries
        if isinstance(value, dict) and isinstance(source.get(key), dict):
            deep_update(source[key], value)
        else:
            source[key] = value
    return source

config = {
    "db": {"host": "localhost", "port": 5432},
    "logging": {"level": "INFO", "file": "app.log"}
}

env_overrides = {
    "db": {"host": "prod-db"},       # Only change host, keep port
    "logging": {"level": "ERROR"}    # Only change level, keep file
}

deep_update(config, env_overrides)

print(json.dumps(config, indent=2))

Output:

{
  "db": {
    "host": "prod-db",
    "port": 5432
  },
  "logging": {
    "level": "ERROR",
    "file": "app.log"
  }
}

Immutability

The deep_update function above modifies the dictionary in-place. If you need to keep the original dictionary intact, wrap the call with copy.deepcopy():

import copy
new_config = deep_update(copy.deepcopy(base_config), overrides)

Production Considerations

While these helper functions are excellent for scripts and lightweight applications, robust production systems often use specialized libraries for data validation and schema definition.

1. Pydantic: The industry standard for data validation. It allows you to define classes (Models) for your data. If data is missing or wrong, it raises precise errors.
2. Glom: A library specifically designed for restructuring and accessing nested data in Python.

However, for simple configuration merging or API response handling, get_nested and deep_update are lightweight, dependency-free solutions.

Summary

Operation	Standard Method	Recommendation
Deep Access	d['a']['b']['c']	Use get_nested(d, 'a.b.c') to avoid crashes.
Updating	d.update(new)	Use deep_update(d, new) to preserve nested keys.