How to Run Tasks Concurrently in Python Using Threads

Python's threading module allows you to run multiple operations concurrently, making your programs more efficient, especially when dealing with I/O-bound tasks (like reading files or network requests). This guide covers creating threads, synchronizing them, and using advanced tools like ThreadPoolExecutor and synchronization primitives.

Basic Threading

The simplest way to run code in parallel is to create a Thread object, passing it a target function.

import threading
import time

def worker(name):
    print(f"Thread {name} starting...")
    time.sleep(1)
    print(f"Thread {name} finished.")

# ✅ Correct: Creating and starting a thread
t = threading.Thread(target=worker, args=("A",))
t.start()

# Wait for it to complete
t.join()
print("Done.")

note

t.start() begins the execution. t.join() blocks the main program until the thread finishes. Without join(), the main program might exit before the thread completes (unless it's a daemon thread).

Synchronization with Locks

When multiple threads modify the same variable (a "race condition"), data corruption can occur. A Lock ensures only one thread accesses a resource at a time.

import threading

counter = 0
lock = threading.Lock()

def increment():
    global counter
    with lock:  # ✅ Correct: Automatically acquires and releases
        counter += 1

threads = [threading.Thread(target=increment) for _ in range(100)]

for t in threads: t.start()
for t in threads: t.join()

print(f"Final counter: {counter}")

Thread Pools (Best Practice)

Managing threads manually (starting, joining, storing in lists) is tedious. The concurrent.futures.ThreadPoolExecutor handles the lifecycle of threads automatically.

Using executor.submit()

import concurrent.futures

def task(n):
    return n * n

# ✅ Correct: Context manager handles cleanup
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
    # Submit tasks individually
    future = executor.submit(task, 5)
    print(f"Result: {future.result()}")

Using executor.map()

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
    items = [1, 2, 3, 4, 5]
    # Applies 'task' to every item in 'items' concurrently
    results = executor.map(task, items)
    
    for res in results:
        print(res)

Synchronization Primitives (Event, Timer, Barrier)

Python provides specialized objects to coordinate threads.

Event

Allows one thread to signal others to proceed. Ideally used when threads need to wait for a specific condition (e.g., "Database is ready").

ready_event = threading.Event()

def waiter():
    print("Waiting for signal...")
    ready_event.wait()  # Blocks here
    print("Signal received!")

t = threading.Thread(target=waiter)
t.start()

# Signal the thread to continue
ready_event.set()

Timer

Runs a function after a delay in a separate thread.

def delayed_action():
    print("Timer finished!")

# Run after 2.0 seconds
t = threading.Timer(2.0, delayed_action)
t.start()

Barrier

Forces a group of threads to wait until all of them reach a certain point.

# Wait for 3 threads to reach the barrier
barrier = threading.Barrier(3)

def worker():
    print("Worker ready")
    barrier.wait() # Blocks until 3 threads arrive here
    print("Worker GO!")

for _ in range(3):
    threading.Thread(target=worker).start()

Conclusion

To effectively use threading in Python:

1. Use ThreadPoolExecutor for most tasks; it's cleaner and safer than manual thread management.
2. Use Lock whenever threads modify shared state.
3. Use Event or Barrier for complex coordination between threads.
4. Use Daemon Threads (daemon=True) for background tasks that shouldn't prevent the program from exiting.