How to Convert a 1D List to a 2D List with Variable Lengths in Python
Splitting a flat list into sublists of different sizes is a common data transformation task in Python. You might need to do this when parsing structured binary data, reshaping results from a database query, distributing work items across workers with uneven capacities, or converting a flat configuration list into grouped sections.
This guide covers the most efficient and readable approaches, explains how they work, highlights a common performance mistake to avoid, and provides a reusable function you can drop into any project.
Using an Iterator with islice (Recommended)ā
The most Pythonic and efficient way to split a list into chunks of variable length is to create an iterator from the list and consume it sequentially using itertools.islice.
Why an Iterator Works So Well Hereā
An iterator maintains an internal pointer to its current position. When you take a slice of a certain number of elements, the pointer advances by exactly that amount. The next slice starts right where the previous one left off. This means every element is visited exactly once, with no index tracking or data copying required.
from itertools import islice
data = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100]
chunk_sizes = [2, 3, 1, 4]
it = iter(data)
result = [list(islice(it, size)) for size in chunk_sizes]
print(result)
Output:
[[10, 20], [30, 40, 50], [60], [70, 80, 90, 100]]
Here is what happens step by step:
islice(it, 2)takes the first 2 elements:[10, 20]. The iterator now points to30.islice(it, 3)takes the next 3 elements:[30, 40, 50]. The iterator now points to60.islice(it, 1)takes the next 1 element:[60]. The iterator now points to70.islice(it, 4)takes the remaining 4 elements:[70, 80, 90, 100].
Each element is consumed exactly once, and no extra memory is allocated beyond the result list itself.
Using Manual Index Slicingā
If you prefer a solution with no imports, you can manually track a start index and use standard list slicing. This approach is easy to understand and works well for small to medium lists:
data = [10, 20, 30, 40, 50, 60]
chunk_sizes = [1, 2, 3]
result = []
start = 0
for size in chunk_sizes:
chunk = data[start:start + size]
result.append(chunk)
start += size
print(result)
Output:
[[10], [20, 30], [40, 50, 60]]
This is straightforward, but each slice creates a new list from the original data. For very large lists where memory is a concern, the iterator approach is more efficient because islice does not create intermediate copies.
Handling Edge Casesā
Real-world data is rarely perfect. The chunk sizes might not add up to the total length of the list, or some sizes might request more elements than remain. Handling these cases gracefully prevents unexpected crashes.
When Chunk Sizes Exceed Available Elementsā
If the requested sizes add up to more than the list contains, islice handles this naturally by returning fewer elements once the iterator is exhausted:
from itertools import islice
data = [1, 2, 3, 4, 5]
chunk_sizes = [2, 5, 3] # Total requested: 10, but only 5 elements exist
it = iter(data)
result = [list(islice(it, size)) for size in chunk_sizes]
print(result)
Output:
[[1, 2], [3, 4, 5], []]
The first chunk gets 2 elements, the second gets the remaining 3 (not the requested 5), and the third gets an empty list because the iterator is already exhausted.
Filtering Out Empty Chunksā
If you do not want empty sublists in your result, you can filter them out using a conditional expression with the walrus operator:
from itertools import islice
data = [1, 2, 3, 4, 5]
chunk_sizes = [2, 5, 3]
it = iter(data)
result = [chunk for size in chunk_sizes if (chunk := list(islice(it, size)))]
print(result)
Output:
[[1, 2], [3, 4, 5]]
The empty list that would have been produced by the third chunk size is excluded from the result.
When Chunk Sizes Do Not Cover All Elementsā
If the sum of chunk sizes is less than the length of the list, the remaining elements are simply left unconsumed by the iterator:
from itertools import islice
data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
chunk_sizes = [2, 3] # Only covers 5 of 10 elements
it = iter(data)
result = [list(islice(it, size)) for size in chunk_sizes]
# Remaining elements are still accessible from the iterator
remaining = list(it)
print(f"Chunks: {result}")
print(f"Remaining: {remaining}")
Output:
Chunks: [[1, 2], [3, 4, 5]]
Remaining: [6, 7, 8, 9, 10]
What to Avoid: Using pop(0) in a Loopā
A common mistake, especially for developers coming from other languages, is using list.pop(0) repeatedly to consume elements from the front of the list:
# Wrong approach: O(nĀ²) time complexity
data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
chunk_sizes = [3, 3, 4]
result = []
for size in chunk_sizes:
chunk = []
for _ in range(size):
chunk.append(data.pop(0))
result.append(chunk)
print(result)
Output:
[[1, 2, 3], [4, 5, 6], [7, 8, 9, 10]]
While this produces the correct output, it has a serious performance problem. Every call to data.pop(0) removes the first element and shifts every remaining element one position to the left. For a list with n elements, this makes the overall operation O(nĀ²). On a list with a million elements, this can be thousands of times slower than the iterator approach.
The iter + islice approach runs in O(n) time because each element is visited exactly once with no shifting.
Avoid list.pop(0) inside loops when working with large lists. It silently degrades performance because every pop from the front requires shifting all remaining elements. Use an iterator or index-based slicing instead.
Creating a Reusable Functionā
Wrapping the logic in a function makes it easy to reuse across your codebase:
from itertools import islice
def split_into_chunks(data, chunk_sizes):
"""Split a flat list into sublists of specified variable sizes.
Args:
data: The flat list to split.
chunk_sizes: An iterable of integers specifying the size of each chunk.
Returns:
A list of sublists. If the data runs out before all chunk sizes are
fulfilled, the remaining chunks will contain fewer elements or be empty.
"""
it = iter(data)
return [list(islice(it, size)) for size in chunk_sizes]
# Example usage
data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
print(split_into_chunks(data, [3, 3, 4]))
print(split_into_chunks(data, [1, 2, 3, 4]))
print(split_into_chunks(data, [5, 5]))
Output:
[[1, 2, 3], [4, 5, 6], [7, 8, 9, 10]]
[[1], [2, 3], [4, 5, 6], [7, 8, 9, 10]]
[[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]]
Performance Comparisonā
| Method | Time Complexity | Memory Efficiency | Best Use Case |
|---|---|---|---|
iter + islice | O(n) | High (no intermediate copies) | Production code, large datasets |
| Manual index slicing | O(n) | Moderate (creates slice copies) | Simple scripts, no imports needed |
pop(0) in a loop | O(nĀ²) | Low (constant reshuffling) | Not recommended |
Summaryā
Converting a 1D list into a 2D list with variable-length sublists is a common operation that Python handles elegantly. The iterator approach with islice is the recommended solution for most situations. It processes each element exactly once, uses minimal memory, handles edge cases like insufficient data gracefully, and reads clearly as a single list comprehension.
For simpler scripts where you want to avoid imports, manual index slicing is a clean and readable alternative. The one approach to avoid is pop(0) in a loop, which introduces quadratic time complexity that becomes a real problem as your data grows.