Skip to main content

How to Split a Large Text File into Smaller Files in Batch Script

Working with very large text files, such as multi-gigabyte logs or data dumps, can be slow and cumbersome. Standard text editors may struggle to even open them. A common solution is to split the large file into a series of smaller, more manageable "chunks." While Windows has no native split command like Linux, you can build this functionality yourself using a batch script.

This guide will teach you the pure-batch method for splitting a file by line count, a task that requires careful management of loops and counters. We will then demonstrate the vastly superior and simpler modern approach using a PowerShell one-liner, which is the recommended method for its speed and reliability.

The Challenge: No Native split Command​

Unlike other operating systems, cmd.exe provides no built-in utility to split a file. This means we must create the logic from scratch: reading the source file line by line, keeping track of how many lines we've written to the current chunk, and creating a new chunk file once a limit is reached. This process is possible but is significantly slower and more complex than modern alternatives.

The Core Method (Pure Batch): Looping and Counting​

The logic for a pure-batch solution involves a master FOR loop and several counter variables:

  1. Read the source file line by line.
  2. Keep a line_count variable to track progress within the current chunk.
  3. Keep a file_count variable to create sequentially numbered output files (e.g., output_1.txt, output_2.txt).
  4. Append each line to the current output file.
  5. When line_count reaches the maximum lines per file, reset it to zero and increment file_count to start a new output file.

The Script (Pure Batch): Splitting a File by Line Count​

This script implements the above logic. It is complex but demonstrates how to solve the problem with native batch commands.

@ECHO OFF
SETLOCAL ENABLEDELAYEDEXPANSION

REM --- Configuration ---
SET "SOURCE_FILE=large_data_file.log"
SET "OUTPUT_PREFIX=chunk_"
SET "MAX_LINES=10000"

IF NOT EXIST "%SOURCE_FILE%" (ECHO [ERROR] Source file not found. & GOTO :EOF)

ECHO Splitting "%SOURCE_FILE%" into chunks of %MAX_LINES% lines...

SET "line_count=0"
SET "file_count=1"

REM Use FINDSTR /N "^" to robustly handle empty lines.
FOR /F "tokens=1,* delims=:" %%A IN ('FINDSTR /N "^" "%SOURCE_FILE%"') DO (
    IF !line_count! EQU 0 (
        ECHO Creating file: %OUTPUT_PREFIX%!file_count!.log
    )

    SET /A "line_count+=1"
    
    REM Append the original line content (%%B) to the current output file.
    ECHO %%B >> "%OUTPUT_PREFIX%!file_count!.log"

    IF !line_count! EQU %MAX_LINES% (
        SET "line_count=0"
        SET /A "file_count+=1"
    )
)

ECHO.
ECHO --- Splitting complete ---
ENDLOCAL

How the script works

  • SETLOCAL ENABLEDELAYEDEXPANSION: This is essential. It allows us to use !line_count! and !file_count! to get the current value of the counters inside the FOR loop.
  • FINDSTR /N "^": This is a critical trick. A standard FOR /F loop skips empty lines. This command prefixes every line with its number (e.g., 123:content), ensuring that no lines are skipped. We then use tokens=1,* delims=: to separate the line number (%%A) from the original content (%%B).
  • ECHO %%B >> ...: The >> is the append operator. It adds the line to the current output file without overwriting it.
  • IF !line_count! EQU %MAX_LINES% ...: This is the logic that triggers a "rollover." When the current chunk is full, the line counter is reset, and the file counter is incremented to start the next chunk.

For this task, PowerShell is dramatically faster, simpler, and more efficient. It can read files in large chunks, which is much better for performance than the line-by-line approach of batch.

This single command, executed from a batch file, replaces the entire complex script from above.

@ECHO OFF
SET "SOURCE_FILE=large_data_file.log"
SET "OUTPUT_PREFIX=chunk_"
SET "MAX_LINES=10000"

ECHO Splitting with PowerShell...
powershell -Command "$i=0; Get-Content '%SOURCE_FILE%' -ReadCount %MAX_LINES% | ForEach-Object { $i++; $_ | Set-Content -Path '%OUTPUT_PREFIX%-$i.log' }"
ECHO Done.
  • Get-Content -ReadCount %MAX_LINES%: This is the key to its performance. It reads up to 10000 lines from the source file into memory at once.
  • ForEach-Object { ... }: This loop runs once for each chunk of lines.
  • $_ | Set-Content ...: $_ represents the current chunk of lines, which is written directly to a new output file.

Common Pitfalls and How to Solve Them​

Problem: Performance on Very Large Files​

The pure-batch method is extremely slow on large files (hundreds of megabytes or more). It reads the file line by line, and the overhead for each line adds up significantly.

Solution: Use the PowerShell method. Its -ReadCount parameter is specifically designed for high-performance processing of large files, making it orders of magnitude faster than the batch loop.

Problem: Special Characters and Empty Lines​

  • Empty Lines: The pure-batch script shown above correctly handles empty lines because of the FINDSTR /N "^" trick. A simpler FOR /F loop would fail.
  • Special Characters (!, ^, &): These can cause major problems in batch scripts, especially with DelayedExpansion.

Solution: The PowerShell method is not affected by these issues. It handles all characters and empty lines correctly by default, making it far more robust.

Practical Example: Splitting a Large Log File​

This script uses the recommended PowerShell method to split a massive web server log into smaller, 50,000-line chunks for easier analysis.

@ECHO OFF
SETLOCAL

REM --- Configuration ---
SET "LOG_FILE=C:\IISLogs\W3SVC1\u_ex231027.log"
SET "OUTPUT_DIR=C:\LogAnalysis\Chunks"
SET "OUTPUT_PREFIX=log_part"
SET "LINES_PER_FILE=50000"

ECHO --- Log File Splitter ---
ECHO.
IF NOT EXIST "%LOG_FILE%" (ECHO [ERROR] Log file not found. & GOTO :End)
MKDIR "%OUTPUT_DIR%" 2>NUL

ECHO Splitting log file into %LINES_PER_FILE%-line chunks...
powershell -NoProfile -ExecutionPolicy Bypass -Command ^
    "$i=0; Get-Content '%LOG_FILE%' -ReadCount %LINES_PER_FILE% | ForEach-Object { " ^
        "$i++; $outPath = '%OUTPUT_DIR%\%OUTPUT_PREFIX%-$i.log'; " ^
        "$_ | Set-Content -Path $outPath " ^
    "}"

ECHO.
ECHO [SUCCESS] Splitting complete. Chunks are in "%OUTPUT_DIR%".

:End
ENDLOCAL

Conclusion​

While it is possible to split a large file by line count using a pure batch script, the method is a complex academic exercise with significant performance limitations.

  • The pure-batch FOR /F loop is a good demonstration of advanced scripting logic but is too slow and fragile for most real-world applications.
  • The PowerShell Get-Content -ReadCount method is the overwhelmingly superior choice. It is faster, more robust, handles all edge cases correctly, and is simpler to write and maintain.

For any task involving the splitting of large files, leveraging PowerShell from within your batch script is the professional and recommended approach.

Table of Contents