Batch file to delete lines in large CSV file

November 13, 2009 at 04:59:06
Specs: Windows XP
I need to delete the first 27 lines of a file then write in a line on top; I have the script which do exactly the same but since the file I need to change is about 1MB, the run time is very long.

Is there any way to improve my run time?

my script:
@echo off
setLocal EnableDelayedExpansion

if exist *.txt del *.txt

for /f "tokens=* delims= " %%a in ('dir/b/a-d *.csv') do (
echo mystring >> %%~Na.txt
for /f "tokens=* skip=27 delims= " %%e in (%%a) do (
echo %%e >> %%~Na.txt

See More: Batch file to delete lines in large CSV file

Report •

November 13, 2009 at 06:10:13
I may be missing something. If you just process one csv, eliminate the double FOR.

I ran this on a monster csv, 44MB, and it took just about I minute.

@echo off & setLocal enableDELAYedexpansion

> newfile echo mystring

for /f "skip=27 tokens=* delims= " %%a in (big.csv) do (
>> newfile echo.%%a

Helping others achieve escape felicity


Report •

November 13, 2009 at 06:21:55
"tokens=* skip=27 delims= "
You might (Might, as in I haven't tested it) get an improvement by using the following options:
"skip=27 delims="

If that doesn't work, your best bet would be to change your language. Even VBScript should be faster.

Const infile = "in.txt"
Const outFile = "out.txt"
Set fso = CreateObject("Scripting.FileSystemObject")

With fso.OpenTextFile(infile)
  For i = 1 To 27
  Next 'i
  fso.OpenTextFile(outFile, 2, True).Write .ReadAll
End With

Report •

November 13, 2009 at 06:42:46
Sorry I should have mentioned that - I need to repeat the exercise for all csv in the folder....

And even worse I just received some files which need processing are reaching 100MB!

Report •

Related Solutions

Ask Question