Remove duplicate lines in pure .bat

Dell Dimension 4550
September 24, 2007 at 00:07:52
Specs: Windows XP Pro SP2, P4 2.60 Ghz\ 768 MB
I was working on a script for someone and needed a way to remove duplicate lines from a text file. I searched the forums for a solution and came across a thread where M2 mentioned his UNIQUE.COM utility to do just that. I was looking for a way to do it in pure batch and found the following code on JSI FAQ

It's very fast and very handy.

::==sortDup.bat
@echo off
setlocal
if {%1} EQU {} goto syntax
if not exist %1 goto syntax
set file=%1
set file="%file:"=%"
set work=%TEMP%\%~nx1
set work="%work:"=%"
set work=%work:\\=\%
sort %file% /O %work%
del /f /q %file%
for /f "Tokens=*" %%s in ('type %work%') do set record=%%s&call :output
endlocal
goto :EOF
:syntax
@echo ***************************
@echo Syntax: SortDup Input_File
@echo ***************************
endlocal
goto :EOF
:output
if not defined prev_rec goto write
if "%record%" EQU "%prev_rec%" goto :EOF
:write
@echo %record%>>%file%
set prev_rec=%record%

"Computer security." — Oxymoron


See More: Remove duplicate lines in pure .bat

Report •


#1
September 25, 2007 at 01:47:49
I also got thinking about doing it in pure bat, not least because UNIQUE requires the file to be sorted, which will be unacceptable in some situations.

==================================
@echo off > outfile
if %1'==' echo which file? && goto :eof
if not exist %1 echo %1 not found && goto :eof

for /f "tokens=* delims= " %%a in (%1) do (
find "%%a" < outfile > nul
if errorlevel 1 echo %%a >> outfile
)



=====================================
If at first you don't succeed, you're about average.

M2



Report •

#2
September 25, 2007 at 10:46:30
Nice, yours is much simpler, but does create a new output file. I think I'll use yours.

"Computer security." — Oxymoron


Report •

Related Solutions


Ask Question