Hi,

I have a file with duplicate records in it. I want to identify the duplicate records and put them into a separate file. Please let me know how to do it.

Please note that each record is 2 lines in length and has spaces in each record.

Thanks,
Anil

I see no clear method of doing this. My idea is to place each 2-line record into a file, compare the files and delete the duplicates. Here is a 3 step kludge:

1) Create a mytmp directory in the directory where the file resides. Then use the unix split command to break apart the file into 2-line files:

split -l 2 database.txt mytmp/m1

all the file names will start with "m1".

2) In ksh, find the unique check sum, using cksum, of each file. sort the file and any file that's unique, delete it:

#!/bin/ksh
fr=0
cksum $(find mytmp -type f -name "m1*" -print)|sort -k 1,1n |
while read c1 c2 n3
do
if [ fr -eq 0 ]; then
prevobj=$c1
fr=1
continue
fi
# save check sums and file sizes that are duplicate
if [[ $prevobj -eq $c1 ]]; then
rm $n3
else
prevobj=$c1
fi
done

Finally, put everything back together from mytmp:

cat $(find mytmp -name "m1*" -print) > newdatabase.txt

Sorry, it can't be cleaner.