How to find duplicate records in Unix

December 15, 2010 at 02:59:47
Specs: Windows XP
A file contain duplicate records like,

File 1 :
A
A
B
C
C
C
E
F

Out put should be like:

A
A
C
C
C

If A is having duplicate record, then i need both the original and the duplicate one in a separate file.

Please help me in solving this.

Thanks,
Thiru


See More: How to find duplicate records in Unix

Report •

#1
December 15, 2010 at 18:23:21
This script will sort your input file uniquely and then grep for instances of the string from the input file (I named it alpha.dat) and write those multiple occurrences to a file called alpha1.dat.

#!/bin/ksh

File=./alpha.dat

sort -u $File | while read line
do
Count=$(grep -c $line $File)
if [[ $Count -gt 1 ]]
then
grep $line $File >> alpha1.dat
fi
done

If this is a homework assignment (as I suspect it is), I'd suggest that you find your own method of working this problem. It is really trivial and any one who is working as a system administrator should be able to figure it out in 2 minutes or less. If they can't, they should consider becoming a Certified Windows Rebooter.


Report •
Related Solutions


Ask Question