hai,
i want to remove duplicate lines from any part of the file...uniq command is used to remove only the adjacent duplicate lines...if we use sort command with -u option it removes duplicate lines from any part of the file,but the file is sorted...i dont want the file to be sorted,but i want to remove duplicate lines from any part of the file.... this is my problem...if anybody knows answer for this please reply....

If your file is not extremely large, the awk script below should work.

awk '{
if ($0 in stored_lines)
   x=1
else
   print
   stored_lines[$0]=1
}' filein > fileout

Report Offensive Follow Up For Removal

Reply:

hai Jim,

thanx a lot for ur response....the sript u gave is working...i got wat i want...
thank u very much...

My requirement is little different, actually I have 2 files, one of the file has 800 line another has 50. No I wanted to check with respect to first file if there are any duplicate lines in 2nd file. And if it is present then the output file should remove that line from the 800 lines. It should not add any lines of file2. Only it should check with respect to 1st file.

awk 'BEGIN{
while ((getline < "file50") > 0)
   list50[$1] = 1}
!list50[$1] {print}' file800 > file800new

Whn i use this script it says

> awk 'BEGIN
Unmatched '.

Some platforms require each end-of-line to be escaped with a backslash when a quoted string overflows to multiple lines. Try this:

awk 'BEGIN{ \
while ((getline < "file50") > 0) \
   list50[$1] = 1} \
!list50[$1] {print}' file800 > file800new

Report Offensive Follow Up For Removal

It worked well after going into bash mode.

Thanks a lot Jim