sed - delete duplicate lines

Computing.Net > Forums > Unix > sed - delete duplicate lines

sed - delete duplicate lines

Original Message

Name: snomys
Date: August 2, 2006 at 05:06:30 Pacific
Subject: sed - delete duplicate lines
OS: win nt
CPU/Ram: pentium 4/256
Model/Manufacturer: IBM

Comment:

ive tried all the possibilties on the sed one liners sites, and tried everything mysefl, but i cant figure it out...
ive got a file, containing duplicate lines, like this

delete from itm.tb_wf_objects where alert_id = 51573
delete from itm.tb_wf_objects where alert_id = 51597
delete from itm.tb_wf_objects where alert_id = 51617
delete from itm.tb_wf_objects where alert_id = 51617
as you can see alert id 51617 appears twice, i want to rmeove all lines that have the same alert_ids. table names can change as well.

Report Offensive Message For Removal

Response Number 1

Name: nails
Date: August 2, 2006 at 08:34:42 Pacific
Subject: sed - delete duplicate lines

Reply: (edit)

I don't think sed is the best tool. If your file is already sorted simply use the uniq command:
cat myfile|uniq
If the file isn't sorted, sort it with the unique switch:
sort -u myfile

Report Offensive Follow Up For Removal


Delete first line of a file sed delete minus one to remove duplicate lines from file Creating duplicate lines Delete all lines containing pattern	Awk scripting help sed, deleting 2nd matching line Deleting Empty lines using sed Deleting multiples lines in one SED duplicate line in middle of file

Last Line is not complete

grep/awk/conditional prin...

Use following form to reply to current message:

Name: From My Computing.Net Settings E-Mail: From My Computing.Net Settings Subject: sed - delete duplicate lines

Comments:

  Homepage URL (*): 
Homepage Title (*): 
         Image URL:

How often do you use Computing.Net?

Poll Finishes In 4 Days.
Discuss in The Lounge

Extracting multipe lines from file

Cannot reboot W2K

blank hard drive/no bios

Xbox 360 Serial Number

webbased user management

All Posts Awaiting Replies