Computing.Net > Forums > Unix > delete repeated entries

Computer Problems? Computing.Net has over 1,000,000 posts about all things technology related! Over 90% answered within 24 hours! Click here to start participating now! Also, be sure to check out the New User Guide.

delete repeated entries

Reply to Message Icon

Name: vsantana
Date: November 3, 2003 at 15:15:32 Pacific
OS: Solaris 8
CPU/Ram: ultra2 1GB
Comment:

Hi all,
the output of my nawk/sed script is leaving me with many repeated entries, ie:
entry1
entry1
entry2
entry2
...
There are not similarities on the entries, how can I eliminate all repeated entries, and leave only one occurence for each different entry?

your help is greatly appreciated.




Sponsored Link
Ads by Google

Response Number 1
Name: nails
Date: November 3, 2003 at 21:31:47 Pacific
Reply:

Hi:

Your output appears to be sorted. If that's the case why not send the output of your script to sort -u. (-u means unique).

You also might check out the uniq command.

Regards,

Nails


0

Response Number 2
Name: Anagram
Date: November 4, 2003 at 13:20:19 Pacific
Reply:

Nails has you on the right track, but I get the impression that some of the records are mostly similar but perhaps are not precise duplicates. Perhaps the same except for a datestamp which you wish to ignore in terms of uniqueness. If this is the case your explanation of the problem should have been more detailed.
Thus do I point you to:
man sort

You will there discover that you probably want to do something like:
cat datafile | sort -u -k 1,1 > output

which will send the data through the sort command, which will "key" in on only the first field, as delimited by white-space. Remember you must check the man pages for yourself, I am not familiar with Solaris, which may or may not have a different sort command.



0

Sponsored Link
Ads by Google
Reply to Message Icon

Related Posts

See More


date in file name Sco 5.05



Post Locked

This post is quite old and has been locked from receiving new replies. Please create a new posting instead.


Go to Unix Forum Home


Sponsored links

Ads by Google


Results for: delete repeated entries

delete repeated words in a line www.computing.net/answers/unix/delete-repeated-words-in-a-line-/5862.html

OpenBSD - fstab and boot problem www.computing.net/answers/unix/openbsd-fstab-and-boot-problem/5188.html

Accidentally deleted directory!!! www.computing.net/answers/unix/accidentally-deleted-directory/2981.html