Computer Problems? Computing.Net has over 1,000,000 posts about all things technology related! Over 90% answered within 24 hours! Click here to start participating now! Also, be sure to check out the New User Guide.
sort on 60M lines
Name: dips Date: October 9, 2007 at 22:20:39 Pacific OS: sunos.5.8/8.7 CPU/Ram: 512 MB Product: Pentium (4)
Comment:
A flat file with 60 million lines needs to be sorted out (and then uniq to be done on it). I want to know that if anybody knows how fast it would be? coz right now we are handling that in our ETL tool (Informatica) because of sorting such a huge amount of data cache fails. Doing sort and then uniq will help?
Name: fpmurphy Date: October 15, 2007 at 16:58:30 Pacific
Reply:
Using the standard sort and uniq utilities to sort 60M lines on a UNIX box will be difficult for a number of reasons. The sort utility typically creates a temporary file and an output file - each of which will be at least the same size as the original file. This means that you need plenty of free disk space.
I suggest that you instead look at some third party application such as CoSort.
Summary: I'm using grep to search for a string in a text file once i've found a match what i really want to do is substitute the line above the match. How would i go about doing this? I understand how to use ...
Summary: account Last login ===MB=== Charlie Brown Nov 5 16:15 182.22 Exc 5MB James Boothe Oct 23 14:52 27.83 Exc 5MB (ora816) Nov 5 13:09 3.85 Trish Davis Nov 5 0...
Summary: First of all Guy, if you want to nest commands in the middle of other commands, the nested commands need to be delimited with backward quotes, such as: echo I am `whoami`, logged into `uname -n` Secon...
0 
