Alternatives to writing to file

September 13, 2011 at 00:48:32
Specs: Macintosh , 2 x 2.66 GHz Dual-Core Intel Xeon
I'm a self taught at writing unix scripts. These have worked for me in the past but now I'm scaling them up I think some of my ways of approaching things could be far better :-)

For example I have a text file with 13495 lines on which I want to change the text. The replacement text is different for each of these lines.

At the moment I'm producing a script which does something like this:

sed 's/TextA/ReplacedTextA/g' File > File1; rm File; mv File1 File;
sed 's/TextB/ReplacedTextB/g' File > File1; rm File; mv File1 File;
.
.
.
This is repeated 13495 times!

Obviously this is very slow to run and I'm sure this is probably the worst way possible of doing this so any alternatives are very gladly received!

I had tried prior to this to link the commands in the mother of all pipes but obviously ran out of memory.

Thanks for any suggestions


See More: Alternatives to writing to file

Report •

#1
September 13, 2011 at 14:05:57
I have two suggestions:

First, I am not a Mac guy so I have to ask: is the sed version you are running GNU sed - the version that most all Linux versions use? If it is, GNU sed provides the -i option which allows editing a file in place.

# GNU sed only:
sed -i 's/TextA/ReplacedTextA/g' File

It will make a backup of file if you give -i an extension:

sed -i.bak 's/TextA/ReplacedTextA/g' File


Second, consider doing multiple edits on one file at one time using sed's -e option

# Untested
sed -e 's/TextA/ReplacedTextA/g' -e 's/TextB/ReplacedTextB/g' File > File1


Report •

#2
September 13, 2011 at 23:33:41
thanks nails I really appreciate you answering.

Yup I do have a -i option in my version of sed.

I tried both of your suggestions.

-e: won't work as it complained that the argument was too long.

-i: Does work but unfortunately doesn't speed up the process. In fact it was a little slower which surprise me.

On exactly the same files each of which were 10,000 long:

Original way: 4m24.704s
-i option: 4m35.194s


Strange as I would have expected that the lack of I/O to files with the -i option would have significantly speeded it up!


Report •
Related Solutions


Ask Question