Unix script to sort .dat files

Sun microsystems / Sunblade 2000
March 9, 2009 at 10:55:12
Specs: Sun Solaris 2002
Hi,

I have used IDL to generate data from MODIS satellite images of array based, which has been stored into files named- testop.400.215.dat, testop.400.216.dat etc etc. The text is arranged into 3 columns- Year, Day of Year, Data Value an example below-

2002 182 -0.171602
2002 183 -0.150347
2002 184 -0.158976
2002 186 -0.137255
2002 187 -0.147758
2002 188 -0.169247
2002 190 -0.161879
2002 191 -0.151100
2002 192 -0.108192
2002 195 -0.158958
2002 196 -0.176081
2002 197 -0.162375
2002 198 -0.170275
2002 199 -0.165494
2002 200 -0.186664
2002 201 -0.159151
2002 202 -0.167174
2002 203 -0.169629
2002 204 -0.155896
2002 205 -0.169563
2002 206 -0.168418
2002 207 -0.180519
2002 208 -0.176451
2002 209 -0.174065
2002 211 -0.187861
2002 212 -0.171713
2002 213 -0.169252

I would ideally like to write a unix shell script which enables me to move each file of my dataset into 1 text file with the same format as before but additional columns for each pixel data value (-0.16252 for example), so effectively a new column for the information stored in column 3 of each file, however the series for column 1 and colum 2 is not consistent, as my IDL script removed values which were outside my threshold range. Therefore I was wondering if it was possible to write a script to deal with the gaps in the data as well, so I can produce 1 complete data set.

I hope this makes sense, I am generally quite new to unix and also on a tight time schedule.

Many Thanks! :-)


See More: Unix script to sort .dat files

Report •


#1
March 9, 2009 at 13:24:17
What I think you are asking for is to append all your *.dat files into one text file and then sort on the first two columns so they are in order?

This script should do it:

#!/bin/ksh

# cd <to your data directory>
rm -f datafile.txt
# append all the data files
for i in *.dat
do
   cat "$i" >> datafile.txt
done

sort -k1,2 datafile.txt > newdatafile.txt
# end script

Let me know if I've missed your requirements


Report •

#2
March 12, 2009 at 07:19:36
Hi, I have now managed to sort the problem with the first two columns in order. All I need a script for now is to add the third column from each testop* file into a new column in the one file so effectively will look like:

Col 1 Col 2 Col 3 Col 4 etc

2002 182 -0.171602 -0.169629
2002 183 -0.150347 -0.167174
2002 184 -0.158976 -0.155896
2002 186 -0.137255 -0.180519
2002 187 -0.147758 -0.171713
2002 188 -0.169247 -0.174065
2002 190 -0.161879 -0.169563
2002 191 -0.151100 -0.171756

but each new column needs to correspond with the next file in the sequence...I hope thats clearer, thanks for your response none the less.

Thanks :-)


Report •

#3
March 12, 2009 at 13:43:19
Not exactly certain of your requirements, but I think you want the 3rd column of all testop* files to be placed as columns of the original datafile that contains 2 columns. The unix paste command pastes columns from 1 file onto anoter file:

#!/bin/ksh

# cd <to your data directory>

# original mydatafile.txt contains only 2 cols
rm -f ./tmpfile ./tmpfile1
for i in testop*
do
    # don't know how many columns testop* file, but at least 3
    rm -f ./tmpfile
    # create a tmpfile with the 3rd column, 1 per line
    while read col1 col2 col3 therest
    do
       echo $col3 >> ./tmpfile
    done <  ${i}
    paste -d " " mydatafile.txt ./tmpfile > ./tmpfile1
    mv ./tmpfile1 mydatafile.txt
done


Report •

Related Solutions

#4
March 16, 2009 at 04:06:40
hmmm... it wont let me run the script stating :permission denied.

this may sound like a daft question but what language is this written in?

Thanks again


Report •

#5
March 16, 2009 at 07:53:03
This is a korn shell script. You can tell by the first line - ksh.

I'm assuming you've placed this code in a file on your Solaris system? If so, you have to set the file permissions to execute using the chmod command:

chmod 755 <yourscript>


Report •


Ask Question