linux - process data from 10 files

April 13, 2010 at 22:44:35
Specs: linux
what we want to do is we have 10 files, each file have 100,000 lines, they have equal line number.
first six columns in each files is the same, but 7th column is not,

we need to merge them into a big report, keep first 6 columns, and concat seven column from each file,

finanl output should be like the following:
column1, coulmn2... column6, file1 column 7, fil2 column7, file3 column7, file 10 column 7
file 1
1,2,3,4,5,6 60
1,2,3,4,5,6 70
fil3 3
1,2,3,4,5,6 80

file10 130

final files:
1,2,3,4,5 60,70,80,....130

See More: linux - process data from 10 files

Report •

April 28, 2010 at 18:13:46
I think you're going to have to have all 10 files open at once, so a utility like 'awk' probably isn't going to help. A C program could open all 10 files. Loop through line 1, line 2, etc. Within each line, loop through the open files and read the 7 columns. Here you could do a sanity check that the first 6 are identical across all files. Output the first 6, and then output the 7th from each file. That's a Quick 'n' Dirty way of doing it, but it should work.

Report •
Related Solutions

Ask Question