Hi everyone, I am trying to remove some lines from all the files in a directory, for example I am looking at the fields 25-27 and if it is 19 I am trying to remove the whole line from that file, similarly from all the other files in that directory. The files are named as abcxxxxxxD.txt and abcxxxxxxDM.txt. I tried to use grep -v to find the needed lines where it is not matched and tried to write in a temp file and replace the original file with the temporary file, but this is taking long time as there are 6500 files in the directory. Is there any simpler way to do it. And also I am unable to run on all the files, I can only run the script on one file at a time, is there a way to run the script on all the files in the directory. Thank you lp

You can easily run a command on all the files you want by piping 'ls' into 'xargs command' or by using the 'for name in $(/bin/ls) ; do command $name ; done' construct at the commandline. The command: awk '{ if (substr($0,25,3) != "019") print $0 > ARGV[1]}' might do the example job for you but your "...fields 25-27...if it is 19..." is ambiguous: Do you mean fields 25-27 contain a 3 digit +ive integer?

Thank you wolfbone, my fields are 25-26 two digit number, I will try it out.

Hi, I have used it in the following way. Src=/MonthlyData Tgt=/MarketData ls $Tgt | xargs -i awk '{ if (substr($0,7,2) != "14") print $0 > ARGV[1]}' I got the following error, awk: A print or getline function must have a file name The input line number is 1. The source line number is 1. Any help is really appreciated. Thank you

I don't know why that particular error came up but on my system, if I use the -i option to xargs like that, nothing at all happens. Try without the -i option. Also 'ls $Tgt' may not be printing /MarketData/file_name but just file_name. Make sure ls is printing out the correct absolute or relative paths to each file (and nothing else). Note that the command as it stands will modify the files immediately so I also suggest you test it out on a copy of a subset of the files first and perhaps remove the '> ARGV[1]' bit to make the output go to stdout until you're happy with the results. Finally - what is the Src=/MonthlyData for?

There are two sets of files (*D.txt and *DM.txt that have the value to be checked at 25-26 and 7-8 fields respectively) in the folder so I separated the files into MonthlyData and MarketData folders, so that i can work on one set at a time. I made sure the ls is giving the right set of the file list. The script is running, but unable to get the desirable result, the records have 19 in 7-8 position are not getting eliminated. And I tried the script on just five files and the script is not coming out of the loop. This is what i used. xargs awk '{if(substr($0,7,2)!="14") print $0}' Thanks for your help

Well it can't be the awk bit because it's too simple but at least try it out on a single file like so: awk '{if(substr($0,7,2)!="14") print $0}' datafile If that doesn't work I can't explain it but it certainly looks to me like xargs is behaving differently on your system so I'd use the shell construct instead.