Filter IIS log files to remove internal IP's

April 29, 2009 at 02:45:30
Specs: Windows 2003 Server, 3Ghz / 1GB
It's a long time since I last wrote a .bat file. I need to parser an IIS log file and and output another file which contains just external IP addresses. I have written two batch files to do this. The 1st contains the lines:

SET yy=%date:~8,4%
SET mm=%date:~3,2%
SET dd=%date:~0,2%
SET N=%yy%%mm%%dd%

FOR /F "TOKENS=* SKIP=3 DELIMS=\n" %%a IN (^%N%.log) DO (
	ECHO %%a > log_temp.tmp
	CALL log_filter.bat log_temp.tmp
)

And the 2nd called log_filter.bat contains

TYPE d:\log_temp.tmp|FIND "192.168"
IF ERRORLEVEL 1 GOTO NOT_FOUND
IF ERRORLEVEL 0 GOTO XIT 

:NOT_FOUND
	TYPE d:\log_temp.tmp >> d:\log_cleaned.txt
:XIT

It works - but hits the server resources, CPU 100% whilst processing a 35MB file and takes an hour to complete.

Is there a way to read each line into memory to improve performance, rather than output for each line being written to a temp file?


See More: Filter IIS log files to remove internal IPs

Report •


#1
April 29, 2009 at 04:52:23
I think you'll probably want a real program to get any speed. But if you're stuck with a bat here's a couple observations.

The first FOR CALLs the other bat for every line in the log. A recipe for overload.

IF ERRORLEVEL 0 GOTO

is meaningless because it's ALWAYS 0 or more.


=====================================
If at first you don't succeed, you're about average.

M2


Report •

#2
April 29, 2009 at 05:41:25
Thanks for your input. I'll have a review of my options. Maybe a BATCH file isn't the best solution.

Report •

#3
April 29, 2009 at 06:24:07
Try this:

SET yy=%date:~8,4%
SET mm=%date:~3,2%
SET dd=%date:~0,2%
SET N=%yy%%mm%%dd%

(FOR /F "TOKENS=* SKIP=3 DELIMS=\n" %%a IN (^%N%.log) DO (
   ECHO %%a|find /v "192.168"
)) > d:\log_cleaned.txt

By the way, I'm not sure what the delims=\n is for, or what ^ is doing in ^%N% in your code. But I've left them as they were.


Report •

Related Solutions

#4
April 29, 2009 at 07:40:17
I don't figure delims=\n but I don't have the log.

If you need to skip 3 lines and get out all the lines not containing 192.168 try this:

=======================
@echo off > log_temp.tmp & setLocal EnableDelayedExpansion

FOR /F "TOKENS=* SKIP=3 DELIMS=\n" %%a IN (the.log) DO (
ECHO %%a >> log_temp.tmp
)

FIND /v "192.168" < log_temp.tmp > d:\log_cleaned.txt


=====================================
If at first you don't succeed, you're about average.

M2


Report •

#5
April 29, 2009 at 18:57:08
tedious. use logparser.

Unix Win32 tools | Gawk for Windows


Report •


Ask Question