Variable Memory Usage

October 14, 2011 at 20:36:50
Specs: Windows 7, 8GB
I have a batch script that continually adds one variable to another, and constantly overwrites the same ones over and over again. I am wondering, does it overwrite the variable value in RAM or does it leak and just keep on adding up? My batch script starts to chow down GBs of RAM after awhile. This is isn't a problem for me, but it may be for anyone who uses my script. It might be overly complex for batch, but its about the only language I partially know.

I'll try to go into it a bit further with examples. The batch scans a folder for RAR or ZIP files, and uses 7-zip to decompress and convert to 7z. It compares the files, deletes the larger one, records the space saved/preserved, and in the end outputs totals-files converted, files not converted, files that failed, space recovered, and space preserved. Now in the loop, it calls these functions to calculate sizes.

IF %%~zF LSS 2140000000 ( CALL:CalculateSize old "%%F" ) ELSE ( CALL:CalculateTruncatedSize old "%%F" )
IF %%~zG LSS 2140000000 ( CALL:CalculateSize szp "%%G" ) ELSE ( CALL:CalculateTruncatedSize szp "%%G" )
IF %%~zF LSS 2140000000 ( IF %%~zG LSS 2140000000 ( CALL:CalculateSaved "%%F","%%G",0 ) ELSE ( CALL:CalculateSaved "%%F","%%G",1 ) ) ELSE ( CALL:CalculateSaved "%%F","%%G",1 )

Files over 2GB will exceed the DOS 32-bit limit, which is the purpose for truncation. The last number in bytes is striped, then that number is converted to KB, and then multiplied by 10. Now this is where it might get a bit confusing on what I'm doing here at first glance, but it's actually rather simple. I'll add a (-) to each line with an explanation to guide through the process.

:CalculateSize (-) %~1 is used for variable prefix, %~2 for the file to be calculated.
SET bytes=%~z2 (-) Get the bytes of the archive.
SET /a %~1KB=bytes/1024 (-) Convert from bytes to kilobytes.
SET /a multKB=%~1KB*100 (-) Multiply by 100 to preserve 2 digits for MB decimal value.
SET /a multMB=multKB/1024 (-) Convert to MB, will be 2 digits too large.
SET %~1MB_S=%multMB:~0,-2% (-) Remove the last 2 digits for the whole MB value.
SET %~1MB_E=%multMB:~-2% (-) Extract the last 2 digits for the decimal value.
GOTO:EOF

:CalculateTruncatedSize (-) %~1 is used for variable prefix, %~2 for the file to be calculated.
SET bytes=%~z2 (-) Get the bytes of the archive .
SET bytes=%bytes:~0,-1% (-) File is over 2GB so truncate to avoid errors.
SET /a truncKB=%bytes%/1024 (-) Convert to kilobytes, 1 digit is lost.
SET /a %~1KB=truncKB*10 (-) Multiply by 10 to restore lost digit, a few KB are lost.
SET /a multKB=%~1KB*100 (-) Multiply by 100 to preserve 2 digits for MB decimal value.
SET /a multMB=multKB/1024 (-) Convert to MB, will be 2 digits too large.
SET %~1MB_S=%multMB:~0,-2% (-) Remove the last 2 digits for the whole MB value.
SET %~1MB_E=%multMB:~-2% (-) Extract the last 2 digits for the decimal value.
GOTO:EOF

:CalculateSaved (-) %~1 and %~2 are files to be calculated, %~3 argument is a switch for truncation.
SET OldSize=%~z1 (-) Get the bytes of the original archive .
SET SzpSize=%~z2 (-) Get the bytes of the 7zip archive .
SET /a savedKB=szpKB-oldKB (-) Calculate the space spaved in the conversion.
IF %savedKB% LSS 0 SET /a savedKB=savedKB*-1 (-) If negative, reverse the output.
SET /a t_szpMB=(szpMB_S*100)+szpMB_E (-) Get temporary value of 7z MB without decimal.
SET /a t_oldMB=(oldMB_S*100)+oldMB_E (-) Get temporary value of original MB without decimal.
SET /a t_savedMB=t_szpMB-t_oldMB (-) Get temporary value saved MB without decimal.
IF %t_savedMB% LSS 0 SET /a t_savedMB=t_savedMB*-1 (-) If negative, reverse the output.
SET savedMB_S=%t_savedMB:~0,-2% (-) Remove the last 2 digits for the whole MB value.
SET savedMB_E=%t_savedMB:~-2% (-) Extract the last 2 digits for the decimal value.
IF %~3 == 1 ( SET OldSize=%OldSize:~0,-1% & SET SzpSize=%SzpSize:~0,-1% ) (-) Parameter to truncate archive size if exceeding 2GB.
IF %SzpSize% GTR %OldSize% ( SET /a preservedKB=preservedKB+savedKB & SET /a preservedMB=preservedMB+t_savedMB ) (-) Save to preserved variable- MB without decimal.
IF %OldSize% GTR %SzpSize% ( SET /a totalsavedKB=totalsavedKB+savedKB & SET /a totalsavedMB=totalsavedMB+t_savedMB ) (-) Save to recovered variable- MB without decimal.
GOTO:EOF

I had to use 2 functions for calculations.. an argument wasn't working to truncate for some reason. This works, so I left it as this. Then, finally at the end of the batch (when it has found no more files to convert and the loop ends)...

SET preservedMB_S=%preservedMB:~0,-2% (-) Remove the last 2 digits for the whole MB value.
SET preservedMB_E=%preservedMB:~-2% (-) Extract the last 2 digits for the decimal value.
SET totalsavedMB_S=%totalsavedMB:~0,-2% (-) Remove the last 2 digits for the whole MB value.
SET totalsavedMB_E=%totalsavedMB:~-2% (-) Extract the last 2 digits for the decimal value.

This gives lots of variables to reference in the loop with EnableDelayedExpansion: SzpSize,OldSize,szpKB,oldKB,szpMB_S/szpMB_E,oldMB_S/oldMB_E,savedKB,savedMB_S/savedMB_E,preservedMB_S/preservedMB_E,totalsavedMB_S/totalsavedMB_E

Values in megabytes are referenced as: (%totalsavedMB_S%.%totalsavedMB_E% MB) would read something like (312.56 MB). The accuracy is perfect, everything works as intended. The problem is after so many archives are converted the RAM usage starts to soar. This isn't an issue when converting maybe 100 archives, but anything more and it climbs to an absurd number, even if they are kilobytes-megabytes in size... I'm constantly overwriting the same variables but it seems that the old value is still stored in RAM. In short, is there any way to reduce the amount of RAM being used? Can I free it up? What can I do?? I'm so close to perfection then this issue arises..

The problem may lie in preservedMB and totalsavedMB because I'm constantly adding to them, but these numbers don't really get that big.. at least not 3GB's in length... I could post the entire script but its a little over 200 lines, but if it's requested I will do so.


See More: Variable Memory Usage

Report •


#1
October 14, 2011 at 21:41:05
I think I solved my issue. It seems that batch does indeed have memory leaks if not properly prevented. The variable value isn't properly decremented in RAM before it is overwritten, it just assigns a new space every time a variable is reused. It's an ugly fix, but resetting every variable to 0 before it is reused worked. Every time the CalculateSave functions were called, RAM would climb almost 30MB each time! Now it's just a few kilobytes, but that's to be expected considering I'm constantly adding to a few of the variables.

Guess my long elegant post was pretty pointless. =/


Report •

#2
October 14, 2011 at 23:23:45
I didn't really force myself to understand all of the code, ect.

But as an aside you may be interested in this: http://judago.webs.com/byteconversi...

It's fairly old and I'm not sure how it fairs on win7...


Report •
Related Solutions


Ask Question