Solved CMD script : avoiding many CMD calls

June 18, 2019 at 07:39:53
Specs: Windows 7, 8GB
I'm having the following challenge:

As part of a bigger conversion, I'm creating a temporary file that contains many calls to 1 specific script, each time with different parameters. Example:

call script.cmd parameterAAA
call script.cmd parameterBBB
call script.cmd parameterCCC
...

In this context, such a file can have up to 1 million of such lines. I already fine tuned script "script.cmd" to be running as fast as possible. What I am thinking is that making 1 million calls to a script, is something that on itself can be done faster. I presume that the "call to a CMD" function costs quite some time, also when it is doing that on repetition.

Note: technically it is working 100% OK.

The question I have is how this could possibly be improved in context of performance.


See More: CMD script : avoiding many CMD calls

Reply ↓  Report •

✔ Best Answer
June 21, 2019 at 03:46:45
Ok, I screwed up on my first post. BUT still, I think that "fm_1file.cmd" could incorporate a text-file, or a direct std-out as an outer loop, based on a text-file list of the items you want it to process. Since I don't know what fm_1file.cmd does with each entry/argument, I cannot say for sure. (no code for fm_1file.cmd). You would need to build the for-loop (based on: dir C:\* /S /B /A-D /ON ) to process directly from that, into in fm_1file.cmd:
for /f "tokens=*" %%a in ('dir C:\* /S /B /A-D /ON ') do (...)
then fm_1file.cmd would be self-contained and not multiply called.


#1
June 18, 2019 at 09:45:21
I don't believe CALL takes up a significant amount of time compared to other commands, and generally if you're using batch scripts, you've already lost on the performance front. Still, if you're sure CALL is holding you back and since you control both scripts, you can use the classic SHIFT GOTO loop setup. Just keep in mind the command line length. All one million lines can't be compressed in to one call.

CALL script.cmd parameterAAA parameterBBB parameterCCC

[Start of script.cmd]

:scriptStart

. . . . . . . . .

SHIFT
IF NOT "%~1"=="" GOTO scriptStart
[End of script.cmd]

How To Ask Questions The Smart Way


Reply ↓  Report •

#2
June 18, 2019 at 14:11:56
I agree performance will remain an issue using these scripts, but I need to see if I can make some improvement. I will actually create a separate thread to see how all of this can be done differently. So not just the action discussed here, but the broader concept of it.

But, returning to the initial question: I like that approach (the shift loop method), but the parameters are actually pretty long: 150 characters on average, per line. I'll consider it though.

message edited by Looge


Reply ↓  Report •

#3
June 18, 2019 at 19:16:10
as another alternative, you could either write the parameters out to a textfile, or filter the source material in such a way as to provide the parameters directly to the script. Then the script just uses "for /f ... in (xxx.txt) do...
Might help if you post your current code and a sample of your source-data, and any other supporting information/scripts/applications that are in the pipeline.
Ooops! well DUH. obviously this requires multiple calls... my bad!

message edited by nbrane


Reply ↓  Report •

Related Solutions

#4
June 21, 2019 at 00:29:59
That would indeed still require a call to a CMD file for each line ..

On your request, here is a more detailed example of the code that is being run:

So I create my DIRB file as follows:

dir C:\* /S /B /A-D /ON > %p_fm_dirb%


Then I use that file as input to create my temprorary batch file:
I'm using GNU's awk.exe

awk " { print \"call fm_1file.cmd \\\"\" $0 \"\\\"\" } " > %p_fm_tmp_cmd%

so the resulting file looks like:

(..)
call fm_1file.cmd "C:\Windows\AppCompat\Appraiser\APPRAISER_TelemetryBaseline_RS3.bin"
call fm_1file.cmd "C:\Windows\AppCompat\Appraiser\APPRAISER_TelemetryBaseline_RS4.bin"
call fm_1file.cmd "C:\Windows\AppCompat\Appraiser\APPRAISER_TelemetryBaseline_RS5.bin"
call fm_1file.cmd "C:\Windows\AppCompat\Appraiser\APPRAISER_TelemetryBaseline_TH2.bin"
call fm_1file.cmd "C:\Windows\AppCompat\Appraiser\APPRAISER_TelemetryBaseline_UNV.bin"
call fm_1file.cmd "C:\Windows\AppCompat\Appraiser\GatedDefaultCache.bin"
call fm_1file.cmd "C:\Windows\AppCompat\Appraiser\Telemetry\Appraiser.JSON"
call fm_1file.cmd "C:\Windows\AppCompat\Programs\Amcache.hve"
call fm_1file.cmd "C:\Windows\AppCompat\Programs\Amcache.hve.LOG1"
call fm_1file.cmd "C:\Windows\AppCompat\Programs\Amcache.hve.LOG2"
call fm_1file.cmd "C:\Windows\AppCompat\Programs\Amcache.hve.tmp.LOG1"
(..)


Reply ↓  Report •

#5
June 21, 2019 at 03:46:45
✔ Best Answer
Ok, I screwed up on my first post. BUT still, I think that "fm_1file.cmd" could incorporate a text-file, or a direct std-out as an outer loop, based on a text-file list of the items you want it to process. Since I don't know what fm_1file.cmd does with each entry/argument, I cannot say for sure. (no code for fm_1file.cmd). You would need to build the for-loop (based on: dir C:\* /S /B /A-D /ON ) to process directly from that, into in fm_1file.cmd:
for /f "tokens=*" %%a in ('dir C:\* /S /B /A-D /ON ') do (...)
then fm_1file.cmd would be self-contained and not multiply called.

Reply ↓  Report •

#6
June 21, 2019 at 04:12:43
I think I understand what you mean. I had already tested something similar, but without the FOR loop.

So if I get it right, something like this (and all in 1 CMD file): see below. A call is done, but it is a call to a function inside the same script. Actually, it is the full code of the script called fm_1file.cmd which is to be injected into this script.
( Script fm_1file.cmd doesn't really do anything special, it gathers some info about the file which is given as a parameter, like checking if it actually exists, but much more. I've tuned that one to run as fast as possible.)

---- script X.cmd
for /f "tokens=*" %%a in ('dir C:\* /S /B /A-D /ON ') do call :fm_routine %%a
goto :EOF

:fm_routine
< entire code of script fm_1file.cmd >
goto :EOF
---- end of script X.cmd


Reply ↓  Report •

#7
June 21, 2019 at 09:09:04
Going to be honest here, but this sounds like something that can be done faster in VBScript. Or PowerShell, assuming Win8+.

How To Ask Questions The Smart Way


Reply ↓  Report •

#8
June 21, 2019 at 21:31:30
That was my idea - containing the fm_1 code inside your envelope script. Usually, as Razor suggests, these other two options are faster and more efficient (although my win-7 seems to run vbscript and pwrshell OK and even XP handles vbscript OK but that depends on what level of operation is being attempted.)

Reply ↓  Report •

#9
June 21, 2019 at 22:04:32
Every version of Windows, starting with Win98, will run VBScript. That's the only scripting language that can make that claim. Even batch scripts as we know them didn't happen until WinNT 4, with differences between each version of Windows. VBScript's strongest point was in the WinXP era, and now features are slowly being rolled back to encourage people to move to PowerShell.

How To Ask Questions The Smart Way


Reply ↓  Report •

#10
June 22, 2019 at 00:45:33
You're right, it'll never be any fast this way. As stated above: I am going to start another thread to ask how I can do the full project in another way. The question will then assume Windows 10, as Windows 7 is going near its end.

Reply ↓  Report •

#11
June 22, 2019 at 00:48:21
I tried the changed code (all inside the same batch) ... but it is not faster, if anything. It seems that calling an actual CMD script isn't what is slowing it down, or not in the way I thought.

Reply ↓  Report •

#12
June 24, 2019 at 12:54:23
As stated : the question in this thread ( https://www.computing.net/answers/p... ) explains the context of what I'm doing. Obviously, I'm already trying something, which isn't really mentioned in that thread.

Reply ↓  Report •

#13
July 3, 2019 at 07:04:00
I found quite an improvement for the code as posted earlier. Initially, this wasn't delivering the expected gain in time:

---- script X.cmd
for /f "tokens=*" %%a in ('dir C:\* /S /B /A-D /ON ') do call :fm_routine %%a
goto :EOF

:fm_routine
< entire code of script fm_1file.cmd >
goto :EOF
---- end of script X.cmd

But, I realised that the "entire code of script fm_1file.cmd" is actually making a call, to yet another CMD script.
He would still be doing that CMD call, for every iteration of the FOR loop !
So, behold the script "my_sub_script.cmd" which is being called from fm_1file.cmd

I adapted my initial improvement to something like this:

---- script X.cmd
for /f "tokens=*" %%a in ('dir C:\* /S /B /A-D /ON ') do call :fm_routine %%a
goto :EOF

:fm_routine
< first section of code of script fm_1file.cmd >
call :my_sub_script some_parameters
< second section of code of script fm_1file.cmd >
goto :EOF

:my_sub_script
< entire code of script my_sub_script.cmd >
goto :EOF
---- end of script X.cmd


What I now have is that for all of the files in the FOR loop, each two calls are a call to a function inside this one script,
as opposed to two calls to a .CMD script on the file system. It's now processing about 2.5 the amount of data in the same time.

Additionally:
Script X.cmd is generated (by ... a script) and from the original copy of the source .CMD scripts, I also removed ALL
lines that start with "REM ", and ALL lines that are empty.
I'm not sure that actually helps a lot, but since I was copying source .CMD files anyway, I tried to do this also for
target of gaining time. These generated copies (inside X.CMD) are hard to read, but they technically work.


Reply ↓  Report •

Ask Question