I am trying to do something for each token in a line of a text file but I have an unknown number of tokens. The tokens are separated by "/". How do I process all valid tokens?

for /F "tokens=<what goes here?> delims=/" %%A in (List.txt) do (
<do something for each valid token>
)

Thanks.

:: process each token

@echo off
setLocal EnableDelayedExpansion

for /f "tokens=* delims=/" %%a in (myfile) do (
call :sub1 %%a
)
goto :eof

:sub1

:loop
echo do %1
shift
if not %1'==' goto loop
goto :eof

=====================================
If at first you don't succeed, you're about average.

M2

set line=one/two/three
:processToken
for /f "tokens=1* delims=/" %%a in ("%line%") do (
echo Got one token: %%a
set line=%%b
)
if not "%line%" == "" goto :processToken

To Mechanix2Go, when calling a 'function' ..i.e. call :sub1 %%a the function uses the space as a delim instead of keeping the same delim as the for loop. How do I make it treat the parameters (%1,%2,...%9) with delim "/" ?

Hi Daniel,

You're absolutely right. Why it behaves that way is beyond me. Maybe somebody can enlighten us.

Try this:

::============================
@echo off
setLocal EnableDelayedExpansion

for /f "tokens=* delims= " %%a in (myfile) do (
set str=%%a
set str=!str:/= !
call :sub1 !str!
)
goto :eof

:sub1

:loop
echo do %1
shift
if not %1'==' goto loop
goto :eof

=====================================
If at first you don't succeed, you're about average.

M2

So basically my current solution combines both of your responses..

@echo off
setLocal EnableDelayedExpansion

for /f "tokens=* delims=/" %%a in (List.txt) do (
set line=%%a
call :sub1 line
)
goto :eof

:sub1

:processToken
for /f "tokens=1* delims=/" %%a in ("%line%") do (
echo Got one token: %%a
set line=%%b
)
if not "%line%" == "" goto :processToken
goto :eof

First part to grab the line, 2nd part to process the line.. is this the most efficient way? I am operating on a large text file so efficiency is somewhat important. Thanks for both your help.

I see what you are trying to do with replacing the / with the space except that my tokens are allowed to have spaces and thus what should be one token is split into multiple tokens. Do you see my previous post as a decent solution?

Yeah, I think it will work but this behavior of / as delim is very troubling. I'm wondering what else is lurking in the weeds, waiting to not work at a really inconvenient time.

=====================================
If at first you don't succeed, you're about average.

M2

If, as you say, you've got a large file and efficiency is important, consider using a text processing utility such as awk, which can process the file much faster than a batch file.

Hi klint,

Any idea why / does not work as delim?

=====================================
If at first you don't succeed, you're about average.

M2

M2, I think it's just that a subroutine's arguments are processed the same way as a batch file's command-line arguments: the delimiters are space, tab and semicolon. When you specify for /l "delims=/", that only affects the loop variables (so that %%a is the first token delimited by /, %%b the next, etc.)

Notice that in the first section, it spits out the tokens as expected.

2nd section it echos the /, wich is the delim.

What's going on?

::==================
@echo off > newfile
setLocal EnableDelayedExpansion

for /f "tokens=1-5 delims=/" %%a in (myfile) do (
echo %%a
echo %%b
echo %%c
)

for /f "tokens=* delims=/" %%a in (myfile) do (
echo %%a
)

=====================================
If at first you don't succeed, you're about average.

M2

It's because of the tokens=*. Here's a snippet from the for/? help text:

----
tokens=x,y,m-n - specifies which tokens from each line are to
be passed to the for body for each iteration.
This will cause additional variable names to
be allocated. The m-n form is a range,
specifying the mth through the nth tokens. If
the last character in the tokens= string is an
asterisk, then an additional variable is
allocated and receives the remaining text on
the line after the last token parsed.

----

So when you write tokens=* there is only one variable and it "receives the remaining text on the line" which is, in this case, the whole line.

Hi klint,

I guess you're right but it seems pretty lame to include delims.

I'm going to look at your first solution and see if I can get my head round it.

=====================================
If at first you don't succeed, you're about average.

M2

@echo off
for /f "tokens=* delims= " %%f in (myfile) do (
  set line=%%f
  call :processToken
  )
  goto :eof
:processToken
  for /f "tokens=1* delims=/" %%a in ("%line%") do (
  echo Got one token: %%a
  set line=%%b
  )
  if not "%line%" == "" goto :processToken
  goto :eof
::==============================================
Thanks to klint for bailing us out on this one.

=====================================
If at first you don't succeed, you're about average.

M2

Report Offensive Follow Up For Removal