Solved Bat script to auto upload to gs

February 18, 2020 at 02:59:18
Specs: Windows 10
I'm trying to set up a system/bat script that reads all the folders in a network location drive. then save the list of folders. this is what i have so far: dir /D/Ad-s T:\ > N:\_backup_test\Log.txt
I then want to insert each line in the log into the pre-written script i already have. this is the part im having difficulties with

thanks


See More: Bat script to auto upload to gs


✔ Best Answer
March 2, 2020 at 21:45:30
As far as I can tell, this might approach:
::----- begin test script a.bat
@echo off & setlocal
for /f "tokens=*" %%a in (M:\_backup_test\Log.txt) do (
echo gsutil -m cp -n -r Z:\%%a gs://z_backup/Z$/Z_NEW/Z$/Z$/
gsutil -m cp -n -r Z:\%%a gs://z_backup/Z$/Z_NEW/Z$/Z$/
)
:: -------- end script
The echo is just to see what's being executed on what.


#1
February 18, 2020 at 03:24:51
the way it saves each filename is [filename1]
[filename2]
[filename3]

Report •

#2
February 18, 2020 at 21:51:23
@echo off & setlocal
pushd X:\Y
dir /b /s>N:\_backup_test\Log.txt

Report •

#3
March 2, 2020 at 04:04:31
That definitely helped. here is what i have now.

This makes a log.txt file with all the names of the folders.
dir /D/Ad-s Z:\ > M:\_backup_test\Log.txt

This reads the log.
type M:\_backup_test\Log.txt

this is the next bit that i need help on, everything outside of the () works but i need a way to insert the name inside the [ ] into the "Z:\ " one at a time. run then repeat till it gets the end.

gsutil -m cp -n -r Z:\(insert text in"[ ]") gs://z_backup/Z$/Z_NEW/Z$/Z$/

For example:
List:
[ABC101_Test1]
[ABC102_Test2]
[ABC10n_Testn]

gsutil -m cp -n -r Z:\ABC101_Test1 gs://z_backup/Z$/Z_NEW/Z$/Z$/
gsutil -m cp -n -r Z:\ABC102_Test2 gs://z_backup/Z$/Z_NEW/Z$/Z$/
gsutil -m cp -n -r Z:\ABC10n_Testn gs://z_backup/Z$/Z_NEW/Z$/Z$/


Report •

Related Solutions

#4
March 2, 2020 at 21:45:30
✔ Best Answer
As far as I can tell, this might approach:
::----- begin test script a.bat
@echo off & setlocal
for /f "tokens=*" %%a in (M:\_backup_test\Log.txt) do (
echo gsutil -m cp -n -r Z:\%%a gs://z_backup/Z$/Z_NEW/Z$/Z$/
gsutil -m cp -n -r Z:\%%a gs://z_backup/Z$/Z_NEW/Z$/Z$/
)
:: -------- end script
The echo is just to see what's being executed on what.

Report •

#5
March 30, 2020 at 04:33:26
This is the message i get, %%a was unexpected at this time. it doesnt appear to be reading the file M:\__Test\Test.txt

I have the list of files i want to upload but

C:\Users\matthew.r\AppData\Local\Google\Cloud SDK>@echo off & setlocal
for /f "tokens=*" %%a in (M:\__Test\Test.txt) do (
%%a was unexpected at this time.
echo gsutil -m cp -n -r X:\ %%a gs://t_drive_backup/T$/
gsutil -m cp -n -r X:\ %%a gs://t_drive_backup/T$/
gsutil -m cp -n -r X:\ %%a gs://t_drive_backup/T$/
Copying file://X:\ABB108_Mitral_Valve\Video\Heart_Blood_Cycle_Blood_Flow.mov [Content-Type=video/quicktime]...
Copying file://X:\ABB108_Mitral_Valve\Video\SC010_Body_intro.mov [Content-Type=video/quicktime]...
Copying file://X:\ABB108_Mitral_Valve\Video\SC005_MitClip_Intro_B.mov [Content-Type=video/quicktime]...
Resuming upload for file://X:\ABB108_Mitral_Valve\Video\SC005_MitClip_Intro_B.mov
==> NOTE: You are uploading one or more large file(s), which would run
significantly faster if you enable parallel composite uploads. This
feature can be enabled by editing the
"parallel_composite_upload_threshold" value in your .boto
configuration file. However, note that if you do this large files will
be uploaded as `composite objects
<https://cloud.google.com/storage/docs/composite-objects>`_,which
means that any user who downloads such objects will need to have a
compiled crcmod installed (see "gsutil help crcmod"). This is because
without a compiled crcmod, computing checksums on composite objects is
so slow that gsutil disables downloads of composite objects.


Report •

#6
March 30, 2020 at 23:01:26
Only thing so far is that space. Should not be there:
echo gsutil -m cp -n -r X:\ %%a gs://t_drive_backup/T$/
gsutil -m cp -n -r X:\ %%a gs://t_drive_backup/T$/

this,maybe:
echo gsutil -m cp -n -r X:\%%a gs://t_drive_backup/T$/
gsutil -m cp -n -r X:\%%a gs://t_drive_backup/T$/

But that shouldn't have caused problem. Still not sure. Needing copy/sample of:
M:\__Test\Test.txt
One line should suffice.
May need quotes around %%a if it has spaces or other disruptors:
gsutil -m cp -n -r X:"\%%a" gs://t_drive_backup/T$/
or use short-name notation:
gsutil -m cp -n -r X:\%%~sa gs://t_drive_backup/T$/

message edited by nbrane


Report •

#7
April 4, 2020 at 08:09:02
I run: dir /O-D X:\ > M:\__Test\Test.txt
which reads X and gives me a list of the folder names in the drive
ive been using these folder names to keep it as simple as possible:
ACB113_XIE
BIA126_SMR
MBE128_Cold
RLS107_Imsom
written out in M:\__Test\Test.txt

I keep getting this: %%a was unexpected at this time
A common error i seem to be getting. the Google Cloud SDK Shell doesnt seem to understand the meaning of %%a even with "" around it.

Sorry this is such a pain btw, ive been working on this for a while and nothing ive tried seems to work


Report •

#8
April 4, 2020 at 20:21:43
If you're running from the command-prompt, and not from a batch-script, you will get this error. ("%%a not expected"). Execution from the command-prompt direct requires only ONE percent. From a batch-script requires two percent-signs. Post #4 was a script, so it had %%a. If you're testing from cmd direct,, the %%a will be unrecognized.
f/e, at C:> try typing "for /f %%a in ("xxx") do echo %%a
at the command prompt, and you will see "%%a was unexpected..."
put it into a script and it should run with no error. Aside from that, if running from script, try NOT turning echo off, and see what the commands look like.
Here's a "bare-bones" "from-scratch" test script:
@ECHO OFF & SETLOCAL
rem this should get you a list of bare-format dir.names from root of T:
DIR /B /AD T:\ > test
rem ---------- end part 1 script
into a local file. You've supposedly done this as`N:\_backup_test\Log.txt' Let'
use "test" to keep things simple. "test" is obviously wherever you are at execution.
ok, test looks like:
ABC101_Test1
ABC102_Test2
ABC10n_Testn

Part two simply reads this file and injects each line into the gsutil utility:
::========= begin SCRIPT part-2.bat
@ECHO OFF & SETLOCAL
FOR /F "TOKENS=*" %%a IN (TEST) DO (
echo gsutil -m cp -n -r Z:\%%a gs://z_backup/Z$/Z_NEW/Z$/Z$/
ECHO DOES THIS LOOK RIGHT?
PAUSE
gsutil -m cp -n -r Z:\%%a gs://z_backup/Z$/Z_NEW/Z$/Z$/
)
::========= end SCRIPT "part-2.bat"
That's about all I can do at this point. Remember, %%a HAS to be INSIDE a script to
be recognized.


Report •

#9
April 6, 2020 at 07:34:05
Oh thats brilliant, thank you. For the first part i was wondering is there a way to limit the list to number of result or but a date. i was looking around and i found this:

@echo off
set "file=M:\__Test\Test.txt"
set c=0
dir X:\ /b /o:-d /-p > "%file%"

for /f "delims=" %%a in ('type "%file%"') do call :next "%%a"
pause
goto :EOF

:next
set /a c=c+1
if %c% GTR 10 goto :EOF
echo %1

..............................

Unfortunately this doesnt edit the test.txt file to the same limit, in this case being 10. I was wondering if you know a solution to this

thank you


Report •

#10
April 6, 2020 at 21:09:51
Just run the above batch piped out to another file. F/e, if above is saved as "test.bat":
test > "M:\__Test\Test10.txt". 'TEST10.TXT' will now have the first 10 lines of test.

Report •

#11
April 7, 2020 at 00:28:41
i want it to be able to write over itself everytime its ran and not have to create a new file each time

Report •

#12
April 7, 2020 at 22:46:09
ok, to cut down on steps, this might work. Capture top ten date-sorted filenames from X:\ into M:\__Test\Test.txt
:: begin batchscript test.bat

@echo off & setlocal
set file="M:\__Test\Test.txt"
(for /f "tokens=1* delims=[]" %%a in ('dir /b /o-d X:\ ^|find /n /v ""') do (
if %%a gtr 10 goto :eof
echo %%b
)
) > %file%

:: END SCRIPT. more or less tested on random directory. Hope this helps.

message edited by nbrane


Report •

Ask Question