FTP reverse multiple download to upload

September 2, 2009 at 03:31:13
Specs: Windows 2000 NT/ 2k3 /PX, 2GHz / 2GB
Hi guys (and gals),

I'm stuck with the following:
I need to create a script to upload multiple unique files to an FTP server. I while ago I've found a script to do just the reverse: http://www.dostips.com/DtTipsFtpBat...
What this script does is it downloads unique files from an FTP server to the local directory. I need the reverse- a script to upload files (multiple files) that are not yet on the FTP server.
However, I cannot figure out how to reverse the script found to achieve this.
Hilfe!


See More: FTP reverse multiple download to upload

Report •


#1
September 2, 2009 at 04:33:47
If you were to let us see the script you are using maybe someone could tell you what needs changing. It is propably just a single line that needs to be changed.

Stuart


Report •

#2
September 2, 2009 at 04:47:12
My bad. The script is called:
DOS Batch FTP - Get New Files Only - Ftp script to download only files that don`t exist in local folder, i.e. to avoid overwrite I thought i got myself link to that specific post, but apparently not. Sorry- a bit stressed :)

No changes done yet, cause I'm stuck at the logic level. Basically how I understand it is:
The script when executed gets ls of the directory on the FTP server and compares output files' list with the file list from the directory the script runs from. Then by executing Do if not exist created a list of mget commands to download files that are not yet locally.
But I want to reverse the process - and mput only files that are not yet on FTP. Hope it clear things a bit :)


Report •

#3
September 2, 2009 at 06:08:46
Get the files to send into FILELIST and try this:

=============================
@echo off > scr.ftp & setLocal EnableDELAYedExpansion

>> scr.ftp echo o mysite.com
>> scr.ftp echo username
>> scr.ftp echo password
>> scr.ftp echo bin
for /f "tokens=* delims= " %%a in (filelist) do (
>> scr.ftp echo put %%a
)
>> scr.ftp echo bye

ftp -s:scr.ftp


=====================================
If at first you don't succeed, you're about average.

M2


Report •

Related Solutions

#4
September 2, 2009 at 06:57:37
Hi M2,
Thanks for the reply.
A stupid question first off:
Would I be able to use these commands to connect to SFTP? I haven't got a test machine yet so i can't test them myself (they'd work on good old FTP).

The problem is that I haven't the faintest clue how to compare two text files in batch and spit out the output to a third. Hence I've tried using the example from link to reverse their logic.
The text files could have the following structure:

====file1.txt====
line 1
line 2
line 3
=============

====file2.txt====
line 1
extraline
line 3
extraline
=============


Report •

#5
September 2, 2009 at 07:38:41
[1] no idea about SFTP

[2] What's up with comparing file contents?


=====================================
If at first you don't succeed, you're about average.

M2


Report •

#6
September 2, 2009 at 08:56:08
[1] pity. I'll see what will happen once I get that test machine.

[2] Get the files to send into FILELIST and try this:
The thing about it is that I don't know specifically what file will have to be transferred across from local machine to FTP server. Hence I thought that you're suggesting to get output dir /b of local directory and remote ls, compare them, spit the differences out and feed it back to uploading script to create (m)put commands, which will upload files that are not yet stored remotely.

Looks like I'm close to completion anyway and I'd be happy to post it somewhere for others to see.
Once I'm done that is. And since I'm new here-is there a section where I could post it? This board looks more like Q&A forum to me.
Anyway, thanks for the help. Much appreciated :)


Report •

#7
September 2, 2009 at 09:30:59
Let us know how it goes.


=====================================
If at first you don't succeed, you're about average.

M2


Report •

#8
September 4, 2009 at 01:56:37
Hallelujah, seems like I managed it-somehow. The solution is elaborate and involves 2 .bat-for executing FTP commands and other functions, and 2 .vbs for text formatting. (if anyone is going to ask why didn't I start with VBScript initially the answer is this: I've tried and failed miserably).
I'll post it when I clean it up.

Report •

#9
September 4, 2009 at 03:54:35
Mother-file. Starting point for the whole "program".

=======mycopy.bat=======

@echo off

dir /B /ad "C:\symmetrix" >"%temp%\symmList.txt"

set "symmlist=%temp%\symmList.txt"
::change these paths to match your environment
set "origPath=H:\tests\FTPmput\FTP_GetNewFilesOnly.bat"
set "splitpath=H:\tests\FTPmput\split.vbs"
set "comparePath=H:\tests\FTPmput\compTxtOutput.vbs"

setlocal enabledelayedexpansion

For /F %%i in (%symmList%) do (

set "pathL=C:\Symmetrix\%%i\daily"
set "pathR=www\Symmetrix\%%i\daily"
set "filename=getFiles%%i.bat"

Call:Unique dateStamp

copy /Y %origPath% "!pathL!\getFiles%%i.bat"
copy /Y %splitPath% "%temp%\split.vbs"
copy /Y %comparePath% "%temp%\compareText.vbs"

Call:logDir %%i

pushd !pathL!
Start /d !pathL! /b /w cmd /c !filename! !pathl! !pathR! >"%SystemDrive%\SymmTransferLogs\%%i\%%i_!datestamp!.log"

del "getFiles%%i.bat"
)
Call:clearVar

goto:eof
exit /b

:clearVar
::clears environmental variables as a precaution
::deletes temporary files
del "%temp%\symmList.txt"
del "%temp%\split.vbs"
del "%temp%\compareText.vbs"
set "origPath="
set "symmList="
set "dateStamp="
del "%temp%\dirLocalOutput.txt"
del "%temp%\dirRemoteOutput.txt"
del "%temp%\differencelog.txt"

exit /b

:Unique
::returns a unique string based on a date-time-stamp
::[OUT] dateStamp in YYYYMMDDhhmm format

SETLOCAL
for /f "skip=1 tokens=2-4 delims=(-)" %%a in ('"echo.|date"') do (
for /f "tokens=1-3 delims=/.- " %%A in ("%date:* =%") do (
set %%a=%%A&set %%b=%%B&set %%c=%%C))
for /f "tokens=1-2 delims=:. " %%A in ("%time: =0%") do @set UNIQUE=%yy%%mm%%dd%%%A%%B%
ENDLOCAL & IF "%~1" NEQ "" (SET %~1=%UNIQUE%)
EXIT /b


:logDir
::checks whether directories to store log files from transfer exist
::if not the structure is created
::requires 'enableextensions' option must be turned on for the following to execute,
::which is available in Windows 2000 and later NT OS. 'On' by default.

setlocal ENABLEEXTENSIONS

set "j=%~1"

if not exist "%SystemDrive%\SymmTransferLogs\!j!\nul" (mkdir "%SystemDrive%\SymmTransferLogs\!j!")
)

exit /b
========end of mycopy.bat=======


Report •

#10
September 4, 2009 at 03:57:20
File that does the FTP transfer.


=======FTP_GetNewFilesOnly.bat=======
@Echo Off

REM -- Define File Filter, i.e. files with extension .btp
Set FindStrArgs=/E /C:".btp"

REM -- Extract Ftp Script to create List of Files
Set "FtpCommand=ls"
Set "lcdFtp=%~1"
Set "cdFtp=%~2"

set "var2=%temp%\dirLocalOutput.txt"
set "var4=%temp%\dirRemoteOutput.txt"
set "diffLog=%temp%\differenceLog.txt"

Call:localDir %lcdFtp% %var2%

Call:extractFileSection "[Ftp Script 1]" "-">"%temp%\%~n0.ftp"

REM -- Execute Ftp Script, collect File Names from FTP server
Set "FileList="
For /F "Delims=" %%A In ('"Ftp -v -i -s:"%temp%\%~n0.ftp"|Findstr %FindStrArgs%"') Do (
Call Set "FileList=%%FileList%%%%A,")
::exports list of files stored on FTP to a txt file
Echo.%FileList% >%var4%

Call:compareText

REM -- Extract Ftp Script to upload files that don't exist in remote folder
Set "FtpCommand=mput"

set "FileList="
For /F "Delims=" %%B In (%diffLog%) Do (
Call Set "FileList=%%FileList%% "%%B"")

For %%B In (%FileList%) Do Call Set "FtpCommand=%%FtpCommand%% %%~B"

Call:extractFileSection "[Ftp Script 1]" "-">"%temp%\%~n0.ftp"

For %%C In (%FtpCommand%) Do Echo.%%C

REM -- Execute Ftp Script, download files
:: carries out the mput command previously saved to .ftp file
ftp -i -s:"%temp%\%~n0.ftp"

Call:clearEnvVar
GOTO:EOF
Exit /b

:extractFileSection StartMark EndMark FileName
rem --extract a section of file that is defined by a start and end mark
:: -- [IN] StartMark - start mark, use '...:S' mark to allow variable substitution
:: -- [IN,OPT] EndMark - optional end mark, default is first empty line
:: -- [IN,OPT] FileName - optional source file, default is THIS file
SETLOCAL
set "smk=%~1"
set "emk=%~2"
set "src=%~3"
set "bExtr="
set "bSubs="
if "%src%"=="" set src=%~f0& rem if no source file then assume THIS file
for /f "tokens=1,* delims=]" %%A in ('find /n /v "" "%src%"') do (
if /i "%%B"=="%emk%" set "bExtr="&set "bSubs="
if defined bExtr if defined bSubs (call echo.%%B) ELSE (echo.%%B)
if /i "%%B"=="%smk%" set "bExtr=Y"
if /i "%%B"=="%smk%:S" set "bExtr=Y"&set "bSubs=Y"
)
EXIT /b

:clearEnvVar
:: clears environmental variables that were set during program's life
Del "%temp%\%~n0.ftp"
set "FindStrArgs="
set "FileList="
set "FtpCommand="
EXIT /b

:localDir
:: creates a list of files currently stored on local machine and saves it into a .txt file
:: location of local directory to be checked is stored inside
SETLOCAL
set "loc=%~1%"
set "out=%~2%"
dir %loc% /B > %out%
EXIT /b

:compareText
::calls vbscript that compares two lists and outputs the difference
setlocal
set "arg1="%TEMP%\dirLocalOutput.txt""
set "arg2="%TEMP%\dirRemoteOutput.txt""
:: Echo.%arg1% %arg2%

findstr /b /v /c:"getFiles" "%temp%\dirLocalOutput.txt" >"%temp%\test.txt"
del "%temp%\dirLocalOutput.txt"
ren "%temp%\test.txt" "dirLocalOutput.txt"

Wscript "%TEMP%\split.vbs"
del "%temp%\dirRemoteOutput.txt"
ren "%temp%\testW.txt" "dirRemoteOutput.txt"

Wscript "%TEMP%\compareText.vbs" %arg1% %arg2%

exit /b

::internal FTP script that sends FTP commands to the server. Fails if @Echo On
[Ftp Script 1]:S
open your.ftpserver.com
yourusername
yourpassword
verbose off
cd %cdFtp%
lcd %lcdFtp%
binary
%FtpCommand%
disconnect
bye
=======end of FTP_GetNewFilesOnly.bat=======


Report •

#11
September 4, 2009 at 04:03:53
VBScript file that compares two text files generated by FTP_GetNewFilesOnly.bat. Both text files contain lists of files stored-one locally other remotely. After comparing them it redirects lines that don't match to another text file called differenceLog.txt which is fed back into FTP_GetNewFilesOnly.bat.
Note for others: Problem arises when the changes in text files occur somewhere in the middle as opposed to end of one of the files. Then script grabs all of the lines after where the change occurred regardless of anything.
Works for me though.

=======compareTxtOutput.vbs=======

Set objArgs = Wscript.Arguments

Set WshShell = WScript.CreateObject("WScript.Shell")
Set WshProccessEnv = WshShell.Environment("Process")

dim diffPath
diffPath=WshShell.ExpandEnvironmentStrings("%TEMP%")

Set fso = CreateObject("Scripting.FileSystemObject")
Dim pathLog, f

pathLog=diffPath&"\differenceLog.txt"
fso.CreateTextFile pathLog,True
set f= fso.OpenTextFile(pathLog, 8)

Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objFile1 = objFSO.GetFile(objArgs(0))
Set objFile2 = objFSO.GetFile(objArgs(1))

Set FileStream1 = objFSO.OpenTextFile(objFile1.Path)
Set FileStream2 = objFSO.OpenTextFile(objFile2.Path)

While (Not FileStream1.AtEndOfStream) Or (Not FileStream2.AtEndOfStream)
if Not FileStream1.AtEndOfStream Then
sLine1 = FileStream1.ReadLine()
Else
sLine1 = ""
End If
If Not FileStream2.AtEndOfStream Then
sLine2 = FileStream2.ReadLine()
Else
sLine2 = ""
End If
If sLine1 <> sLine2 Then

f.writeLine(sLine1 & chr(10) & chr(13))

End If
Wend
=======End of compTxtOutput.vbs=======


Report •

#12
September 4, 2009 at 04:11:59
Last but not least, file splitting the output from dirRemoteLog.txt file (which basically contains the list of files stored remotely). Split occurs from following format
filename,filename2,filename3,
to

filename
filename2
filename3

Notice that environmental variables used are case sensitive (or so I read, haven't checked). As script uses RegExp (regular expression) object to change delimiter sign or expression you need to change objRE.Pattern line (in bold) to match your needs.

=======split.vbs=======
'vbs to split string

Set WshShell = WScript.CreateObject("WScript.Shell")
Set WshProccessEnv = WshShell.Environment("Process")

dim diffPath,s, MyArray
diffPath=WshShell.ExpandEnvironmentStrings("%TEMP%")

Set fso=CreateObject("Scripting.FileSystemObject")
Set fso2=CreateObject("Scripting.FileSystemObject")

'1=forReading, 2=forWriting, 8=forAppend
pathRead=diffPath&"\"&"dirRemoteOutput.txt"
pathWrite=diffPath&"\"&"testW.txt"
Set f=fso.OpenTextFile(pathRead, 1)
set createF=fso2.CreateTextFile(pathWrite, True)

do while not f.atendofstream
s=f.readline 'this is your line from text file

arrTest = SplitAdv(s)

loop
for i=0 to (Ubound(arrTest)-1)
'Wscript.echo Ubound(arrTest)
createF.writeLine(arrTest(i))
Next

'splits string read from text file at delimiter
'to change delimiter replace coma (,) inside quotes at objRE.Pattern = "" line
Function SplitAdv(strInput)
Dim objRE
Set objRE = CreateObject("VBScript.RegExp")
objRE.IgnoreCase = true
objRE.Global = true
objRE.Pattern = ","
SplitAdv = Split(objRE.Replace(strInput, "\b"), "\b")
End Function
=======split.vbs=======

Author's note
Pheew...that's it. If anyone needs more information (like scenario settings for which this thing was created for, parameters, names, etc.) give me a shout and I will try my best to accommodate any questions.
Cheers


Report •

#13
September 4, 2009 at 06:28:44
I'm getting hyper active today, but hey I might be getting paid for it one day :)
Use the file below instead of compTxtOutput.vbs if your input files have differ somewhere in the body at not at the end like in this situations:
File1
Line1
Line2
Line3
Line4
End of File1

File2
Line1
Line4
End of File2

Note that when executing from command line or other script path to to-be-compared files have to be provided on line calling the script.

=======mycompare.vbs=======
Set objArgs = Wscript.Arguments

Set WshShell = WScript.CreateObject("WScript.Shell")
Set WshProccessEnv = WshShell.Environment("Process")

dim diffPath
diffPath=WshShell.ExpandEnvironmentStrings("%TEMP%")

Set fso = CreateObject("Scripting.FileSystemObject")
Dim pathLog, f, match

pathLog=diffPath&"\differenceLogTest.txt"
fso.CreateTextFile pathLog,True
set f= fso.OpenTextFile(pathLog, 8)

Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objFile1 = objFSO.GetFile(objArgs(0))
Set objFile2 = objFSO.GetFile(objArgs(1))

Set FileStream1 = objFSO.OpenTextFile(objFile1.Path)
Set FileStream2 = objFSO.OpenTextFile(objFile2.Path)

While (not FileStream1.AtEndOfStream) or (not FileStream2.AtEndOfStream)
line2=FileStream2.ReadLine()

Do While not (FileStream1.AtEndOfStream)
line1=FileStream1.ReadLine()
'Wscript.Echo line1 &line2
If line1=line2 Then
match="True"
'Wscript.echo match
Exit Do
Else
match="False"

End if

If match="False" Then
f.writeLine(line1)
'Wscript.Echo "Writing to file" &line1
End if
Loop


Wend

=======end of my compare.vbs =======

Worked like a charm for me in tests. I'll give it more testing and update if any problems are found.


Report •


Ask Question