Computing.Net > Forums > Programming > ftp batch to download newest files

Computing.Net: Over 1,000,000 posts about all things technology related! Over 90% answered within 24 hours! Click here to sign up now, it's free!

ftp batch to download newest files

Reply to Message Icon

Original Message
Name: goompa
Date: November 12, 2008 at 17:29:18 Pacific
Subject: ftp batch to download newest files
OS: xp
CPU/Ram: 2gig
Manufacturer/Model: acer travelmate 4060
Comment:

Hi there, M2 is a legend!
I've been searching the posts for what would probably be a batch file that would replace an automatic download ftp client.

Basic process:
log in to ftp server
check for new files (against what has been downloaded already)
download new files (including newer versions of files already downloaded)
Log off
format c:\ (maybe not but just making sure you were watching)

I have not found any specific options with mget concerning synchronizing of folders etc.

Thanks muchly.


Report Offensive Message For Removal


Response Number 1
Name: Holla
Date: November 12, 2008 at 21:14:48 Pacific
Reply:

goompa,

I endorse your statements about M2!

comparing files across ftp is possible, but cumbersome. You have to ftp, list the directory, redirect the entire output to a file, then parse this output and interpret the dates, then compare it with the dates of local file, if you can get this done, I say you get a PhD in batch scripting :-).

M2 might still be able stun us.

Having said that, if you can find a tool to map to ftp drive, then the scripting becomes simple. One tool that does this is netdrive
http://www.divshare.com/download/20...
You have to buy this as per Novell policy.
It might work for you, it may not.

Also, if the FTP server has NetBUI interface(if it is on a Windows machine), mapping may be easy.

Hope I made sense.

--
Holla.


Report Offensive Follow Up For Removal

Response Number 2
Name: Mechanix2Go
Date: November 13, 2008 at 04:27:37 Pacific
Reply:

The script below will get a list of files and spit out:

filename.ext year mon date

It makes a few shakey assumptions. One is that the output of ls -l looks pretty much like this:

drwxr-xr-x 2 ftpuser ftpusers 1088 Jun 6 2005 CMhouse
-rw-r--r-- 1 ftpuser ftpusers 0 Jul 2 2003 Dining.html
-rw-r--r-- 1 ftpuser ftpusers 30804 Jul 2 2003 Lodging.html
-rw-r--r-- 1 ftpuser ftpusers 32796 Jun 26 2004 Orchid4.gif

The next hump to get over is making the date layouts conform.

=========================================
@echo off
setLocal EnableDelayedExpansion

> ft.do echo o yoursite.com
>> ft.do echo username
>> ft.do echo password
>> ft.do echo bin
>> ft.do echo cd mystuff
>> ft.do echo ls -l TLIST
>> ft.do echo bye

ftp -s:ft.do
del ft.do

findstr /b /v "d" TLIST > svrlist

for /f "tokens=6-9 delims= " %%a in (svrlist) do (
echo %%d %%c %%a %%b
)


=====================================
If at first you don't succeed, you're about average.

M2


Report Offensive Follow Up For Removal

Response Number 3
Name: goompa
Date: November 13, 2008 at 12:40:31 Pacific
Reply:

Um M2, did you invent DOS and then Gates looked over your shoulder with a note pad when you wern't looking?


Report Offensive Follow Up For Removal

Response Number 4
Name: goompa
Date: November 13, 2008 at 12:47:44 Pacific
Reply:

In the line "> ft.do echo o yoursite.com" is the extra "o" supposed to be there or is it a typ"o"? :)


Report Offensive Follow Up For Removal

Response Number 5
Name: goompa
Date: November 13, 2008 at 13:41:45 Pacific
Reply:

M2, I need to understand your code if I am to apply it so if you look through my comments in your code please correct them where needed. Thanks.

@echo off
setLocal EnableDelayedExpansion

::This builds ft.do a script for ftp to execute
> ft.do echo o yoursite.com
>> ft.do echo username
>> ft.do echo password
>> ft.do echo bin
>> ft.do echo cd mystuff
::This builds a file called TLIST which is a file list of the current remote folder.
::I can't find out what "-l" does.
>> ft.do echo ls -l TLIST
>> ft.do echo bye
::This executes the above then logs off
ftp -s:ft.do
del ft.do

::This is quite pretty this part, the "/b" m atches pattern if at the beginning of a line and the "/v" prints only lines that do not contain a match.
::It then takes that output and puts it into "svrlist"
::The question I do have, what is this comparing to?
findstr /b /v "d" TLIST > svrlist

::These lines confuse the living daylights out of me so I'll leave this to you.
for /f "tokens=6-9 delims= " %%a in (svrlist) do (
echo %%d %%c %%a %%b
)

So I'm gathering that you have provided me the key. I'm assuming that I'll have to hunt around and do a similar thing for the local directory.
My main point of confusion is where does the local get compared with the remote?

Thanks.


Report Offensive Follow Up For Removal


Response Number 6
Name: Mechanix2Go
Date: November 13, 2008 at 17:52:47 Pacific
Reply:

'::I can't find out what "-l" does.'
The usual directory lister in unix is ls ; the -l option makes it list sizes, dates, owners etc, as well as filenames.
--------------------
'findstr /b /v "d" TLIST > svrlist'

Notice that the first entry in the TLIST starts with drw which means it's a directory. We don't want thise; just files.
----------
'::These lines confuse the living daylights out of me so I'll leave this to
you.
for /f "tokens=6-9 delims= " %%a in (svrlist) do (
echo %%d %%c %%a %%b
)'

The TOKENS are the 'chunks' of chars in a line. In this case separated by spaces.

========================================
Seems like you've got a pretty good handle on this. I'm willing to stick with it if you are.

[1\ yes, the o before sitename belongs there.

[2] You need to run the basic script for your ftp site to see if it conforms. If it doesn't, we can work with it.

[3] Run this little bat so we can see hoe your rig puts out the file date:
==============================
@echo off
setLocal EnableDelayedExpansion

for /f "tokens=* delims= " %%a in ('dir/b/a-d') do (
echo %%~Ta


=====================================
If at first you don't succeed, you're about average.

M2


Report Offensive Follow Up For Removal

Response Number 7
Name: goompa
Date: November 13, 2008 at 18:54:18 Pacific
Reply:

Hey M2, here is the output of the svrlist file without running the little batch file you included at the bottom of the message.

test20081113.mp3
test20081111.mp3
test20081110.mp3
test20081119.mp3
test20081112.mp3
test20081114.mp3
test20081118.mp3
test20081117.mp3
test20081120.mp3
test20081121.mp3

Here is the TLIST

test20081113.mp3
test20081111.mp3
test20081110.mp3
test20081119.mp3
test20081112.mp3
test20081114.mp3
test20081118.mp3
test20081117.mp3
test20081120.mp3
test20081121.mp3

I'll be back with the output of the little batch file in a tick


Report Offensive Follow Up For Removal

Response Number 8
Name: goompa
Date: November 13, 2008 at 18:57:00 Pacific
Reply:

Do you know that you've pretty much made a $3500 radio station package redundant in two different countries?


Report Offensive Follow Up For Removal

Response Number 9
Name: goompa
Date: November 13, 2008 at 19:06:18 Pacific
Reply:

I changed your little bat slightly:
@echo off
setLocal EnableDelayedExpansion

for /f "tokens=* delims= " %%a in ('dir/b/a-d') do (
echo %%~Ta > littlebatout.txt
)
pause
echo done

An this was it's output, I'm assuming because it's running on my desktop this data is from the files or directories residing there. I'm guessing you are looking to get this data from both ends and compare them?


Your move. :) Thanks again.

14/11/2008 01:38 p.m.
24/09/2008 01:48 p.m.
29/10/2008 12:28 p.m.
16/10/2008 10:35 a.m.
24/10/2008 05:11 p.m.
14/11/2008 10:17 a.m.
14/11/2008 10:17 a.m.
23/10/2008 01:20 p.m.
23/10/2008 01:33 p.m.
06/11/2008 04:21 p.m.
06/11/2008 04:21 p.m.
14/11/2008 04:02 p.m.
14/11/2008 04:02 p.m.
04/10/2007 11:54 a.m.
18/09/2008 04:33 p.m.
25/09/2008 02:10 p.m.
14/10/2008 11:53 a.m.
14/10/2008 11:54 a.m.
09/10/2008 05:28 p.m.
18/09/2008 05:19 p.m.
31/07/2008 06:05 p.m.
17/06/2008 06:25 p.m.
14/11/2008 10:17 a.m.
14/11/2008 10:17 a.m.
01/08/2008 12:29 p.m.
14/11/2008 01:38 p.m.


Report Offensive Follow Up For Removal

Response Number 10
Name: goompa
Date: November 13, 2008 at 19:47:23 Pacific
Reply:

Ah I now understand (I hope)

'findstr /b /v "d" TLIST > svrlist'

to say the above string verbosely:

find the string in TLIST at the beginning of the line only if it does not have a "d" in it then use the output created by this line to create the file svrlist

I'm assuming this line would be key to compare "the output of the local side directory listing" with ""the output of the remote side directory listing" with the additional calculation of the modified dates for each file or something.


Report Offensive Follow Up For Removal

Response Number 11
Name: Holla
Date: November 13, 2008 at 22:10:25 Pacific
Reply:

goompa,

If you can spend money, jpsoft has a cmd.exe replacement called TCC, which appears to this job seamlessly. (Not by the free TCC LE).
An elaborate full fledged script for updating files from ftp site using TCC:

http://www.jpsoft.com/forums/showth...

You can 'try' their software also.

I have no affiliations with jpsoft :-)
I just happen to love their products.

--
Holla.


Report Offensive Follow Up For Removal

Response Number 12
Name: Mechanix2Go
Date: November 14, 2008 at 04:01:19 Pacific
Reply:

That svrlist looks odd. Can you post a few lines of TLIST?


=====================================
If at first you don't succeed, you're about average.

M2


Report Offensive Follow Up For Removal

Response Number 13
Name: goompa
Date: November 14, 2008 at 16:22:24 Pacific
Reply:

Hey M2, the output from both are in Response # 7
For the little batch file, was I supposed to execute that on a remote folder or is it ok to run it where ever?


Report Offensive Follow Up For Removal

Response Number 14
Name: Mechanix2Go
Date: November 14, 2008 at 18:48:41 Pacific
Reply:

' For the little batch file, was I supposed to execute that on a remote folder or is it ok to run it where ever?'

wherever

----
Still mystified about your TLIST.

Run the script below, after putting in your particulars.

======================================
@echo off
setLocal EnableDelayedExpansion

> ft.do echo o yoursite.com
>> ft.do echo username
>> ft.do echo password
>> ft.do echo bin
>> ft.do echo cd mystuff
>> ft.do echo ls -l TLIST
>> ft.do echo bye

ftp -s:ft.do
del ft.do


=====================================
If at first you don't succeed, you're about average.

M2


Report Offensive Follow Up For Removal






Post Locked

This post is quite old and has been locked from receiving new replies. Please create a new posting instead.


Go to Programming Forum Home



Results for: ftp batch to download newest files

FTP check to see if file exists
    Summary: I am implementing an application using ftp to transfer files from ftp server to local location. I want to see if a particular file exists before starting the download for all files. I implement this s...
www.computing.net/answers/programming/ftp-check-to-see-if-file-exists/14691.html

Using Batch to Rename 'some' files
    Summary: We pull down files via FTP, these files are in hex format with no extension (eg. 008C334D) and the company that hosts the FTP site has said they can not change these files to have a different filename...
www.computing.net/answers/programming/using-batch-to-rename-some-files/15862.html

batch to shutdown computer!
    Summary: Dear Mechanix2Go; I am looking for a batch file like the one you had write for me to download update files. It works cool. Now I would like a batch that shutdown my PC when a download is completed. Fo...
www.computing.net/answers/programming/batch-to-shutdown-computer/14394.html








Which MP3 player do you have?

iPod/iPhone
Zune
Something Else
None


View Results

Poll Finishes In 2 Days.
Discuss in The Lounge
Poll History






Data Recovery Software