Batch FTP download of html CSV data

December 19, 2010 at 08:24:08
Specs: Windows server 2008 R2
There is a website that provides csv data in html format only. I need to automate the process of securely logging into the site, capturing the data from the HTML page and saving it as a csv file on a server.

See More: Batch FTP download of html CSV data

Report •


#1
December 19, 2010 at 12:13:07
HTML is a text file meant to be read by humans, made pretty with the usage of tags, and typically associated style sheets and JavaScripts.

CSV is a text file representing tabular data, where columns are separated by commas, and rows by new lines.

If it's an HTML file, it is not a CSV file. If it's a CSV file, it isn't a HTML file. They are mutually exclusive.

Perhaps more details are in order?

How To Ask Questions The Smart Way


Report •

#2
December 19, 2010 at 14:51:17
although sendkeys is generally frowned upon, it might be what you need to get the job done, since it involves getting logged in to a secure site. Once the page with
the data is loaded, you would send:
alt-f (that is: %f)
a
{tab}
t
{enter}
which saves the page as text instead of html (the "t" after the {tab}). You will then
have to format the text into the strict csv requirements using either batch or a
language. Although you could save the text as html, the formatting would be alot
more cumbersome and difficult (doing the html extraction).

Report •

#3
December 19, 2010 at 19:02:08
This is the data as it appears on the web page. This needs to be turned into a CSV file at the receiving computer.

base,quote,amount,date,ask,bid
USD,ADF,1,12-18-2010,0.7585,0.7579
USD,ADP,1,12-18-2010,0.7585,0.7579
USD,AED,1,12-18-2010,3.6740,3.6720
USD,AFA,1,12-18-2010,45.2900,45.0900
USD,AFN,1,12-18-2010,45.2900,45.0900
USD,ALL,1,12-18-2010,107.440,103.640
USD,AMD,1,12-17-2010,360.250,360.250
USD,ANG,1,12-17-2010,1.8200,1.7800
USD,AOA,1,12-18-2010,92.5430,92.0820
USD,AON,1,12-18-2010,92.5430,92.0820
USD,ARS,1,12-18-2010,3.9669,3.9601
USD,ATS,1,12-18-2010,0.7585,0.7579
USD,AUD,1,12-18-2010,1.0126,1.0116
USD,AWG,1,12-17-2010,1.7998,1.7798
USD,AZM,1,12-30-2009,4015.50,4015.50
USD,AZN,1,12-14-2010,0.7992,0.7982
USD,BAM,1,12-18-2010,1.4832,1.4831
USD,BBD,1,12-18-2010,2.0515,1.9757
USD,BDT,1,12-18-2010,72.4190,69.6890
USD,BEF,1,12-18-2010,0.7585,0.7579
USD,BGL,1,12-18-2010,1.4907,1.4755
USD,BGN,1,12-18-2010,1.4907,1.4755
USD,BHD,1,12-18-2010,0.3779,0.3741
USD,BIF,1,12-18-2010,1263.10,1220.60
USD,BMD,1,12-17-2010,1.0000,1.0000
USD,BND,1,12-18-2010,1.3406,1.3027
USD,BOB,1,12-18-2010,7.2093,6.9059
USD,BRL,1,12-18-2010,1.7116,1.7056
USD,BSD,1,12-18-2010,0.9985,0.9880
USD,BTN,1,12-18-2010,45.1830,45.1680
USD,BWP,1,12-18-2010,6.7537,6.5110


Thanks for your help with this.


Report •

Related Solutions

#4
December 19, 2010 at 19:32:09
Although it looks like that onscreen, it probably won't look like that once you get
a "snapshot" using your browser's text-file-format save. If you want it to be
completely automated (which i suspect you do, trading futures/stocks), and a "download"
option is not supplied by your data-source (or is it? check into that...), the only
other means is to do the text-file-save and capture the data from the resultant
text-file. If you choose to go this route, I can maybe help some with both the
sendkey stuff and the formatting, but for the formatting you would need to post
the results of a text-file-save from your browser.
I've worked on a similar system to handle my own broker's website, so I am familiar
with this scenario. automation can be a real bitch because your program rarely
knows what's going on in the browser except from text-file-save results which are
subject to misinterpretation.

Report •


Ask Question