Computing.Net > Forums > Programming > Appending the timestamp as a column

Computer Problems? Computing.Net has over 1,000,000 posts about all things technology related! Over 90% answered within 24 hours! Click here to start participating now! Also, be sure to check out the New User Guide.

Appending the timestamp as a column

Reply to Message Icon

Name: sunitha_1984
Date: June 9, 2008 at 08:14:01 Pacific
OS: xp
CPU/Ram: intel
Product: 2003
Comment:

I have to write a .bat script , which will extract the timestamp value from a filename i.e i will get filesnames as below (.csv file pipe (|) seperated)

RBC.20080112010032.csv
RBC.20080211130012.csv
RBC.20080309130027.csv
RBC.20080408130049.csv

I have to extract the timestamp values as below from each file

20080112010032
20080211130012
20080309130027
20080408130049

and add a column by name file_timestamp at the end of the files for all the corresponding files.

below is a batch script which will extract the timestamp value and will place it in a .txt file

@echo off
setLocal EnableDelayedExpansion
pushd c:\working\703985

for /f "tokens=2,3 delims=." %%a in ('dir /b RBC*.CSV') do (
echo %%a|find "200" > nul
if errorlevel 1 (
set filename=%%b
) else (
set filename=%%a)
echo !filename! > filename.txt)

After this i need to combine all the .csv files in that folder into 1 single file .txt file.
Remove the header row from each of these files
All my CSV files are compatible, i.e. they have the same structure as columns and their contents.

I tried the below batch script for removing the headers
@echo off
setLocal EnableDelayedExpansion

if exist *.new del *.new

for /f "tokens=* delims= " %%A in ('dir /b RBC*') do (
set NAME=%%~nA
set fileNAME=%%A
for /f "tokens=* skip=1 delims= " %%L in (!fileNAME!) do (
echo %%L >> !NAME!.txt
)
)


but the above script is taking more time..as my files are above 60 mb in size is there any alternate way which will be performance efficient.


I tried to combined the batch script but as this is my first batch script i am not able to acheive my result..

Request you to guide me or suggest me how i can do this in a single .bat script.

Thanks,
Sunitha



Sponsored Link
Ads by Google

Response Number 1
Name: klint
Date: June 9, 2008 at 08:55:14 Pacific
Reply:

The problem is that using batch (or even VBScript) for reading a large file is always going to take a long time. If you can use a third-party utility, such as GNU awk, it will be much faster. The following command strips off the first line of a file:

awk "NR>1" <from-file >to-file


0

Response Number 2
Name: sunitha_1984
Date: June 9, 2008 at 09:24:07 Pacific
Reply:

Hi klint,

Thanks for the response.

I need to do every thing in .bat scripting only..i can't use awk or any other third party utility.

Thanks,
Sunitha


0

Response Number 3
Name: klint
Date: June 9, 2008 at 10:30:47 Pacific
Reply:

I don't understand what you're doing with this code:


for /f "tokens=2,3 delims=." %%a in ('dir /b RBC*.CSV') do (
echo %%a|find "200" > nul
if errorlevel 1 (
set filename=%%b
) else (
set filename=%%a
)
echo !filename! > filename.txt
)

It seems to me that you want to write "20080112010032" if that file doesn't contain the string "200", and "CSV" if it does. So you'll end up with a filename.txt that looks like this:

20080112010032
CSV
20080211130012
CSV
CSV
20080309130027

Is this what you want or have I misunderstood your code?


0

Response Number 4
Name: sunitha_1984
Date: June 9, 2008 at 10:42:58 Pacific
Reply:

Klint,

I am extracting the timestamp value from the file name , all the files which i receive will have a valid date time , which starts with 200,
so from the above script i am extracting the timestamp value and passing it to .txt file.But what i want is i have to extract the timestamp value of all the files, suppose if i have 4 .csv files..i have to extract the timestamp from these four files and add a column to the source file and should pass the extracted value to that column.

see the below exzmple for clear understanding

RBC.20080213233443.csv is my source file name

i want to populate data
and below to my source file layout.

RBCSEC_ID|ORIG_ID|SEC_ID|file_id
1|2|3|20080213233443
2|9|3|20080213233443
2|8|2|20080213233443
3|7|5|20080213233443
4|6|7|20080213233443

similarly for all the files.

Hope you understood.

Thanks,
Sunitha


0

Response Number 5
Name: klint
Date: June 9, 2008 at 14:26:25 Pacific
Reply:

Ok, I think I know what you're trying to do, but I think there's still a small error in the first batch you wrote above:


for /f "tokens=2,3 delims=." %%a in ('dir /b RBC*.CSV') do (

Here, you're scanning the directory listing, and using only tokens 2 and 3 so that %%a is the date stamp and %%b will be "CSV" (the third token).


echo %%a|find "200" > nul

Testing if this filename contains a date stamp.


if errorlevel 1 (
set filename=%%b

If this file name is not in the expected format, i.e. has no date stamp, then set filename to "CSV".


) else (
set filename=%%a
)

If the file name is as expected, set filename to the date stamp.


echo !filename! > filename.txt
)

So, here, you are echoing a variable that contains either the date stamp, or the string "CSV". This doesn't sound like what you wanted to do. Also, I suspect you meant to write

echo !filename! > !filename!.txt

rather than creating the same file, called filename.txt, and overwriting the previous one each time.

Now, to combine your two batch files, it's just a matter of continuing inside this outer loop, just after the above statement where you create !filename!.txt, adding the lines from your second batch file, but using >> !filename!.txt to append to the file. However, I think you will still find that this is very slow.

The command-line interpreter is not intended for line-by-line manipulation of large files, and is inefficient at it. Every time you do the following inside your for-loop:

echo %%L >> !filename!.txt

the system opens the file !filename!.txt, jumps to the end of the file, writes the line, flushes the file buffer to disk, and closes the file. For the next line, it opens the file again and repeats. It just will never be anywhere near as fast as a utility that opens the file just once, writes everything it needs to write, and then closes it at the end.


0

Related Posts

See More



Sponsored Link
Ads by Google
Reply to Message Icon






Post Locked

This post is quite old and has been locked from receiving new replies. Please create a new posting instead.


Go to Programming Forum Home


Sponsored links

Ads by Google


Results for: Appending the timestamp as a column

Program to add filename as column www.computing.net/answers/programming/program-to-add-filename-as-column/16698.html

Batch File- Subfolder as a Variable www.computing.net/answers/programming/batch-file-subfolder-as-a-variable/14153.html

Performance issue with the.bat scri www.computing.net/answers/programming/performance-issue-with-thebat-scri/16754.html