Solved Concatenate multiple csv files

November 9, 2012 at 08:01:23
Specs: Windows 2003 server, 32GB
Concatenate multiple csv files
Hi Guys, I'm need to merge multiple csv files within a folder into one csv file without duplicate headers, and I'm using the solution provided in this forum
Concatenate multiple csv files
http://www.computing.net/answers/pr...
and it works fine. The main issue is when file size is >6526KB, more +1 command is hanging.
For us we may even have up to 50MB of files to be processed.

Can anyone please help if there is any limitation on file size using more command or any other solution to perform the task.
Thankyou
Vennela


See More: Concatenate multiple csv files

Report •


✔ Best Answer
November 10, 2012 at 02:06:39
@thebest123456

cap locks up because I was so hurry to prefer that to bold style. In fact below you can read the ultimate (optimized) version of my original script

for %%i in (*.csv) do (
  set /P header=< %%~snxi
  goto :DONE
)
:DONE
echo.%header%> output.tmp
for %%i in (*.csv) do (
  for /F "skip=1 delims=" %%j in (%%~snxi) do echo.%%j>> output.tmp
)
ren output.tmp output.csv



#1
November 9, 2012 at 09:52:39
WAIT, IT IS WRONG!

I guess the size of your files switches on the pause that is builtin in more by default.
So try the following variant (untested)

setlocal EnableDelayedExpansion
for /F "delims=" %%i in ('dir /B *.csv') do (
  set /P header=< "%%i"
  echo.!header!> output.hdr
)
for %%i in (*.csv) do for /F "skip=1 delims=" %%j in ('type "%%i"') do type "%%j" >> output.tmp
type output.tmp >> output.hdr
ren output.hdr output.csv


Report •

#2
November 9, 2012 at 10:16:51
THIS IS RIGHT (ALMOST I HOPE)

setlocal EnableDelayedExpansion
for %%i in (*.csv) do (
  set /P header=< "%%i"
  echo.!header!> output.hdr
)
for %%i in (*.csv) do (
  for /F "skip=1 delims=" %%j in (%%i) do echo.%%j>> output.tmp
)
type output.tmp >> output.hdr
ren output.hdr output.csv


Report •

#3
November 9, 2012 at 10:26:20
lol IVO what's up with the cap locks?
Just finished my version with a part of yours integrated, messy but gets the job done.
@echo off & setlocal EnableDelayedExpansion
dir /b *.csv> %tmp%\list
set cnt=0
for /f "delims=" %%a in ('type %tmp%\list') do (
set /A cnt +=1
set line!cnt!=%%a
)
set limit=%cnt%
set number=0
:LOOP
set /A number +=1
set /P header=<!line%number%!
find /C:"%header%" output.hdr || echo.!line%number%!>> output.tmp
if "%limit%" NEQ "%number%" Goto :LOOP
del %tmp%\list
for %%i in (*.csv) do for /F "skip=1 delims=" %%j in ('type "%%i"') do type "%%j" >> output.tmp
type output.tmp >> output.hdr
ren output.hdr output.csv


Report •

Related Solutions

#4
November 9, 2012 at 13:50:04
Are all the headers the same? And if so, can you post the content of the header? (and maybe one sample-line of data would be nice).(Unless of course this post is already solved)

Report •

#5
November 10, 2012 at 02:06:39
✔ Best Answer
@thebest123456

cap locks up because I was so hurry to prefer that to bold style. In fact below you can read the ultimate (optimized) version of my original script

for %%i in (*.csv) do (
  set /P header=< %%~snxi
  goto :DONE
)
:DONE
echo.%header%> output.tmp
for %%i in (*.csv) do (
  for /F "skip=1 delims=" %%j in (%%~snxi) do echo.%%j>> output.tmp
)
ren output.tmp output.csv


Report •

#6
November 12, 2012 at 09:57:23
Great..thanks Ivo..That worked
what was the issue with More.
I earlier looked for some documentation on more and couldn't find any limitation.

Thanks for the help. I'm new to writing scripts in DOS and this helped a lot.

Regards,
Vennela


Report •

#7
November 12, 2012 at 12:43:19
I guess the size of your files switches on the pause that is builtin in more by default; after all MORE is aimed to display screen by screen setting a pause between. By the way this is NT batch, nothing to do with DOS that does not exist in NT based Windows systems like Windows 2003 server.

Report •

Ask Question