Solved How to I move files into new folders upto 4gb size folders

November 16, 2018 at 02:30:06
Specs: Windows 7
Hi

I have a folder of which contain files of all different sizes.

What im trying to do is have a script (VBS or CMD, CMD preferable as can actually see it running on the MS DOS screen ) that runs in order starting from the top of the list moving files into new sub folders but only upto 4gb of data in each folder.

Lets say I got 6 folders

1- 2gb
2- 750mb
3- 4 gb
4- 3 gb
5- 1gb
6- 3gb

What I would need to do is move files 1 &2 into a new folder as file 3 would take it over the limit.
Then file 3 into new on on its own as is the limit
Then files 4 & 5 together as the limt and finally file 6 and next file if less that 1gb.

Folder where original download is in is X:\DOWNLOADS

and new location I need them is X:\DOWNLOADS\SPLIT and then all the subfolders as numbers.

The files will not be all the same size .

I know how to use robocopy but thats about it.

Any Ideas.

Thanks.

message edited by tweacle


See More: How to I move files into new folders upto 4gb size folders

Reply ↓  Report •

✔ Best Answer
November 21, 2018 at 02:55:31
64bit calculation is handled by powershell.

you can replace @echo off with @echo on if you want to see more data on screen, tho it does slow down the process a bit.

files are not moved, but copied, so you have to remove the files manually yourself

there is very little error-handling, so for example running the script 2 times in a row without deleting the split1-x can cause weird behavior.

it does not check if there is another .bat file running, etc.

TL;DR it's basically a 1 time only script.
if you have a question, please ask :)


@echo off&setlocal enabledelayedexpansion
::2^32-1
set THRESHOLD=4294967295
set "WORKINGDIR=X:\DOWNLOADS"
pushd "!WORKINGDIR!"
if not !errorlevel!==0 exit/b
if not exist "!WORKINGDIR!\SPLIT" mkdir SPLIT
set SPLIT#=0
call :MAKESPLIT
:MAIN
for /f "skip=7 tokens=4-5" %%a in ('dir/AD') do (
if "%%a"=="<DIR>" (
if not "%%a"=="SPLIT" (
call :GETDIRSIZE "%%b"
)
)
)
exit/b
:GETDIRSIZE
pushd "%~1"
for /f "tokens=3" %%a in ('dir/a/s/-c ^| findstr "File(s)"') do set "DIRSIZE=%%a"
popd
for /f %%a in ('powershell.exe %SPLITSIZE%+%DIRSIZE%') do set VAL=%%a
for /f %%a in ('powershell.exe "if (%VAL% -le %THRESHOLD%) {echo yes} else {echo no}"') do if "%%a"=="yes" (
echo %%a
set SPLITSIZE=!VAL!
xcopy /E /H "%~1" "!WORKINGDIR!\SPLIT\SPLIT!SPLIT#!"
) else (
call :MAKESPLIT
call :GETDIRSIZE "%~1"
)
exit/b
:MAKESPLIT
set/a SPLIT#+=1
set SPLITSIZE=0
mkdir "!WORKINGDIR!\SPLIT\SPLIT!SPLIT#!"
exit/b

i5-6600K[delid]@4.8GHz/4.4GHz@1.424v LLC=6 | 2x4GB Crucial-DDR4-2133CL15@14-14-14-30 1T 2800MHz@1.37v
MSI Armor RX 570 4GB@1410Mhz core@1.2v/1920MHz



#1
November 16, 2018 at 04:30:43
sounds fun, .bat will be used

I'll finish the script between tomorrow and Monday evening, while asking for feedback.

if anyone else wants to jump in and take over, no problem, im not gonna start scripting before tomorrow probably

i5-6600K[delid]@4.8GHz/4.4GHz@1.424v LLC=6 | 2x4GB Crucial-DDR4-2133CL15@14-14-14-30 1T 2800MHz@1.37v
MSI Armor RX 570 4GB@1420Mhz core@1.2v/1920MHz


Reply ↓  Report •

#2
November 16, 2018 at 06:03:00
Thats absolutely fine I hope it made sense what I was saying

Reply ↓  Report •

#3
November 20, 2018 at 01:58:57
currently testing with 20 .jpg files worth 638KB each, just checking if i can properly give the copy job a threshold of xxxx bytes.

Don't want to mess up, getting it right the first time will save you some time :)

i5-6600K[delid]@4.8GHz/4.4GHz@1.424v LLC=6 | 2x4GB Crucial-DDR4-2133CL15@14-14-14-30 1T 2800MHz@1.37v
MSI Armor RX 570 4GB@1410Mhz core@1.2v/1920MHz


Reply ↓  Report •

Related Solutions

#4
November 20, 2018 at 03:10:21
Big Big thanks

Reply ↓  Report •

#5
November 20, 2018 at 06:53:22
Known issue, it will copy an empty SPLIT1 dir into SPLIT2, does not repeat after that.

edit: found some more issues...


i5-6600K[delid]@4.8GHz/4.4GHz@1.424v LLC=6 | 2x4GB Crucial-DDR4-2133CL15@14-14-14-30 1T 2800MHz@1.37v
MSI Armor RX 570 4GB@1410Mhz core@1.2v/1920MHz

message edited by hidde663


Reply ↓  Report •

#6
November 20, 2018 at 07:16:21
this still attempts to copy split1 once, but fails :), everything else works

@echo off&setlocal enabledelayedexpansion
::in Bytes:
set THRESHOLD=2000000000
set "WORKINGDIR=X:\DOWNLOADS"
pushd "!WORKINGDIR!"
if not !errorlevel!==0 exit/b
if not exist "!WORKINGDIR!\SPLIT" mkdir SPLIT
set SPLIT#=0
call :MAKESPLIT
:MAIN
for /f "skip=7 tokens=4-5" %%a in ('dir/AD') do (
if %%a==^<DIR^> (
if not %%a==SPLIT (
call :GETDIRSIZE "%%b"
)
)
)
exit/b
:GETDIRSIZE
pushd "%~1"
for /f "tokens=3" %%a in ('dir/a/s/-c ^| findstr "File(s)"') do set "DIRSIZE=%%a"
popd
set/a VAL=!SPLITSIZE!+!DIRSIZE!
if !VAL! LSS !THRESHOLD! (
set/a SPLITSIZE=!VAL!
xcopy /E /H "%~1" "!WORKINGDIR!\SPLIT\SPLIT!SPLIT#!"
) else (
call :MAKESPLIT
call :GETDIRSIZE "%~1"
)
exit/b
:MAKESPLIT
set/a SPLIT#+=1
set SPLITSIZE=0
mkdir "!WORKINGDIR!\SPLIT\SPLIT!SPLIT#!"
exit/b

i5-6600K[delid]@4.8GHz/4.4GHz@1.424v LLC=6 | 2x4GB Crucial-DDR4-2133CL15@14-14-14-30 1T 2800MHz@1.37v
MSI Armor RX 570 4GB@1410Mhz core@1.2v/1920MHz

message edited by hidde663


Reply ↓  Report •

#7
November 20, 2018 at 10:23:26
hidde663: set THRESHOLD=4000000000
You DO know 1GB is defined as 2^30 = 1,073,741,824, right? Also, you need your test data greater than 2GB to see the true hassle of this problem. Specifically, CMD uses signed 32 bit variables, meaning your max size without bugs is (2GB - 1)

How To Ask Questions The Smart Way

message edited by Razor2.3


Reply ↓  Report •

#8
November 20, 2018 at 13:37:37
i had 4GB somehow burned into my head, and yes, 2147483647 would be the limit then.

anyhow ill change it to 2.000.000.000 to be safe :)

but thats not entirely true tho, i've been working on a dec to bin to dec adder, max value so far is about 9223372036854775807.

binary arithmetic >.< very inefficient and time consuming tho.

would be better off calling a vbscript doing the 64bit calculations and catching them in a variable in this case tho

will do tomorrow

i5-6600K[delid]@4.8GHz/4.4GHz@1.424v LLC=6 | 2x4GB Crucial-DDR4-2133CL15@14-14-14-30 1T 2800MHz@1.37v
MSI Armor RX 570 4GB@1410Mhz core@1.2v/1920MHz

message edited by hidde663


Reply ↓  Report •

#9
November 20, 2018 at 13:47:24
P.S. thanks for letting me know Razor2.3

i5-6600K[delid]@4.8GHz/4.4GHz@1.424v LLC=6 | 2x4GB Crucial-DDR4-2133CL15@14-14-14-30 1T 2800MHz@1.37v
MSI Armor RX 570 4GB@1410Mhz core@1.2v/1920MHz


Reply ↓  Report •

#10
November 21, 2018 at 02:55:31
✔ Best Answer
64bit calculation is handled by powershell.

you can replace @echo off with @echo on if you want to see more data on screen, tho it does slow down the process a bit.

files are not moved, but copied, so you have to remove the files manually yourself

there is very little error-handling, so for example running the script 2 times in a row without deleting the split1-x can cause weird behavior.

it does not check if there is another .bat file running, etc.

TL;DR it's basically a 1 time only script.
if you have a question, please ask :)


@echo off&setlocal enabledelayedexpansion
::2^32-1
set THRESHOLD=4294967295
set "WORKINGDIR=X:\DOWNLOADS"
pushd "!WORKINGDIR!"
if not !errorlevel!==0 exit/b
if not exist "!WORKINGDIR!\SPLIT" mkdir SPLIT
set SPLIT#=0
call :MAKESPLIT
:MAIN
for /f "skip=7 tokens=4-5" %%a in ('dir/AD') do (
if "%%a"=="<DIR>" (
if not "%%a"=="SPLIT" (
call :GETDIRSIZE "%%b"
)
)
)
exit/b
:GETDIRSIZE
pushd "%~1"
for /f "tokens=3" %%a in ('dir/a/s/-c ^| findstr "File(s)"') do set "DIRSIZE=%%a"
popd
for /f %%a in ('powershell.exe %SPLITSIZE%+%DIRSIZE%') do set VAL=%%a
for /f %%a in ('powershell.exe "if (%VAL% -le %THRESHOLD%) {echo yes} else {echo no}"') do if "%%a"=="yes" (
echo %%a
set SPLITSIZE=!VAL!
xcopy /E /H "%~1" "!WORKINGDIR!\SPLIT\SPLIT!SPLIT#!"
) else (
call :MAKESPLIT
call :GETDIRSIZE "%~1"
)
exit/b
:MAKESPLIT
set/a SPLIT#+=1
set SPLITSIZE=0
mkdir "!WORKINGDIR!\SPLIT\SPLIT!SPLIT#!"
exit/b

i5-6600K[delid]@4.8GHz/4.4GHz@1.424v LLC=6 | 2x4GB Crucial-DDR4-2133CL15@14-14-14-30 1T 2800MHz@1.37v
MSI Armor RX 570 4GB@1410Mhz core@1.2v/1920MHz


Reply ↓  Report •

#11
November 21, 2018 at 11:52:16
Many Thanks you are a legend

Reply ↓  Report •

Ask Question