function to greatly speed up recursive file or folder count

This topic contains 0 replies, has 1 voice, and was last updated by  Forums Archives 5 years, 8 months ago.

  • Author
    Posts
  • #6029

    by teezee at 2013-02-21 06:36:28

    This is a simple script I've been working on this morning to make some speed test on recursive file and folder for another script I'm working on.
    since this prove to be a significant boost while dealing with a really large amount of files... well I thought about sharing it here.

    You can copy paste this into PowerShell ISE and press f5 has is.
    sorry for not using powershell syntax, but I found out quick enough that it 'mess up' the copy/paste

    it will simply count all the files and folders pf c:\windows\ and show the total and some statistics as in the commented examples.


    #You can copy paste this into Powershell ISE and press f5 as is.
    $ErrorActionPreference = "silentlycontinue"

    write-host "Method 1 :: Folder count light" -ForegroundColor Green -BackgroundColor Black

    function FolderCountlight{
    $path = $args[0]

    function Recursive{
    $dir = $args[0]
    $i = $args[1]

    foreach ($d in $dir.GetDirectories()){
    #$d #uncomment $d to get objects
    $i.Count++ #comment this if you uncomment the $d above...
    Recursive $d $i
    }
    }

    $statistics = New-Object PsObject -Property @{Count = 0}
    Recursive (new-object System.IO.DirectoryInfo $path) $statistics
    return $statistics.count
    }

    Measure-Command{$statistics = FolderCountlight c:\windows}
    Write-Output "Count : $statistics"

    write-host "Method 2 :: File count light" -ForegroundColor green -BackgroundColor black
    function FolderCountlight{
    $path = $args[0]

    function Recursive{
    $ErrorActionPreference = "silentlycontinue"
    $dir = $args[0]
    $i = $args[1]

    foreach ($d in $dir.GetDirectories()){

    foreach ($f in $d.GetFiles()){
    #$f #uncomment $f to get objects
    $i.Count++ #comment this if you uncomment the $f above...
    }

    Recursive $d $i
    }
    }
    $ErrorActionPreference = "continue"
    $leftover = New-Object System.IO.directoryinfo -ArgumentList $path
    $addleftover = $leftover.GetFiles()
    $statistics = New-Object PsObject -Property @{Count = 0}
    Recursive (new-object System.IO.DirectoryInfo $path) $statistics
    $ErrorActionPreference = "continue"
    $statistics = ($addleftover.count + $statistics.count) #remove the .count if you happen to use the $f option above. and ADD '.COUNT' TO THE LINE !!!!BELOW!!!!
    return $statistics
    }

    Measure-Command{$statistics = FolderCountlight c:\windows}
    Write-Output "Count : $($statistics)" # !!!!THIS LINE!!!! or you might not enjoy the unCTRL-C able listing in ISE... !!!! simply add '.count' before the ) after $statistics
    # for some reason you might want to substact 1 to the total if you use the 'object'.count version...

    write-host "Method 3 :: normal folder count" -ForegroundColor green -BackgroundColor black
    Measure-Command{$statistics = (Get-ChildItem -Directory -Recurse -Force -Path c:\windows).count}
    Write-Output "Count : $statistics"

    write-host "Method 4 :: normal file count" -ForegroundColor green -BackgroundColor black
    Measure-Command{$statistics = (Get-ChildItem -file -Recurse -Force -Path c:\windows).count}
    Write-Output "Count : $statistics"

    < #
    Method 1 :: Folder count light

    Days : 0
    Hours : 0
    Minutes : 0
    Seconds : 4
    Milliseconds : 14
    Ticks : 40146658
    TotalDays : 4.64660393518518E-05
    TotalHours : 0.00111518494444444
    TotalMinutes : 0.0669110966666667
    TotalSeconds : 4.0146658
    TotalMilliseconds : 4014.6658

    Count : 16914
    Method 2 :: File count light
    Days : 0
    Hours : 0
    Minutes : 0
    Seconds : 6
    Milliseconds : 64
    Ticks : 60644034
    TotalDays : 7.01898541666667E-05
    TotalHours : 0.0016845565
    TotalMinutes : 0.10107339
    TotalSeconds : 6.0644034
    TotalMilliseconds : 6064.4034

    Count : 79676
    Method 3 :: normal folder count
    Days : 0
    Hours : 0
    Minutes : 0
    Seconds : 8
    Milliseconds : 338
    Ticks : 83387421
    TotalDays : 9.651321875E-05
    TotalHours : 0.00231631725
    TotalMinutes : 0.138979035
    TotalSeconds : 8.3387421
    TotalMilliseconds : 8338.7421

    Count : 16914
    Method 4 :: normal file count
    Days : 0
    Hours : 0
    Minutes : 0
    Seconds : 14
    Milliseconds : 868
    Ticks : 148685243
    TotalDays : 0.00017208940162037
    TotalHours : 0.00413014563888889
    TotalMinutes : 0.247808738333333
    TotalSeconds : 14.8685243
    TotalMilliseconds : 14868.5243

    Count : 79676
    #>

    You can also use the 2 first method to populate the "$statistics" variable with 'filesystem object' instead of a simple number... it slows down the process a tad, but it is still alot faster then the get-child item way.
    just see comments 🙂

    Jon go.
    -tz

    by mjolinor at 2013-02-21 15:32:20

    It's a good coding exercise.

    If it's a matter of pure performance, I've never found anything that benchmarks faster than the legacy dir.

    $DirCount =
    (cmd /c dir c:\windows /b /s /ad).count
    $FileCount=
    (cmd /c dir c:\windows /b /s /a-d).count

    by teezee at 2013-02-24 12:23:28

    haha! good one, but yes, as you stated. this is mostly to exercise my coding experience. 🙂

    by nohandle at 2013-02-24 13:12:11

    Mjolinor: Nice boost.

    I am unable to get same readings from the code with my method. Does any one see why?
    Running it as normal user:

    PS C:\TEMP> (Get-ChildItem -Path C:\Windows -Directory -Recurse -ErrorAction silentlycontinue).count
    20231

    PS C:\TEMP> (cmd /c dir c:\windows /b /s /ad).count
    20347

    runing as system:
    PS C:\Windows\system32> (cmd /c dir c:\windows /b /s /ad).count
    20711
    PS C:\Windows\system32> (gci -recurse -directory c:\windows ).count
    20491

    [quote="teezee"]function FolderCountlight{
        $path = $args[0][/quote]

    This is not how you should pass arguments to a function. This approach is bit easier to use, for even advanced function definitions search for powershell advanced functions.
    Function Get-FolderCountLight ([string]$Path){
    "this is $path"
    }

    by mjolinor at 2013-02-24 20:33:04

    $cmd = cmd /c dir c:\windows /b /s /ad
    $gci = gci -recurse -directory c:\windows |
    select -ExpandProperty fullname

    Compare-Object $cmd $gci

    dir seems to be a more comprehensive counter than gci.

    by nohandle at 2013-02-24 23:59:09

    I can see my method gives me back smaller amounts, but I can't see why 🙂 the permissions should be the same for the two commands and definition of directory should be ( I hope) the same also. Still yours and OP give me back more results.

    by poshoholic at 2013-02-25 06:08:15

    You can run Get-ChildItem with -Force to also process hidden files/folders. By default it doesn't touch hidden files or folders.

    by nohandle at 2013-02-26 00:59:35

    [quote="poshoholic"]You can run Get-ChildItem with -Force to also process hidden files/folders. By default it doesn't touch hidden files or folders.[/quote]
    Thanks Kirk, I was sure I am missing something trivial.

You must be logged in to reply to this topic.