Multithreading Script

[kvprasoon]

I am attempting to multithread a script and any time I increase the amount of threads, the performance of the script greatly decreases.  For example, if I set a max of 5 threads I can write 635 objects/min.  When I set to 10 threads, I can write only 361 objects/min.  I have beefed up resources on the server I am running this from and never get close to maxing out the hardware, so I know it’s not a hardware limitation.

Here is the script:

PowerShell

$runspacecollection = @() $runspacepool = [runspacefactory]::CreateRunspacePool(1, 5) $runspacePool.Open() $scriptblock = { param($folder) Script goes here } Write-Host ‘Parsing Folder Structure’ $folders = Get-ChildItem \\server\Datastore Write-Host ‘Processing Files’ foreach ($folder in $folders) { $powershell = [powershell]::Create().AddScript($scriptblock).AddArgument($folder) $powershell.RunspacePool = $runspacepool [Collections.ArrayList]$runspacecollection += New-Object -TypeName System.Management.Automation.PSObject -Property @{ runspace = $powershell.BeginInvoke() powershell = $powershell } } while ($runspacecollection) { foreach ($runspace in $runspacecollection.ToArray()) { if ($runspace.Runspace.IsCompleted) { $runspace.Powershell.EndInvoke($runspace.Runspace) $runspace.Powershell.Dispose() $runspacecollection.Remove($runspace) } } }

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

$runspacecollection = @() $runspacepool = [runspacefactory]::CreateRunspacePool(1, 5) $runspacePool.Open()     $scriptblock = {     param($folder)       Script goes here   }     Write-Host ‘Parsing Folder Structure’ $folders = Get-ChildItem \\server\Datastore Write-Host ‘Processing Files’     foreach ($folder in $folders) {     $powershell = [powershell]::Create().AddScript($scriptblock).AddArgument($folder)     $powershell.RunspacePool = $runspacepool     [Collections.ArrayList]$runspacecollection += New-Object -TypeName System.Management.Automation.PSObject -Property @{         runspace   = $powershell.BeginInvoke()         powershell = $powershell     } }     while ($runspacecollection) {     foreach ($runspace in $runspacecollection.ToArray()) {         if ($runspace.Runspace.IsCompleted) {             $runspace.Powershell.EndInvoke($runspace.Runspace)             $runspace.Powershell.Dispose()             $runspacecollection.Remove($runspace)         }     } }

• This topic was modified 1 month, 2 weeks ago by kvprasoon. Reason: Code formatting https://powershell.org/forums/topic/guide-to-posting-code-2/

[Joseph Monarch Participant Topics: 13 Replies: 26 Points: 44Rank: Member]

After checking various settings and syntax, my issue is with the server that I am reading files from.  I can see that I am getting all the threads, however, I am being bottlenecked by the read speed on the server I am reading from.

The more threads the bigger the read speed bottleneck.  Explains a lot.

[Author]

Posts