Spawn Up to 8 processes at a time until all 680 have ran

This topic contains 4 replies, has 3 voices, and was last updated by  Brian B 1 year, 12 months ago.

  • Author
    Posts
  • #33037

    Brian B
    Participant

    Hello there,

    I've been trying to think of a good way to speed up a relatively lengthy, if single threaded, process of scanning every single file share on our NetApp appliance (they're just CIFS file shares, i.e. \\Netapp\Share1 , Share2, Share3, etc..) and Although I can run Robocopy to parse the results quite nicely due to some example scripts i've found, what I want to do is run more than one Robocopy at a time.

    So what i have together is 2 powershell scripts. One uses the NetApp Powershell cmdlets to return a list of shares to me. I can then do 'foreach' on that list and call the 2nd powershell script that does all of the robocopy (to NULL) parsing to retrieve the total size, file count, etc... and output a file per share. This all works great, except when I run it, It's a fire and forget process launcher in the first script. I ran it on a 16 core blade with 32gb of ram and it handled it ok.... but technically I spawned close to 600 powershell windows all at once, all starting robocopy. I almost immediately went from zero cpu/memory consumption to 100% CPU across all cores and about 17gb of RAM. I knew this would happen, i was just curious 🙂

    So what I want to know is... If I have a list of shares.. is there a way to specify a number (lets say 8) and have powershell chunk through the list of shares spawning up to 8 instances at a time... when one finishes... start another... so there are always 8 running until it gets to the end of the list? This should cut the 12 hour runtime of a single script scanning the whole appliance down to mere hours. I want to schedule this daily for trending and when I have a variable timeframe nearly 12 hours or so in length, the results will be skewed for the trending i wanna do.

    Thoughts?

    To Recap: Basically.. 680 'things', I want to do something against 8 at once.. not single threaded, until all 680 are done.

    Brian

  • #33041

    Dan Potter
    Participant

    Sapien has a really cool function I've been using lately.

    Alternatively, Boe Prox has written some cool stuff on the subject that I've used in some of my scripts.

  • #33042

    Daniel Krebs
    Moderator
  • #33043

    Brian B
    Participant

    Thanks guys. i'll do some quick reading on those and report back later today. Much appreciated!

    Brian

  • #33057

    Brian B
    Participant

    That RunSpaces code works perfectly! Took me a bit to figure out what he was doing in the script. Ripped out his Scriptblock and replaced with my own, changed some of the parameters, etc... and voila! Works like a charm!

    THANK YOU!

    Script i'm talking about is this one:
    https://gallery.technet.microsoft.com/scriptcenter/Speedy-Network-Information-5b1406fb

    Brian

You must be logged in to reply to this topic.