Powershell robocopy one to many multithreaded

This topic contains 5 replies, has 2 voices, and was last updated by Profile photo of Saeed Ahmad Saeed Ahmad 2 years ago.

  • Author
    Posts
  • #26911
    Profile photo of Saeed Ahmad
    Saeed Ahmad
    Participant

    I am a PS binger, I am trying to get few poweshell lines working. The script read simple text file having list of remote server. using foreach loop, I am trying to copy a folder from single source to a destination on all servers on list. instead of running the copy sequentially, I need to run multiple concurrent copies about 30-50 robocopy instances. Can some one help.? thanks
    *********************
    $RP01 = Get-Content C:\admin\_SF-Phase2\Multi-Robocopy-Test\RP01.txt
    $MySource = """\\MasterServer\Public\SQL"" "

    #$MyDest = """\\$RP01\C$\Admin\SQL"" "

    $EXE = "robocopy "

    $Options = "/r:0 /w:1 /log:C:\Admin\_SF-Phase2\Multi-Robocopy-Test\$RP01.log"

    $RoboCmd = $Exe + $MySource + $MYDest + $Options

    foreach($a in $RP01){

    if (Test-Connection $a -Quiet)
    start-job Invoke-Expression { $EXE + $MySource + $MyDest + "/r:1 /w:1 /log:C:\Admin\_SF-Phase2\Multi-Robocopy-Test\$a.log" }

    }

    I cant get this to work. once again thanks in adance

  • #26913
    Profile photo of Dave Wyatt
    Dave Wyatt
    Moderator

    I'd avoid using Invoke-Expression here, as there's no real need for it (and it's sometimes a very bad idea, as it can open you up to code injection attacks.)

    The tricky part of what you're doing involves passing variables to the script block that's used by Start-Job. Those jobs will be running in a separate process, which won't have your source / dest variables available. In PowerShell 3.0 and later, the easy way to deal with this is to use the $using:variable syntax, which will automatically pass those values to the job. That would probably look something like this:

    $RP01 = Get-Content C:\admin\_SF-Phase2\Multi-Robocopy-Test\RP01.txt
    $MySource = '\\MasterServer\Public\SQL'
    
    foreach($server in $RP01)
    {
        if (Test-Connection $server -Quiet)
        {
            Start-Job { robocopy $using:MySource "\\$using:server\C$\Admin\SQL" /r:1 /w:1 /log:C:\Admin\_SF-Phase2\Multi-Robocopy-Test\$using:server.log }
        }
    }
    

    You should probably also have a step that makes a local copy of \\MasterServer\Public\SQL before you upload it to the 30-50 other machines (or just run this script from \\MasterServer). Either way, you avoid a lot of extra network traffic when the copy of the files on MasterServer has to pass through your local workstation for each remote computer, instead of only once.

  • #26947
    Profile photo of Saeed Ahmad
    Saeed Ahmad
    Participant

    HI Dave. Thanks for your response. Its great feeling to have a response. (::)
    I must be missing the very point you mentioned in your response. I would appreciate if clarified further. I am using
    $RP01 = Get-Content C:\admin\_SF-Phase2\Multi-Robocopy-Test\RP01.txt
    $MySource = '\\metrofs01\Public\SQL'
    foreach($a in $RP01){

    start-job { robocopy "$Mysource" "\\$a\C$\Admin\SQL" + "/r:1 /w:1 /log:C:\Admin\_SF-Phase2\Multi-Robocopy-Test\$a.log"}

    }

    The script seem to start job for each sever in the text file, when I do get-job | fl it shows running command as
    Command : robocopy "$Mysource" "\\$a\C$\Admin\SQL" + "/r:1 /w:1 /log:C:\Admin\_SF-Phase2\Multi-Robocopy-Test\$a.log"
    It seem it is not resolving the variables. At one stage was able to resolve variable when I highlighted the script block and ran but same scrip block in start-job does no seem to resolve variables.
    Thanks a lot
    Cheers

  • #26949
    Profile photo of Dave Wyatt
    Dave Wyatt
    Moderator

    You haven't added the using: scope modifier inside your script block yet. Notice that in my example, it's robocopy $using:MySource, rather than robocopy $MySource. That "using:" part is what tells PowerShell to inject the values of your local variables when it creates the job.

  • #26950
    Profile photo of Dave Wyatt
    Dave Wyatt
    Moderator

    (There are other instances of "using:" in my example, not just for $mySource. That's just the one that I called out in my reply.)

  • #26986
    Profile photo of Saeed Ahmad
    Saeed Ahmad
    Participant

    Thank you, Thank you, Thank you.. Dave.
    It worked like a charm.
    Great stuff
    Cheers
    SA

You must be logged in to reply to this topic.