Using ROBOCOPY with powershell script

Tagged: 

This topic contains 21 replies, has 4 voices, and was last updated by  Max Kozlov 1 month, 3 weeks ago.

  • Author
    Posts
  • #72992

    Dapinder Singh
    Participant

    Hi guys, need your guidance in sorting the following out.

    Scenario: I have a file server where new folders with files are created everyday. These folders size may range from few hundred MBs to GBs. For archiving purposes, my task is to copy yesterday's records (Folders and the files within) along with file permissions to another location. Here is my script:

    # Define the variables
    $today = get-date -Hour 0 -Minute 0 -Second 0 -Millisecond 0
    $yesterday = get-date $today.adddays(-1) -Hour 0 -Minute 0 -Second 0 -Millisecond 0

    # Getting output and Copy to another location with timestamps
    get-childitem -Path D:\data\SOURCEFOLDER -recurse | where {$_.CreationTime -gt (get-date $Yesterday) -and $_.CreationTime -lt $today} | ForEach-Object {Robocopy D:\data\SOURCEFOLDER '\\servername\destination' /E /COPY:DATSOU $_.Name}

    Issue: Everything works well until ForEach-Object, somehow ROBOCOPY doesn't copy the extracted content. Robocopy creates the whole folder structure from the SOURCE directory. I tried COPY-ITEM, but it has some limitations, so had to use ROBOCOPY. Any ideas what I am doing wrong here?

  • #73016

    Frank Tucker
    Participant

    I don't think it is powershell. It's the /e switch from robocopy

    • #73024

      Dapinder Singh
      Participant

      Hi Frank, I have tried /S switch too, it hasn't worked either. What i can see, robocopy is going through each and every folder (thousands) but not picking up the one delivered by powershell.

  • #73027

    Kiran P
    Participant

    Wrap the robocopy command into Invoke-Command and then run through each object.

    ForEach-Object {Invoke-Command -Command "Robocopy D:\data\SOURCEFOLDER '\\servername\destination' /E /COPY:DATSOU $_.Name"}
    

    Thank you.

    • #73030

      Dapinder Singh
      Participant

      Thanks Kiran, will try it on Monday and let you know.

    • #73148

      Dapinder Singh
      Participant

      Hi Kiran,
      It fails with the following error:

      Invoke-Command : Cannot bind parameter 'ScriptBlock'. Cannot convert the "Robocopy D:\data\SOURCEFOLDER 'C:\temp\testcopy' /E /COPY:DATSOU FOLDER1.Name" value of type "System.String" to type
      "System.Management.Automation.ScriptBlock".
      At line:4 char:176
      + ... mmand -Command "Robocopy D:\data\SOURCEFOLDER 'C:\temp\testcopy' /E /COPY:DATSOU ...
      + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      + CategoryInfo : InvalidArgument: (:) [Invoke-Command], ParameterBindingException
      + FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.PowerShell.Commands.InvokeCommandCommand

    • #73151

      Kiran P
      Participant

      Sorry, use Invoke-Expression, it's not Invoke-Command

    • #73157

      Dapinder Singh
      Participant

      It creates the entire folder structure from SOURCE at the destination, and then goes through each and every file in the subfolders which i don't want. Imagine going through hundreds and thousands of folders looking for content. What i want Robocopy to do is just pick the results produced by powershell and not to look into every folder.

    • #73162

      Kiran P
      Participant

      Then remove -recurse flag from from Get-ChildItem in your script.

  • #73067

    Frank Tucker
    Participant

    don't use the /s switch either... you code copied just the files for me with just removing the /e switch

    • #73096

      Dapinder Singh
      Participant

      Thanks Frank. I think the U (auditing) switch in COPY /DATSOU could be at fault too. Anyways, will try and let you know.

    • #73159

      Dapinder Singh
      Participant

      Tried this too, no luck. I did a test run on files, it works great, but when folders and subfolders get involved, it doesn't do the job.
      The interface and logs says it is copying files, but the destination has got nothing. Checked the disk space to see if some change is taking place somewhere else, but no. Any other suggestions?

  • #73168

    Max Kozlov
    Participant

    why you don't use robocopy's /MAXAGE /MINAGE switches and play with get-childitem ?
    you can generate values for these switches with PS and let ROBOCOPY filter files.

    and, for better parameter handling try to launch robocopy this way

    $parameters = @($source, $destination)
    if (...) { $parameters += '/MaxAge:{0}' -f $age }
    # [...]
    & robocopy $parameters
    
    • #73216

      Dapinder Singh
      Participant

      Hi Max, can you please elaborate it bit further, sorry I don't have scripting background. What is IF (...) statement doing? Thanks.

  • #73232

    Max Kozlov
    Participant

    "If" intend the non-binding of adding some parameters to robocopy 🙂
    if you will be satisfied not creation but modification time, you need something like "robocopy source destination /minage:1 /maxage:2" without powershell at all
    or look at other examples https://social.technet.microsoft.com/wiki/contents/articles/1073.robocopy-and-a-few-examples.aspx

    the main message: be best way to call robocopy: use "& robocopy $parameterarray" syntax

    because robocopy doesn't support creationtime nor file-with-path selection,
    for exactly your variant I can suggest this:

    $source= 'd:\1'
    $destination= 'd:\2'
    $today = (Get-Date).Date
    $yesterday = $today.AddDays(-1)
    get-childitem $source -Recurse | where-object { $_.CreationTime -gt $yesterday -and $_.CreationTime -lt $today } | Foreach-Object { robocopy (Split-Path $_.FullName) $destination $_.Name }
    

    It copies file WITHOUT folder structure into destination
    for maintaining folder structure there is need some magic for destination folder tree creation

  • #73247

    Max Kozlov
    Participant

    and the magic is:

    get-childitem $source -Recurse |
    where-object {
        $_.CreationTime -gt $yesterday -and $_.CreationTime -lt $today
    } | Foreach-Object {
        $s = (Split-Path $_.FullName)
        $d = $destination + $s.Substring($source.Length)
        robocopy $s $d $_.Name
    }
    

    the additional switches for robocopy at you taste

    • #73259

      Dapinder Singh
      Participant

      I've been away from the office today, will give it a go tomorrow and let you know the outcome. Hopefully, it works!

    • #73406

      Dapinder Singh
      Participant

      Hi Max,

      The 2nd script worked, but i think we are missing something from it. It took 3 hrs 20 mins to copy just 4.56 GB of data from 1 SATA volume to another. Normal copy & paste takes only approx 2 mins.
      I started the transcript to get some logs and found the following:

      Host Application: C:\Windows\system32\WindowsPowerShell\v1.0\PowerShell_ISE.exe
      Process ID: 3420
      **********************
      Transcript started, output file is C:\temp\logs.txt
      Transcript started, output file is C:\temp\logs.txt
      Transcript started, output file is C:\temp\logs.txt
      Transcript started, output file is C:\temp\logs.txt
      ——————————————————————————-
      ——————————————————————————-
      ——————————————————————————-
      ——————————————————————————-
      ROBOCOPY :: Robust File Copy for Windows
      ROBOCOPY :: Robust File Copy for Windows
      ROBOCOPY :: Robust File Copy for Windows
      ROBOCOPY :: Robust File Copy for Windows
      ——————————————————————————-
      ——————————————————————————-
      ——————————————————————————-
      ——————————————————————————-
      Started : Thursday, 22 June 2017 9:54:20 AM
      Started : Thursday, 22 June 2017 9:54:20 AM
      Started : Thursday, 22 June 2017 9:54:20 AM
      Started : Thursday, 22 June 2017 9:54:20 AM
      Source : D:\data\SOURCEFOLDER\
      Source : D:\data\SOURCEFOLDER\
      Source : D:\data\SOURCEFOLDER\
      Source : D:\data\SOURCEFOLDER\
      Dest : C:\temp\testcopy\
      Dest : C:\temp\testcopy\
      Dest : C:\temp\testcopy\
      Dest : C:\temp\testcopy\
      Every line in log file is x4. Does that mean, it is copying multiple (x4) times? I thought it could be the recurse parameter, however after removing the parameter the script would not copy any folder. Thoughts?

  • #73417

    Max Kozlov
    Participant

    Don't know why it repeat many times but this code really call robocopy one time per one file, this is the bottleneck.
    Seems you start-transcrpt somewhere inside cycle but nut before script run

    So, as I'm already said, you should see if you can use Modification time or step away from robocopy.

    what problems you have with copy-item ?
    If it is lfn sopport look on PSAlphaFS module

    • #73549

      Dapinder Singh
      Participant

      I started transcript at the beginning of the script. The problem i had with Copy-item was, it wouldn't copy the timestamps; unless you have any suggestions.

  • #73633

    Dapinder Singh
    Participant

    Any more thoughts?

  • #73642

    Max Kozlov
    Participant

    if file attrubutes is the only problem

You must be logged in to reply to this topic.