Multithreaded file copy like robocopy (But not)

Welcome Forums General PowerShell Q&A Multithreaded file copy like robocopy (But not)

  • This topic has 4 replies, 3 voices, and was last updated 1 month ago by
    js
    Participant
    .
Viewing 4 reply threads
  • Author
    Posts
    • #186844
      Participant
      Topics: 4
      Replies: 6
      Points: 78
      Rank: Member

      Hey all,  I am new to the forum.  I have been writing PowerShell scripts for a couple of years now and am expanding my knowledge.  Recently, I found myself needing to copy files from a database server to a file share for backup purposes.  I wrote a function that copies the file, then verifies the hash value of the copy to the source.  It used to delete the original, but I removed that part of it.  What I want to do is to run the foreach loop in parallel to expedite the completion of the code.

      Feel free to contribute on my github (https://github.com/scriptingcaveman/Move-EBFile).

      Try{
          Get-ChildItem -Path $SourcePath -Recurse | ForEach-Object {
              Write-Verbose -Message 'Replacing source path with destination path'
              $DestinationFile = $_.FullName -replace [regex]::Escape($SourcePath),$DestinationPath
              $SourceFile = $_.FullName
              Try {
                  Copy-Item -Path $SourceFile -Destination $DestinationFile
              } Catch {
                  Write-Error -Message "File Copy not able to be completed."
              }
              IF((Get-Childitem -Path $DestinationFile).Name -eq (Split-Path -Path $SourceFile -Leaf)){
                  Try {
                      $FileHashTable = $null
                      $FileHashTable = [Ordered]@{"SourceFile" = $SourceFile;
                          "SourceHash"                    = (Get-FileHash -Path $SourceFile -Algorithm MD5).Hash;
                          "DestinationFile"               = $DestinationFile;
                          "DestinationHash"               = (Get-FileHash -Path $DestinationFile -Algorithm MD5).Hash
                      }
                      IF ($FileHashTable.SourceHash -ne $FileHashTable.DestinationHash) {
                          $FIlehashTable.Add("Result", $false)
                      }
                      Else {
                          $FilehashTable.Add("Result", $true)
                      }
                  } Catch{
                      Write-Error -Message "File Hash not able to be completed."
                   }
               }
               IF($FileHashTable.Result){
                   Write-Verbose "File copy completed and hash matches the source."
                   Write-Output  $FileHashTable
               } Else {
                   Write-Verbose "File copy hash does not match the source."
                   Write-Output $FileHashTable
               }
          }
      } Catch {
          Write-Verbose -Message 'Copy item script issue.'
      }
      }
      
    • #186856
      Participant
      Topics: 8
      Replies: 1274
      Points: 1,033
      Helping Hand
      Rank: Community Hero

      There is a foreach parallel as part of workflows or you can use a scriptblock. See below:

      https://stackoverflow.com/questions/52137980/copying-multiple-files-in-parallel-through-power-shell-without-using-any-third-p

    • #186865
      js
      Participant
      Topics: 27
      Replies: 716
      Points: 1,879
      Helping Hand
      Rank: Community Hero

      There's a foreach-object -parallel in powershell 7 too with -throttlelimit. https://devblogs.microsoft.com/powershell/powershell-foreach-object-parallel-feature/

    • #186997
      Participant
      Topics: 4
      Replies: 6
      Points: 78
      Rank: Member

      Thank you both.  That's exactly what I needed.

    • #187000
      js
      Participant
      Topics: 27
      Replies: 716
      Points: 1,879
      Helping Hand
      Rank: Community Hero

      There's start-job too, which starts a new process.

Viewing 4 reply threads
  • You must be logged in to reply to this topic.