Invoke-Command Question on Remote File Server

This topic contains 6 replies, has 2 voices, and was last updated by Profile photo of Karson Van Meeteren Karson Van Meeteren 3 years, 5 months ago.

  • Author
    Posts
  • #12742
    Profile photo of Karson Van Meeteren
    Karson Van Meeteren
    Participant

    Hi Gang,

    I'm attempting to get a list of files on some of our file servers that are newer than a given date range. I've been through a couple different methodologies of gathering the objects, but ultimately, running the script block in an invoke-command proved substantially faster.

    My question is, once I invoke the command on the remote machine, I see it running through the remote systems task manager just fine. But then, I want to return those objects I've stored in the invoke-command variable back to the local system and run a foreach loop on it locally. I'm getting hung up, and my output isn't being returned to the host I'm running the script from and going to excel like planned.

    
    Invoke-Command -Session $sessionFS13 {$colFS13Files= Get-ChildItem d:\ -Recurse | where {$_.psiscontainer -eq $false -and $_.CreationTime -gt $date} | Select-Object fullname,length,creationtime}
    		Invoke-Command -Session $sessionFS13 {$colFS13Files}
    		foreach ($file in $colFS13Files) {
    						$filepath = $file.FullName 
    						$excelSheet4.Cells.Item($intRow,1) = $filepath
    						$size = $file.Length
    						$excelSheet4.Cells.Item($intRow,2) = "{0:N2}" -f ($size / 1MB)
    						$creationTime = $file.CreationTime
    						$excelSheet4.Cells.Item($intRow,3) = $creationTime
    						$owner = ((Get-Acl $filepath).owner) #$filepath | Get-Acl | select owner
    						$excelSheet4.Cells.Item($intRow,4) = $owner
    						$intRow ++					
    }#foreach
    		
    # add up length column (column B)
    			$range = $excelSheet4.usedRange
    			$row = $range.rows.count # Takes you to the last used row
    			$Sumrow = $row + 1 #last used row + 1
    			$r = $excelSheet4.Range("B2:B$row") # select the column to Add up
    			$functions = $appExcel.WorkSheetfunction #setup variable for excel to run a built-in function (next line)
    			$excelSheet4.cells.item($Sumrow,2) = $functions.sum($r) # this uses the Excel sum function defined above
    			$rangeString = $r.address().tostring() -replace "\$",'' # convert formula to Text
    			$excelSheet4.cells.item($Sumrow,1) = "Total Size (MB)" # Print string "Total Size" in column A & last row + 1
    			$excelSheet4.cells.item($Sumrow,2).Select() #Print calculation
    			$excelSheet4.range("a${Sumrow}:b$Sumrow").font.bold = "true" # makes text in last row bold
    			$excelSheet4.range("a${Sumrow}:b$Sumrow").font.size=12 # Changes the font size in last row to 12 points   
    

    Thanks!

    Karson

  • #12743
    Profile photo of Dave Wyatt
    Dave Wyatt
    Moderator

    Try it like this:

    $colFS13Files = Invoke-Command -Session $sessionFS13 { Get-ChildItem d:\ -Recurse | where {$_.psiscontainer -eq $false -and $_.CreationTime -gt $date} | Select-Object fullname,length,creationtime}
    
    foreach ($file in $colFS13Files) {
        # .. etc
    }
    

    In your original code, the $colFS13Files variable only existed in the remote session. By assigning the results of Invoke-Command to a local variable, then you can enumerate over them with a foreach loop.

  • #12775
    Profile photo of Dave Wyatt
    Dave Wyatt
    Moderator

    You're on the right track. You do want to use the -AsJob switch to Invoke-Command, and save the resulting Job object to a variable. However, you don't assign the results to (for example) $colFS10Files until you call Receive-Job. Whether you keep track of these in an array, hashtable or whatever is up to you. Here's an example using hashtables to map computer names to job objects (and then later to the jobs' results):

    $computerNames = 'Server01','Server02','Server03'
    
    $jobs = @{}
    $results = @{}
    
    foreach ($computer in $computerNames)
    {
        $jobs[$computer] = Invoke-Command -ComputerName $computer -AsJob -ScriptBlock { Do-Something }
    }
    
    Wait-Job -Job $jobs.Values
    
    foreach ($entry in $jobs.GetEnumerator())
    {
        $results[$entry.Key] = Receive-Job -Job $entry.Value
    }
    
    $results
    
    # At this point you can loop over $results and do things with the data.
    
  • #12778
    Profile photo of Karson Van Meeteren
    Karson Van Meeteren
    Participant

    So, the remote file servers are only on v2, so the -File parameter was my issue there. I'm sacrificing some efficiency by having to include $_.psiscontainer -eq $false in the pipeline, but until I get v3 or upgrade them to 2012 R2 and get PoSH v4, this'll have to do.

    That being said, I've fixed the array issues. Now I just need to figure out the collection of the receive-job to my variables...

    Thinking out loud, so ignore the rambling 🙂

  • #12744
    Profile photo of Karson Van Meeteren
    Karson Van Meeteren
    Participant

    Ahh – perfect. So simple, can't believe I missed that.

    Thanks, Dave.

  • #12774
    Profile photo of Karson Van Meeteren
    Karson Van Meeteren
    Participant

    Well, I'm back at it. This time, trying to optimize this as best as I can from a timeframe aspect. It seems that waiting on each Invoke-Command to finish on the current file server, output the results to Excel line-by-line, then proceed on to the remaining four servers one-by-one made the script finish in ~5 days.

    Brainstorming a little bit, I thought of trying to setup an array to run the invoke-commands in parallel on each server as a separate job. But, I'm still back to my original question of how to store the job's objects as a local variable.

    Maybe I'm way off in left field on this – is there something from an optimization standpoint that would work better?

    #region Setting Array of jobs and checking on their status
    $arrayJobs = @()
    $arrayJobs += $colFS10Files = Invoke-Command -ComputerName fileserver10 -AsJob -JobName fileserver10_Job -ScriptBlock {Get-ChildItem d:\home -Recurse -File | where {$_.CreationTime -gt ((Get-Date).AddDays(-7))} | Select-Object fullname,length,creationtime,@{n="owner";e={(Get-Acl $_.fullname).owner}}}
    
    $arrayJobs += $colFS11Files = Invoke-Command -ComputerName fileserver11 -AsJob -JobName fileserver11_Job -ScriptBlock {Get-ChildItem d:\ -Recurse -File | where {$_.CreationTime -gt ((Get-Date).AddDays(-7))} | Select-Object fullname,length,creationtime,@{n="owner";e={(Get-Acl $_.fullname).owner}}}
    
    $arrayJobs += $colFS12Files = Invoke-Command -ComputerName fileserver12 -AsJob -JobName fileserver12_Job -ScriptBlock {Get-ChildItem e:\home -Recurse -File | where {$_.CreationTime -gt ((Get-Date).AddDays(-7))} | Select-Object fullname,length,creationtime,@{n="owner";e={(Get-Acl $_.fullname).owner}}}
    
    $arrayJobs += $colFS13Files = Invoke-Command -ComputerName fileserver13 -AsJob -JobName fileserver13_Job -ScriptBlock {Get-ChildItem d:\ -Recurse -File | where {$_.CreationTime -gt ((Get-Date).AddDays(-7))} | Select-Object fullname,length,creationtime,@{n="owner";e={(Get-Acl $_.fullname).owner}}}
    
    $arrayJobs += $colFS14Files = Invoke-Command -ComputerName fileserver14 -AsJob -JobName fileserver14_Job -ScriptBlock {Get-ChildItem e:\ -Recurse -File | where {$_.CreationTime -gt ((Get-Date).AddDays(-7))} | Select-Object fullname,length,creationtime,@{n="owner";e={(Get-Acl $_.fullname).owner}}}
    
    #monitor to see when all jobs are completed
    $Complete = $false #$complete will be used to determine when all the jobs are completed
    while (-not $complete) { #checks array to see if the job status is 'running'
        $arrayJobsInProgress = $arrayJobs | 
            Where-Object { $_.State -match 'running' }
        if (-not $arrayJobsInProgress) { "All Jobs Have Completed" ; $complete = $true } 
    }
    
    $arrayJobs = Receive-Job #return all jobs in their respective local variables containing the collection of new files
    
    
  • #12777
    Profile photo of Karson Van Meeteren
    Karson Van Meeteren
    Participant

    Thanks for that example, Dave. It's getting me on the right path to eventually get each server stored in its own respective $colFS10files, $colFS11files, etc... variable.

    I think I'm getting hung up on the first foreach. I'd loop the creation of the jobs, except the invoke-command is different on several of them making the need to break them out into individual jobs, manually, the easiest way (for me) ;o)

    I "borrowed" the while (-not $complete) { chunk from Ed Wilson to replace your Wait-Job since I couldn't understand how to associate the -Job parameter for my entire custom/manual $arrayJobs and wait for them all, as a whole, to be completed.

    I'll try and nail down where my thought process is:
    1) after the last of my manual $arrayJobs entry is finished, I should have a wsmprovhost.exe process using a fair amount of CPU on each remote system as it runs the invoke-command as a job, right? I've estimated it should take at least few hours, as the biggest file server should be recursing through ~4TB of data
    a) If wsmprovhost.exe is indicative of the remote job running, then my array is bonky because that process never kicks off like it normally would on any of the servers. And, the job shows "completed" if I copy/paste a single job from the array into the console in a matter of 10-15 seconds.

    2) if I can get the jobs to stay in "running" status for a respectable amount of time, can I ditch the second "foreach" and manually change the $results variable into, for my example,:

         $colFS10files = Receive-Job -Job fileserver10_job
         $colFS11files = Receive-Job -Job fileserver11_job 

    ....then process data into Excel spreadsheet?
    In my head, I think I can. Once I get objects stored in my array, I'll know for sure.

    Thanks again, Dave.

    Edit, here's my new array without the incorrect variable usage:

    #region Setting Array of jobs and checking on their status
    $arrayJobs = @()
    $arrayJobs += Invoke-Command -ComputerName fileserver10 -AsJob -JobName fileserver10_Job -ScriptBlock {Get-ChildItem d:\home -Recurse -File | where {$_.CreationTime -gt ((Get-Date).AddDays(-7))} | Select-Object fullname,length,creationtime,@{n="owner";e={(Get-Acl $_.fullname).owner}}}
    
    $arrayJobs += Invoke-Command -ComputerName fileserver11 -AsJob -JobName fileserver11_Job -ScriptBlock {Get-ChildItem d:\ -Recurse -File | where {$_.CreationTime -gt ((Get-Date).AddDays(-7))} | Select-Object fullname,length,creationtime,@{n="owner";e={(Get-Acl $_.fullname).owner}}}
    
    $arrayJobs += Invoke-Command -ComputerName fileserver12 -AsJob -JobName fileserver12_Job -ScriptBlock {Get-ChildItem e:\home -Recurse -File | where {$_.CreationTime -gt ((Get-Date).AddDays(-7))} | Select-Object fullname,length,creationtime,@{n="owner";e={(Get-Acl $_.fullname).owner}}}
    
    $arrayJobs += Invoke-Command -ComputerName fileserver13 -AsJob -JobName fileserver13_Job -ScriptBlock {Get-ChildItem d:\ -Recurse -File | where {$_.CreationTime -gt ((Get-Date).AddDays(-7))} | Select-Object fullname,length,creationtime,@{n="owner";e={(Get-Acl $_.fullname).owner}}}
    
    $arrayJobs += Invoke-Command -ComputerName fileserver14 -AsJob -JobName fileserver14_Job -ScriptBlock {Get-ChildItem e:\ -Recurse -File | where {$_.CreationTime -gt ((Get-Date).AddDays(-7))} | Select-Object fullname,length,creationtime,@{n="owner";e={(Get-Acl $_.fullname).owner}}}
    

You must be logged in to reply to this topic.