Parallel usage of imported PSSession - Runspaces/AsJob/Invoke-Parallel

This topic contains 1 reply, has 2 voices, and was last updated by Profile photo of Max Kozlov Max Kozlov 1 month, 1 week ago.

  • Author
    Posts
  • #55808
    Profile photo of Frederik Leed
    Frederik Leed
    Participant

    Hi

    I've been having this headache for a long time now so now i'm asking for assistance.

    -Short issue description: Need to run in parallel using Imported PSsession to performance optimize code.

    First is the extremely simplyfied version of my script where i connect and get all mailboxes from exchange online and using those mailboxes, gather further information about each mailbox. Issue here being since this is exchangeonline, stuff takes time. Also some commands are time consuming. In average each query takes 5-15 sec. Which off-course is quite a lot when you have a lot of mailboxes.

    Function QueryExchangeOnline{
        Param($mailbox)
        
        #Do multiple heavy query's like:
        # -Get Calendar permissions
        # -Get-RecipientPermission
        # -Get-MailboxPermission
        # -Get-MailboxFolderStatistics
        # -Get-MailboxFolderPermission
        # -Get-Casmailbox
    
        $mbx = get-mailbox -Identity $mailbox.Identity
    
        $obj = @{}
        $obj.identity = $mbx.identity
        #emulating commands described above
        $randomnumber = 5..15 | Get-Random
        Sleep -Seconds $randomnumber
    
        $obj
    }
    
    #Get credentials to connect to exchangeonline
        $creds = Get-Credential
    
    #Create and import Exchangeonline PSSession
        $session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://outlook.office365.com/PowerShell-LiveID?PSVersion=4.0" -Credential $creds -Authentication Basic -AllowRedirection
        $null = Import-PSSession $session -AllowClobber -DisableNameChecking
    
    #Get all mailboxes from ExchangeOnline
        $allmailboxes = Get-Mailbox -ResultSize Unlimited
    
    #Do various querying and return hashtable to pipeline
        foreach($m in $allmailboxes){
            QueryExchangeOnline -mailbox $m
        }
    

    I've tried lots of stuff and even have something working in paralel using powershell runspaces. Problem is, it creates a nye pssession for exchange online for each query, which just makes it even slower.

    #Get credentials to connect to exchangeonline
        $creds = Get-Credential
    
    #Create and import Exchangeonline PSSession
        $session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://outlook.office365.com/PowerShell-LiveID?PSVersion=4.0" -Credential $creds -Authentication Basic -AllowRedirection
        $null = Import-PSSession $session -AllowClobber -DisableNameChecking
    
    #Get all mailboxes from ExchangeOnline
        $Array = Get-Mailbox -ResultSize Unlimited
    
    #Create empty Array
        $RunspaceCollection = @()
    
    #Open Define the number of processes the pool should allow.  minimum and maximum values (1 and 5)
        $RunspacePool = [RunspaceFactory]::CreateRunspacePool(1,20)
        $RunspacePool.Open()
    
    #Define the actual work each runspace will perform.
        $ScriptBlock = {
            Param($creds,$mailbox)
    
                $so = New-PSSessionOption -SkipRevocationCheck
    	        $session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://outlook.office365.com/PowerShell-LiveID?PSVersion=4.0" -Credential $creds -Authentication Basic -AllowRedirection -SessionOption $so
    	        $null = Import-PSSession $session -AllowClobber -DisableNameChecking
    
            #Do multiple heavy query's like:
            # -Get Calendar permissions
            # -Get-RecipientPermission
            # -Get-MailboxPermission
            # -Get-MailboxFolderStatistics
            # -Get-MailboxFolderPermission
            # -Get-Casmailbox
    
            $mbx = get-mailbox -Identity $mailbox.Identity
    
            $obj = @{}
            $obj.identity = $mbx.identity
            #emulating commands described above
            $randomnumber = 5..15 | Get-Random
            Sleep -Seconds $randomnumber
    
            $obj
        }
    
    Foreach($mailbox in $Array){
    
       #Create a PowerShell object to run add the script and argument.
    
       $Powershell = [PowerShell]::Create().AddScript($ScriptBlock).AddArgument($creds).AddArgument($mailbox)
    
       
       #Specify runspace to use
       $Powershell.RunspacePool = $RunspacePool
    
       #Create Runspace collection
       [Collections.Arraylist]$RunspaceCollection += New-Object -TypeName PSObject -Property @{
           Runspace   = $PowerShell.BeginInvoke()
           PowerShell = $PowerShell
           Item       = $mailbox
       }
     }
    
    
    #Check the RunSpace and validate that all of the tasks in the pool have completed.  Once completed, send 'EndInvoke' to each one.
    #     Once all of the tasks are completed, remove the runspace.
    
     While($RunspaceCollection){
    
     Foreach($Runspace in $RunspaceCollection.ToArray()){
    
      If($Runspace.Runspace.IsCompleted){
       $Runspace.PowerShell.EndInvoke($Runspace.Runspace)
       $Runspace.PowerShell.Dispose()
       $RunspaceCollection.Remove($Runspace)
      }
     }
    }
    

    My script in it's current form takes about 15-20 hours, why running in parallel would be a very welcome optimization that could optimize by factor <20.

    Has someone done something like this or does someone have any pointers, it would be great! 🙂

  • #55850
    Profile photo of Max Kozlov
    Max Kozlov
    Participant

    I'm found only one way – direct remote session without importing

    $session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://outlook.office365.com/PowerShell-LiveID?PSVersion=4.0" -Credential $creds -Authentication Basic -AllowRedirection
    
    'user1','user2' | Start-RSJob { $mb = $_; Invoke-Command -Session $using:session { get-mailbox $args[0] } -ArgumentList $mb }
    

    .....
    Hmm, forget it, with many users I get "session busy"
    I think you may only pre-open several sessions and use it with some hand-made throttling

You must be logged in to reply to this topic.