Memory optimization

This topic contains 14 replies, has 5 voices, and was last updated by Profile photo of Recco Bucceri Recco Bucceri 6 months, 2 weeks ago.

Viewing 15 posts - 1 through 15 (of 15 total)
  • Author
    Posts
  • #36230
    Profile photo of Recco Bucceri
    Recco Bucceri
    Participant

    Hey all,

    So a new issue i am encountering when working with large data, is my servers running out of memory and then the script grinding to a slow pace.

    The script i am running is as follows:

    $SourceFile = Import-Csv -Delimiter ";" -Path

    foreach($User in $SourceFile)
    {

    $HomeServer = Get-QADObject -Identity $User.DistinguishedName -IncludeAllProperties

    If($HomeServer.msExchHomeServerName.Length -eq 0)
    {

    $User |select DistinguishedName, userPrincipalName, mail, ObjectClass |
    Export-Csv -NoTypeInformation -Delimiter ";" -Path Forest_msExchHomeServerName_Empty.csv' -Append

    }

    }

    Now I am currently scanning around 40000 users and my server (which has 16gb memory) has maxed out and the script is now running extremely slowly. Now i suspect it has to do with my variable management and maybe i need to clear the variable, but i am not to sure.

    How can i fine tune this script so that it doesn't consume so much memory?

    And does anyone have any good articles around performance tweaks that can be used in PowerShell scripting?

    #36231
    Profile photo of MichaelKF
    MichaelKF
    Participant

    Hi Recco,

    here are some suggestions you could try (see comments)

    Since this script generates a PSObject, you are able to output the content however you want, for example:
    Script.ps1 | Export-CSV -Path file.csv -Delimiter ";"
    or
    Script.ps1 | Out-GridView
    ...

    You could also measure the runtime duration with "Measure-Command". Would be interesting to see if it really improves and if so, how much.

    Regards,
    Michael

    #36242
    Profile photo of Dave Wyatt
    Dave Wyatt
    Moderator

    Few things here:

    1. You're importing your entire CSV into memory and holding it there. If it's a very large file, that can be a problem.

    2. You're using the Quest AD module. While I don't have personal experience with this, I have found several posts online stating that this module seems to cause huge memory usage problems. Even when your script doesn't look like it should be causing this, the module seems to hold onto object references so they don't get properly garbage collected.

    3. In conjunction with #2, you're using that -IncludeAllProperties switch. So every object that the QAD module is holding in memory will be that much bigger.

    #36246
    Profile photo of Jeremy Murrah
    Jeremy Murrah
    Participant

    The best way I've found to save on memory usage is to stream everything through one long pipeline. It's not too pretty, probably takes longer to run, but the memory usage is capped quite a bit, especially for those big queries. So something like this

    import-csv -delimiter ';' -path  | 
        get-qadobject -identity $_.distinguishedname -properties mail, objectclass, homeserver |
            ?{$_.msexchhomeservername.length -eq 0} |
                select distinguishedname, userprincipalname, mail, objectclass |
                    export-csv -NoTypeInformation -Delimiter ';' -path 'Forest_msExchHomeServerName_Empty.csv'
        
    
    #36262
    Profile photo of Recco Bucceri
    Recco Bucceri
    Participant

    Thanks for the responses!

    I am working in an environment that has 26 domains in the forest. So the problem i am having, is if i use "get-adobject" it wont search cross domains even if i reference to it through the DN. I need to specify -server contoso1.directory.com and only then will it search for the object in that domain.

    Using the DN, how can i change the -server to match the domain.

    So for example if the DN is "OU=Users,DC=Contoso1,DC=Directory,DC=com", then the switch would be contoso1.directory.com?

    My 1st thought is an "If" statement, followed by "Else", but to do that 26 times seems a little cluttered. I part of me is saying i can use regex somehow, but i am still very new to regex and would need to do more research.

    How best can i match the "get-abobject -server" to domain, based on the info in the DN text?

    #36265
    Profile photo of Dave Wyatt
    Dave Wyatt
    Moderator
    $server = $user.DistinguishedName -split ',' -match '^DC=' -replace '^DC=' -join '.'
    Get-ADUser $user.DistinguishedName -Server $server
    
    #36266
    Profile photo of Bob McCoy
    Bob McCoy
    Participant

    Is your domain name always 3-deep, e.g., contoso1.directory.com?

    #36267
    Profile photo of Recco Bucceri
    Recco Bucceri
    Participant

    @dave
    Cool let me mess around with that.

    @Bob
    Always

    #36268
    Profile photo of Bob McCoy
    Bob McCoy
    Participant

    If it's always the same depth, then you could do a RegEx match and then replace.

    # sample data
    $dnList = @"
    CN=USER_TEMPLATE,OU=sierra,DC=foo,DC=tango,DC=local
    CN=DOC-VIEWER,OU=Users,OU=sierra,DC=foo,DC=tango,DC=local
    CN=Ricoh Copier,OU=Users,OU=sierra,DC=bar,DC=tango,DC=local
    CN=Scan User,OU=Users,OU=sierra,DC=hq,DC=tango,DC=local
    CN=Sonic Wall,CN=Managed Service Accounts,DC=foo,DC=tango,DC=local
    CN=Orderdesk,OU=Web contacts,OU=sierra,DC=foo,DC=tango,DC=local
    CN=Technicalsupport,OU=Web contacts,OU=sierra,DC=bar,DC=tango,DC=local
    CN=tango SALES,OU=Web contacts,OU=sierra,DC=foo,DC=tango,DC=local
    CN=Amanda Bines,OU=Users,OU=sierra,DC=foo,DC=tango,DC=local
    "@ -split "`r`n"
    
    $pattern = ".+,DC=(\w+),DC=(\w+),DC=(\w+)"
    foreach ($dn in $dnList)
    {
        if ($dn -match $pattern)
        {
            $dn -replace $pattern, '$1.$2.$3'
        }
    }
    #36302
    Profile photo of Recco Bucceri
    Recco Bucceri
    Participant

    So this is the final script i am now testing:

    Get-PSSnapin -Registered | Add-PSSnapin

    $SourceFile = Import-Csv -Delimiter ";" -Path Import_file.csv

    $SourceFile | ForEach-Object{

    $server = $_.DistinguishedName -split ',' -match '^DC=' -replace '^DC=' -join '.'

    $HomeServer = Get-ADObject -Identity $_.DistinguishedName -Server $server -Properties msExchHomeServername, DistinguishedName, userPrincipalName, mail, ObjectClass

    If($HomeServer.msExchHomeServerName.Length -eq 0)
    {

    $Report = @()
    ForEach-Object{
    $Report += New-Object psobject -Property @{
    'DistinguishedName' = $HomeServer.DistinguishedName
    'userPrincipalName' = $HomeServer.userPrincipalName
    'mail' = $HomeServer.mail
    'ObjectClass' = $HomeServer.ObjectClass
    }

    }

    }

    }

    My servers memory (16gb) was constantly getting maxed out. So I did some research about when to use Foreach vs Foreach-object, and it seems that in my case using Foreach-object is better. It wont be as fast as Foreach, but will be better on memory usage.

    Is there a way to only search a X number of lines at a time from the $Sourcefile, clear all variables and the process X+1 and so forth?

    #36315
    Profile photo of Jeremy Murrah
    Jeremy Murrah
    Participant

    You're still storing everything in memory for the duration of your script. every ad object, every output property, every list. Try this code:

    Import-csv -Delimiter ";" -Path Import_file.csv | 
        ForEach-Object { Get-adobject -Identity $_.distinguishedname -server $($_.DistinguishedName -split ',' -match '^DC=' -replace '^DC=' -join '.') -Properties msExchHomeServername, DistinguishedName, userPrincipalName, mail, ObjectClass} | 
        where-object { $_.msExchHomeServerName.Length -eq 0} |
        Select-Object distinguishedname, userprincipalname, mail, objectclass
    

    Note there are no equal signs, everything just gets sent down the pipeline. The select-object will create your pscustom object for you, and the where-object is the pipeline equivalent of your if statement, it just filters. Should have much better memory utilization. It will output to the screen, but if you want it to a file just add a pipe to export-csv at the end.

    #36358
    Profile photo of Recco Bucceri
    Recco Bucceri
    Participant

    niiiiiiice!!!

    Thanks Jeremy, ill test that!

    Again, thanks for all the help all!

    #36486
    Profile photo of Recco Bucceri
    Recco Bucceri
    Participant

    @dave
    In the following "-match '^DC=' -replace '^DC='"

    What does the "^" actually stand for? I have been doing some syntax searches and haven't found anything yet.

    #36488
    Profile photo of Dave Wyatt
    Dave Wyatt
    Moderator

    -match and -replace in PowerShell use regular expressions, and ^ is an anchor in regex which matches the beginning of a string (or of a line, depending on how you've configured the options.) http://www.regular-expressions.info/anchors.html for more info. 🙂

    #36489
    Profile photo of Recco Bucceri
    Recco Bucceri
    Participant

    As always a massive help!

    Thanks again!

Viewing 15 posts - 1 through 15 (of 15 total)

You must be logged in to reply to this topic.