star-job | A referral was returned from the server

This topic contains 12 replies, has 3 voices, and was last updated by Profile photo of Dan Potter Dan Potter 6 months, 1 week ago.

  • Author
    Posts
  • #39227
    Profile photo of Recco Bucceri
    Recco Bucceri
    Participant

    Hey guys,

    I have a case where i need to convert the immutable ID of a office 365 user into a GUID and then scan across the forest using that GUID and once located in the correct domain, collect some additional info.

    The problem i have is there are 20+ domains and because I am scanning through around 37 000 users, it can take a considerable amount of time.

    So I started to research the idea of using jobs and run multiple threads at the same time.
    My source material was:

    http://www.get-blog.com/?p=22

    The potential is amazing, but for the life of me i cant get it to work as intended.

    The main script looks as follows:

    Param($ScriptFile = "D:\Docs\Scripts\Speed_Test.ps1",
    $SourceFile = (Import-Csv -Path 'D:\Docs\CSV\LicensedUsers_ImmTest.csv'),
    $MaxThreads = 200,
    $SleepTimer = 5)

    "Killing existing jobs . . ."
    Get-Job | Remove-Job -Force
    "Done."

    $i = 0

    foreach($User in $SourceFile)
    {

    While ($(Get-Job -state running -Verbose).count -ge $MaxThreads)
    {

    Write-Progress -Activity "Creating User List" -Status "Waiting for threads to close" -CurrentOperation "$i threads created – $($(Get-Job -state running).count) threads open" -PercentComplete ($i / $SourceFile.count * 100)

    Start-Sleep -Milliseconds $SleepTimer
    }

    $I++
    Start-Job -FilePath "D:\Docs\Scripts\Speed_Test.ps1" -ArgumentList $User -Name $User.ImmutableId | Out-Null

    Write-Progress -Activity "Creating User List" -Status "Waiting for threads to close" -CurrentOperation "$i threads created – $($(Get-Job -state running).count) threads open" -PercentComplete ($i / $SourceFile.count * 100)

    }

    $Data = ForEach ($Job in (Get-Job))
    { Receive-Job $Job
    #Remove-Job $Job
    }

    $Data | select UserPrincipalName, DistinguishedName |
    Export-Csv -NoTypeInformation -Delimiter ";" -Path 'D:\Docs\Reports\SSRP_Group_SourceInfo_SpeedTest.csv' -Append

    Script referenced in the start-job:

    $Domains = "Domain1", "Domain2","Domain3", Domain 4-20..

    $Decode = [system.convert]::frombase64string($args[0].ImmutableId)
    $GUID = [GUID]$Decode

    foreach($Domain in $Domains)
    {

    $UserInfo = Get-ADUser -Identity $GUID -Server $Domain -Properties UserPrincipalName, DistinguishedName

    }

    If i run the referenced script mentioned in the Start-job independantly it works as intended, but when i attempt to run it through a start-job i get no results and the constant error code:

    A referral was returned from the server
    + CategoryInfo : ResourceUnavailable: (52ad7263-504a-87c8-3224-d0c50ad12ea0:ADUser) [Get-ADUser], ADReferralException
    + FullyQualifiedErrorId : A referral was returned from the server,Microsoft.ActiveDirectory.Management.Commands.GetADUser
    + PSComputerName : localhost

  • #39469
    Profile photo of Don Jones
    Don Jones
    Keymaster

    Would you mind posting this as a Gist, and then pasting that Gist URL into a reply here, so it'll be formatted properly? I'm having some trouble following it. Either that, or use the in-forums code formatting, as shown in the three bullet points immediately above the Reply textbox.

  • #39779
    Profile photo of Recco Bucceri
    Recco Bucceri
    Participant

    Hey Don,

    Gist is sadly blocked at my office, but i will make a plan and repost it.

  • #39823
    Profile photo of Dan Potter
    Dan Potter
    Participant

    It shouldn't take that long to enumerate 37k users. I would consider changing your source anchor to one of the exchangecustomattributes and stamping it with the immutableid. As such you wouldn't have to perform the conversion. You could probably index the attribute and make it much faster as well.

    Please post the contents of your ...speedtest ps1.

  • #39849
    Profile photo of Dan Potter
    Dan Potter
    Participant

    I think I pieced it back together. .. you aren't going to save any time starting a new-job foreach user. Seperate the jobs by domain and return all users with an expression to convert the guid TObase64. Filter users if possible.

    Not complete code but to give you an idea.

    foreach($domain in $domains){

    start-job -name domainone -scriptblock {

    get-aduser -Filter * -server ... |select samaccountname,@{l='immutableid';e={[System.Convert]::ToBase64String($_.objectguid.tobytearray())}}

    }

    }

    # write something for while jobs are running

    foreach($domain in $domains){
    Receive-Job -Name domainone -outvariable $domain

    }

    #now we have separate lists of users for each domain. we can now loop through our csv to find which they are a member of.

  • #39851
    Profile photo of Dan Potter
    Dan Potter
    Participant

    Not seeing an edit option with the new layout.

    $allusers = Get-Job domain* | %{$domain = $_.name; Receive-Job $_ -keep | select samaccountname,immutableid,@{l='domain';e={$domain}}}

  • #39915
    Profile photo of Recco Bucceri
    Recco Bucceri
    Participant

    Hey Dan,

    So here is the problem, we did an investigation prior and sadly a lot of the custom attributes are being used for arb reasons and basically have been told we cant use them... Also we couldnt 100% use the UserPrincipleName as an anchor, because we are still finding cases where a user moved to a new office and instead the AD account was duplicated in another domain...

    So thats why i went the route of collecting all the licensed users in the tenant and used their immutableID (converted to GUID) as the anchor to do our searches. So after some digging and chatting to a few people i managed to sort out my original issue and got the script to work. So a job was created per user, with a max job count of 20. It was then recommended that i should bundle users into batches and instead send that off, instead of 1 user per job.

    So i started amending my script and got it to build an array for every 5 users (currently, still testing. I want to increase this count later) and then process them. The problem i have now got, is of the last couple lines DONT equal 5, they get skipped. So now i am tearing my hair out trying to work out if the Variable CANT equal 5 to process anyways.

    A snippet of the script is as follows. I plan to post the whole script once i have sorted this out for some constructive criticism:

    $myBucket = @()

    foreach ($listitem in $mylist ){

    $myBucket += $listitem

    $tracker ++

    if ($tracker -eq 5 )
    {}

  • #39925
    Profile photo of Dan Potter
    Dan Potter
    Participant

    Look up queue dequeue methods for powershell. The individual call to AD for info is still going to be your bottleneck as well as the report from msol.

    If you're not too far into your implementation I highly suggest a sql db as a frontend for licensing, I did this for my company and it's helped in more ways than one.

    Good Luck, Dan

  • #40094
    Profile photo of Recco Bucceri
    Recco Bucceri
    Participant

    So this is the final product. With a little tweaking the script completes 37000 users in 28mins, but i think i can still get that a little lower.

    I have still got a lot to learn, so always open to constructive crits. Maybe there is an easy way to get the same results or write the script better. Either this was the most fun i have had in a while and super happy with the results!

  • #40102
    Profile photo of Recco Bucceri
    Recco Bucceri
    Participant

  • #40154
    Profile photo of Dan Potter
    Dan Potter
    Participant

    With previously mentioned logic I'm returning the info for 87,000 users across six domains in under 7 minutes.

    Get all users first, loop through csv to match user to domain.

    vs.

    Make 30 some thousand calls to AD..

    The script is also five lines of code.

    • #40329
      Profile photo of Recco Bucceri
      Recco Bucceri
      Participant

      Hey Dan,

      Thanks for all the help and input, really appreciate it!

      Could i ask if you wouldnt mind posting your script, im curious to see how you are going about it?

  • #40340
    Profile photo of Dan Potter
    Dan Potter
    Participant

    Filter pertains to my org, hope you have something similar to designate standard users. Up to you to put in some error handling, I don't have any users that don't match.

     
    
    $csv = import-csv msolusers.csv
    
    'domainone.com','domaintwo.com' | % {start-job -ScriptBlock {param($domain);get-aduser -filter "employeenumber -eq 'primary'" -properties CanonicalName -Server $domain |
    
    select CanonicalName,userprincipalname,@{l='immutableid';e={[System.Convert]::ToBase64String($_.objectguid.tobytearray())}}} -Name $_ -ArgumentList $_}
    
    while((get-job).state -match 'running'){start-sleep 10}
    
    $allusers = Get-Job *.com | %{ Receive-Job $_ }
    
    $allusers | %{ $i = $_; $msuser = $csv | ? {$_.msolid -eq $i.immutableid}; $i | select CanonicalName,userprincipalname,immutableid,@{l='msolupn';e={$msuser.userprincipalname}}}
    
    

You must be logged in to reply to this topic.