Performance reading large array >40K entries

Welcome Forums General PowerShell Q&A Performance reading large array >40K entries

Viewing 4 reply threads
  • Author
    Posts
    • #270322
      Participant
      Topics: 1
      Replies: 3
      Points: 18
      Rank: Member

      Hi

      I’m trying to find the fastest way to search a large array (40K entries) for a value. However, I’m struggling with the way in which the array is working.

      I read in all AD users as shown below

      <span style="font-size: 8pt"><code>$AllADUsers = Get-ADUser -Filter &quot;*&quot; -properties SAMAccountName, DisplayName, UserPrincipalName, Company, Office, Department, Manager,  Description, Created, LastLogonDate, EmployeeType,Info 
      <span style="font-size: 8pt"><code>-Server $ADServer -Credential $c
      <span style="font-size: 8pt"><code>|select SAMAccountName, DisplayName, UserPrincipalName, Company, Office, Department, Manager,
      <span style="font-size: 8pt"><code>Description, Created, LastLogonDate, EmployeeType,Info

      Now this is why I have a query

      If I search the array using the .Contains method it finds the entry in around 2ms

      <span style="font-size: 8pt">$AllADUsers.Contains($SearchID)</span>
      But if I then try to pull the data for the record using the where method it takes 1 second
      <span style="font-size: 8pt">$AllADUsers.Where({$_.UserPrincipalName -eq "$SearchID"})</span>

      So why can it find the value in 2ms using contains but then take more than 1 second to read the actual record. Its as if it is using a different search algorithm in the where which takes longer.

      In fact it was faster to use Get-AdUser for each search using on 900ms than to pull it from the array.

      I’ve also tried other variations to no avail, such as:

      <span style="font-size: 8pt">$AllADUsers | where-object {$_.UserPrincipalName -eq $SearchID}</span>

       

      Any help gratefully received.

       

      Thanks

      Keith

      • This topic was modified 3 weeks, 2 days ago by blairkei74. Reason: formatting screwed up
    • #270499
      Participant
      Topics: 17
      Replies: 1869
      Points: 3,631
      Helping Hand
      Rank: Community Hero

      Why are you pulling all entries just to get individual results? You are loading 40k objects into memory to search for a single item. Why not just get what you need?

      Active Directory most likely is indexing common unique search items for more efficient searching. When you pull this into memory and search this way, you are doing a plain search with no efficiency and are at the mercy of these filters.

    • #270502
      Participant
      Topics: 1
      Replies: 3
      Points: 18
      Rank: Member

      hi

      I’m doing an Azure audit on AAD and pulling out additional information to supplement the report as AAD does not hold all the properties we need. So I am reading 1000’s of accounts and mine was just an example.

      In my original report I issues a get-aduser for each user I wanted to get additional properties for but this was taking around 900ms per call. I thought it would be quicker to read all users into an array and then read from there.

       

      • This reply was modified 3 weeks, 2 days ago by blairkei74.
    • #270631
      Participant
      Topics: 1
      Replies: 3
      Points: 18
      Rank: Member

      What is wrong with the formatting on this site!

      It’s chopping off sentences…

    • #270697
      Participant
      Topics: 1
      Replies: 3
      Points: 18
      Rank: Member

      Update:

      I have switched to using a HASH table which fixes the overall performance.

       

Viewing 4 reply threads
  • You must be logged in to reply to this topic.