Get-unique items from an array

Welcome Forums General PowerShell Q&A Get-unique items from an array

Viewing 5 reply threads
  • Author
    Posts
    • #16851
      Participant
      Topics: 14
      Replies: 15
      Points: 21
      Rank: Member

      Hi,

      I am trying to union two sets of data and get the distinct values back. The below code gets the top 5 processes by cpu and working set memory and unions the results together. I was expecting a distinct list of processes with the following id's ( 4, 900, 2536, 3196, 4452, 6924, 7404, 1524) as you can see the processes with the id's 7404 & 1524 are missing from the last result set that gets a unique set. I can not explain this. Any help would be appreciated.

      $process = Get-Process
      $cpu = $process | Sort-Object CPU -Descending | Select-Object -First 5
      $ws  = $process | Sort-Object WorkingSet -Descending | Select-Object -First 5
      
      $a = @()
      $a += $cpu
      $a += $ws
      
       
      $cpu
      
        NPM(K)    PM(K)      WS(K) VM(M)   CPU(s)     Id ProcessName
          629       0      108        308     3 2,802.36      4 System     
          171      15    15024      21636   106 2,334.49   4452 vmtoolsd   
         3306     321   615960     772640  1349 1,415.93    900 devenv     
          313      24    17740      25096   105 1,393.74   2536 vmtoolsd   
         3558     328   606116     766860  1368 1,356.66   6924 devenv     
      
       
      $ws
      
      Handles  NPM(K)    PM(K)      WS(K) VM(M)   CPU(s)     Id ProcessName
         2969     393   694852     883436  1581   652.15   3196 devenv     
         2621     341   628636     778132  1430   343.76   7404 devenv     
         3306     321   615960     772640  1349 1,415.93    900 devenv     
         3558     328   606116     766860  1368 1,356.66   6924 devenv     
         2732     344   590188     761144  1390   598.62   1524 de
      
      $a | Sort-Object id | Get-Unique
      
      Handles  NPM(K)    PM(K)      WS(K) VM(M)   CPU(s)     Id ProcessName
          629       0      108        308     3 2,802.36      4 System     
          171      15    15024      21636   106 2,334.49   4452 vmtoolsd   
         3306     321   615960     772640  1349 1,415.93    900 devenv     
          313      24    17740      25096   105 1,393.74   2536 vmtoolsd   
         3558     328   606116     766860  1368 1,356.66   6924 devenv     
      
    • #16852
      Participant
      Topics: 1
      Replies: 64
      Points: 0
      Rank: Member

      Get-Unique compares each item in a list with the next item in the list and will only work against sorted lists.

      If you want to user Get-Unique you have to sort the list first:

      $a | Sort-Object -Property Id | Get-Unique
      

      Or you could use Select-Object to select unique entries from the list:

      $a | Select-Object -Unique
      
    • #16853
      Participant
      Topics: 14
      Replies: 15
      Points: 21
      Rank: Member

      Hi Simon,

      I have tried both approaches and they both yield the same results. Also I am already sorting the array am I not

      $a | Sort-Object id | Get-Unique

      . Do you think that I need to sort the collection by something other than Id?

      Thanks

    • #16856
      Participant
      Topics: 1
      Replies: 64
      Points: 0
      Rank: Member

      You are right, Select-Object -Unique and Get-Unique seems to both select unique entries based on ProcessName.

      To select unique entries based on only Id you could use this:

      $a | Sort-Object -Property Id -Unique
      

      If you want to select unique entries based on multiple values you would have to select all properties you want to use like this:

      $a | Select-Object 'Handles', 'NPM', 'PM', 'WS', 'VM', 'CPU', 'Id', 'ProcessName' -Unique
      
    • #16858
      Participant
      Topics: 10
      Replies: 1374
      Points: 1,475
      Helping Hand
      Rank: Community Hero

      Try Sort-Object -Unique since it lets you indicate the property that you want to sort\be unique:

      $process = Get-Process
      $cpu = $process | Sort-Object CPU -Descending | Select-Object -First 5
      $ws  = $process | Sort-Object WorkingSet -Descending | Select-Object -First 5
       
      $a = @()
      $a += $cpu
      $a += $ws
      
      "Total object Count: {0}" -f $a.Count
      $a
      
      "Total unique ID's: {0}" -f ($a | Select ID -Unique).Count
      $a | Select ID -Unique | Format-Table -AutoSize
      
      $unique = $a | sort-object id -Unique
      "Total with Sort-Object -Unique:  {0}" -f $unique.Count 
      $unique

      Output:

      Total object Count: 10
      
      Handles  NPM(K)    PM(K)      WS(K) VM(M)   CPU(s)     Id ProcessName                                                       
      -------  ------    -----      ----- -----   ------     -- -----------                                                       
         3990     116   155344     183300   867 2,941.16   4124 OUTLOOK                                                           
         1529     135   152700     150592   594 2,420.08   3164 chrome                                                            
          221      59   180724     109000   358 2,208.63   2060 chrome                                                            
          564      15     4820       3520   107 1,258.55   2732 SynTPEnh                                                          
          428      32   143480     123344   460 1,190.34   3240 chrome                                                            
         1232     126   364880     313972  1147   955.64   6024 powershell_ise                                                    
         3990     116   155344     183300   867 2,941.16   4124 OUTLOOK                                                           
         3243     252   118944     162424   955   976.08   2692 explorer                                                          
         1529     135   152700     150592   594 2,420.08   3164 chrome                                                            
         2508      75   120028     130288   727   162.45   3368 lync                                                              
      Total unique ID's: 8
      
      
      
        Id
        --
      4124
      3164
      2060
      2732
      3240
      6024
      2692
      3368
      
      
      Total with Sort-Object -Unique:  8
      
      Handles NPM(K)  PM(K)  WS(K) VM(M)   CPU(s)   Id ProcessName   
      ------- ------  -----  ----- -----   ------   -- -----------   
          221     59 180724 109000   358 2,208.63 2060 chrome        
         3243    252 118944 162424   955   976.08 2692 explorer      
          564     15   4820   3520   107 1,258.55 2732 SynTPEnh      
         1529    135 152700 150592   594 2,420.08 3164 chrome        
          428     32 143480 123344   460 1,190.34 3240 chrome        
         2508     75 120028 130288   727   162.45 3368 lync          
         3990    116 155344 183300   867 2,941.16 4124 OUTLOOK       
         1232    126 364880 313972  1147   955.69 6024 powershell_ise

      Reference: http://blogs.technet.com/b/heyscriptingguy/archive/2012/01/15/use-powershell-to-choose-unique-objects-from-a-sorted-list.aspx

    • #16859
      Participant
      Topics: 14
      Replies: 15
      Points: 21
      Rank: Member

      Thank you both Simon and Rob,

      Both solutions worked for me. Simon I cant believe that I didn't try:

      $a | Select-Object 'Handles', 'NPM', 'PM', 'WS', 'VM', 'CPU', 'Id', 'ProcessName' -Unique 

      I had previously tried :

      $a | Select-Object  -Unique 'Handles', 'NPM', 'PM', 'WS', 'VM', 'CPU', 'Id', 'ProcessName'

      which did not work for me.

      Rob I didn't even know that I could use -unique on a sort. That worked really well as using the select-object method did what I wanted but messed up the formatting which required me to format the table using expressions to tidy things back up.

      $a | Select-Object 'Handles', 'NPM', 'PM', 'WS', 'VM', 'CPU', 'Id', 'ProcessName' -Unique | 
              Format-Table @{n='Handles'; ex={$_.Handles}; align='right'; width=10} ,
                           @{n='NPM'; ex={$_.NPM}; align='right'; width=7}, 
                           @{n='PM'; ex={$_.PM}; align='right'; width=15}, 
                           @{n='WS'; ex={"{0:N2}" -f $_.WS}; align='right'; width=17},
                           @{n='VM'; ex={$_.VM}; align='right'; width=15}, 
                           @{n='CPU'; ex={"{0:N2}" -f $_.CPU}; align='right'; width=12},
                           @{n='Id'; ex={$_.Id}; align='right'; width=7}, 
                           @{n='ProcessName'; ex={$_.ProcessName}; align='right'; width=20} -wrap 
      

      vs

      $a | sort-object id -Unique
      

      Once again thanks both of you. I learned something new today.

      • #80231
        Participant
        Topics: 1
        Replies: 3
        Points: 0
        Rank: Member

        Thanks for your help. i know this is post is old but I found it helpful and just wanted to mention you can also use the below code to match all the property values of the object of any array. I have found this very useful importing CSV files that have overlapping lines (like application logs from different days) and no single key column.

        $LineUnique = $a | Select-Object * -Unique

Viewing 5 reply threads
  • The topic ‘Get-unique items from an array’ is closed to new replies.