Removing all instances of duplicates

Welcome Forums General PowerShell Q&A Removing all instances of duplicates

Viewing 4 reply threads
  • Author
    Posts
    • #185774
      Participant
      Topics: 1
      Replies: 0
      Points: 11
      Rank: Member

      I used the below script to export a csv with computer names and object id's from azure

      There are a number of duplicate files that I would like to remove.

      My csv looks like so

      name, object id

      comp1. id1

      comp1, id1

      comp2. id2

      comp3, id3

      I would like to remove these duplicates. I want to remove both the instances so when I output the csv we would only see comp2 and comp3. I have seen plenty scripts to remove just the duplicate but I want to remove any and all instances of duplicates.

      Could someone assist with the coding for this I am fairly new and can't find info online.

      Get-MsolDevice -All -LogonTimeBefore 'may, 2019 12:00:00 AM' | select-object -Property ObjectID, DisplayName | export-csv "C:\temp\test\List.csv" -NoTypeInformation

    • #185792
      Participant
      Topics: 10
      Replies: 117
      Points: 456
      Helping Hand
      Rank: Contributor

      This will do what you need:

      $csv = Import-Csv -Path 'C:\temp\test\List.csv' -NoTypeInformation
      
      $result = foreach ($row in $csv) {
      
          $duplicateCount = ($csv -match $row).Count
      
          if (-not ($duplicateCount -gt 1)) {
              Write-Output $row
          }
      }
      
      # Call $result or pipe back to the CSV file path to update it
      $result

      Edit: Replaced [pscustomobject] with $row.

    • #185795
      Participant
      Topics: 1
      Replies: 1632
      Points: 3,074
      Helping Hand
      Rank: Community Hero

      You can use Group-Object to filter out duplicates ...

      Import-Csv -Path 'Your csv file' |
          Group-Object -Property Name |
              Where-Object -Property Count -eq -Value 1 |
                  Select-Object -ExpandProperty Group
    • #185801
      Participant
      Topics: 10
      Replies: 117
      Points: 456
      Helping Hand
      Rank: Contributor

      Just an FYI in case it may not be clear:

      Olaf has provided an excellent example of how to filter out duplicates based on the Name, and groups them by the name (comp1, comp2, etc.) regardless of what the object id is. So if duplicates in the object id don't matter, go this nice, clean route.

      If you need to find duplicates based on the Name and object id, something like my original post would work.

      Depending on your need, either solution should suffice.

    • #185831
      Participant
      Topics: 1
      Replies: 1632
      Points: 3,074
      Helping Hand
      Rank: Community Hero

      If you really like to remove the duplicates of both properties you can simply append another treatment after the first one for the second property ... like this:

      Import-Csv -Path 'Your csv file' |
          Group-Object -Property Name |
              Where-Object -Property Count -eq -Value 1 |
                  Select-Object -ExpandProperty Group |
                      Group-Object -Property 'object id' |
                          Where-Object -Property Count -eq -Value 1 |
                              Select-Object -ExpandProperty Group
Viewing 4 reply threads
  • You must be logged in to reply to this topic.