Help cleaning an array of ‘duplicates’

[Aaron Participant Topics: 22 Replies: 31 Points: 14Rank: Member]

I have a script that generates an array that has output as shown below:

[test group2] contains [test group1]
[test group3] contains [test group2]
[test group1] contains [test group2]
[test group2] contains [test group3]

 

The lines such as [test group2] contains [test group1] and [test group1] contains [test group2] are essentially duplicates, just in a different order. I really only need one or the other, not important which I keep.

How could I go about clearing out any duplicates from the array?

[grokkit Senior Moderator Topics: 10 Replies: 184 Points: 980Rank: Major Contributor]

Hmmm… if [test group1] = [test group2] then
[test group1] contains [test group2] = [test group2] contains [test group1]

but if [test group1] != [test group2] then
[test group1] contains [test group2] != [test group2] contains [test group1]

I’m not really following the logic here…

[Doug Maurer Participant Topics: 8 Replies: 617 Points: 2,393Rank: Community Hero]

So the same would be true for [test group2] contains [test group3] and [test group3] contains [test group2], correct? If so you could just use a switch utilizing regex with a list and only add to the list if both don’t already exist in a line. It’s a lot easier to show than express with words so please see the example.

To simulate a Get-Content $somefile we’ll use a here-string split at new lines. Then we’ll prepare an empty list and finally process each line through the switch.

PowerShell

$output = @' [test group2] contains [test group1] [test group3] contains [test group2] [test group1] contains [test group2] [test group2] contains [test group3] '@ -split [environment]::NewLine $result = [System.Collections.Generic.List[string]]::new() switch -Regex ($output){ '\[(.+)\].+\[(.+)\]'{ if( -not ($result | Where {$_.contains($Matches.1) -and $_.contains($Matches.2)})) { $result.Add($_) } } }

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

$output = @' [test group2] contains [test group1] [test group3] contains [test group2] [test group1] contains [test group2] [test group2] contains [test group3] '@ -split [environment]::NewLine   $result = [System.Collections.Generic.List[string]]::new()   switch -Regex ($output){     '\[(.+)\].+\[(.+)\]'{         if( -not ($result | Where {$_.contains($Matches.1) -and $_.contains($Matches.2)}))         {             $result.Add($_)         }     } }

And what ends up in $result are the first two that weren’t duplicated.

PowerShell

$result [test group2] contains [test group1] [test group3] contains [test group2]

1 2 3 4

$result   [test group2] contains [test group1] [test group3] contains [test group2]

The switch statement can also read directly from files. See the following expanded example.

PowerShell

$tempfile = New-TemporaryFile @' [test group2] contains [test group1] [test group3] contains [test group2] [test group1] contains [test group2] [test group2] contains [test group3] [test group2] contains [test group8] [test group3] contains [test group7] [test group1] contains [test group8] [test group4] contains [test group7] [test group7] contains [test group4] [test group7] contains [test group3] [test group7] contains [test group1] '@ | Set-Content $tempfile -Encoding UTF8 $result = [System.Collections.Generic.List[string]]::new() switch -Regex -File ($tempfile){ '\[(.+)\].+\[(.+)\]'{ if( -not ($result | Where {$_.contains($Matches.1) -and $_.contains($Matches.2)})) { $result.Add($_) } } } $result [test group2] contains [test group1] [test group3] contains [test group2] [test group2] contains [test group8] [test group3] contains [test group7] [test group1] contains [test group8] [test group4] contains [test group7] [test group7] contains [test group1]

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

$tempfile = New-TemporaryFile   @' [test group2] contains [test group1] [test group3] contains [test group2] [test group1] contains [test group2] [test group2] contains [test group3] [test group2] contains [test group8] [test group3] contains [test group7] [test group1] contains [test group8] [test group4] contains [test group7] [test group7] contains [test group4] [test group7] contains [test group3] [test group7] contains [test group1] '@ | Set-Content $tempfile -Encoding UTF8   $result = [System.Collections.Generic.List[string]]::new()   switch -Regex -File ($tempfile){     '\[(.+)\].+\[(.+)\]'{         if( -not ($result | Where {$_.contains($Matches.1) -and $_.contains($Matches.2)}))         {             $result.Add($_)         }     } }   $result   [test group2] contains [test group1] [test group3] contains [test group2] [test group2] contains [test group8] [test group3] contains [test group7] [test group1] contains [test group8] [test group4] contains [test group7] [test group7] contains [test group1]

[Aaron Participant Topics: 22 Replies: 31 Points: 14Rank: Member]

This works perfectly!

 

Thank you!

[Author]

Posts