Deleting duplicate entries

Tagged: 

This topic contains 4 replies, has 3 voices, and was last updated by Profile photo of Venkata Kalyan Venkata Kalyan 2 months, 1 week ago.

Viewing 5 posts - 1 through 5 (of 5 total)
  • Author
    Posts
  • #48094
    Profile photo of Venkata Kalyan
    Venkata Kalyan
    Participant

    Hi,
    I have a text file withe below data & I am trying to delete duplicate content.
    Aggregation Start Time – 2016-07-23 22.30.01
    Aggregation End Time – 2016-07-24 00.07.15
    Time taken for whole process – 5833684
    Aggregation Start Time – 2016-07-23 22.30.01
    Aggregation End Time – 2016-07-24 00.07.15
    Time taken for whole process – 5833684
    Aggregation Start Time – 2016-07-22 22.30.00
    Aggregation End Time – 2016-07-23 00.06.30
    Time taken for whole process – 5790047
    Aggregation Start Time – 2016-07-22 22.30.00
    Aggregation End Time – 2016-07-23 00.06.30
    Time taken for whole process – 5790047
    Aggregation Start Time – 2016-07-21 22.30.00
    Aggregation End Time – 2016-07-22 01.10.59
    Time taken for whole process – 9659321
    Aggregation Start Time – 2016-07-21 22.30.00
    Aggregation End Time – 2016-07-22 01.10.59
    Time taken for whole process – 9659337
    Aggregation Start Time – 2016-07-20 22.30.00
    Aggregation End Time – 2016-07-20 23.35.56
    Time taken for whole process – 3956665
    Aggregation Start Time – 2016-07-20 22.30.00
    Aggregation End Time – 2016-07-20 23.35.56
    Time taken for whole process – 3956665
    Aggregation Start Time – 2016-07-19 22.30.00
    Aggregation End Time – 2016-07-19 23.30.55
    Time taken for whole process – 3655256
    Aggregation Start Time – 2016-07-19 22.30.00
    Aggregation End Time – 2016-07-19 23.30.55
    Time taken for whole process – 3655256
    Aggregation Start Time – 2016-07-17 22.30.00
    Aggregation End Time – 2016-07-17 23.32.02
    Time taken for whole process – 3722827
    Aggregation Start Time – 2016-07-17 22.30.00
    Aggregation End Time – 2016-07-17 23.32.02
    Time taken for whole process – 3722827
    Aggregation Start Time – 2016-07-18 22.30.00
    Aggregation End Time – 2016-07-18 23.30.56
    Time taken for whole process – 3656366
    Aggregation Start Time – 2016-07-18 22.30.00
    Aggregation End Time – 2016-07-18 23.30.56
    Time taken for whole process – 3656366

    Code:
    $hash = @{} # define a new empty hash table
    gc C:\Opscripts\Kalyan\Vision_Data_Aggregation\filter.txt | %{if($hash.$_ -eq $null) { $_ }; $hash.$_ = 1} > C:\Opscripts\Kalyan\Vision_Data_Aggregation\final.txt

    But output is coming as below:
    Aggregation Start Time – 2016-07-21 22.30.00
    Aggregation End Time – 2016-07-22 01.10.59
    Time taken for whole process – 9432
    Time taken for whole process – 9432
    Aggregation Start Time – 2016-07-20 22.30.00
    Aggregation End Time – 2016-07-20 23.35.56
    Time taken for whole process – 3863
    Aggregation Start Time – 2016-07-19 22.30.00
    Aggregation End Time – 2016-07-19 23.30.55
    Time taken for whole process – 3569
    Aggregation Start Time – 2016-07-17 22.30.00
    Aggregation End Time – 2016-07-17 23.32.02
    Time taken for whole process – 3635
    Aggregation Start Time – 2016-07-18 22.30.00
    Aggregation End Time – 2016-07-18 23.30.56
    Time taken for whole process – 3570

    Can someone please help me, where I am missing?

    -Kalyan

    #48097
    Profile photo of Dan Potter
    Dan Potter
    Participant

    (gc .\text.txt | group).name

    #48127
    Profile photo of random commandline
    random commandline
    Participant
    Get-Content "C:\Opscripts\Kalyan\Vision_Data_Aggregation\filter.txt" | 
    Select-Object -Unique | Out-File "C:\Opscripts\Kalyan\Vision_Data_Aggregation\final.txt"
    
    #48184
    Profile photo of Venkata Kalyan
    Venkata Kalyan
    Participant

    Hi,
    Thanks. Tried the same & it worked fine.

    Aggregation Start Time – 2016-07-24 22.30.00
    Aggregation End Time – 2016-07-25 00.12.13
    Time taken for whole process – 6133243
    Aggregation Start Time – 2016-07-23 22.30.01
    Aggregation End Time – 2016-07-24 00.07.15
    Time taken for whole process – 5833684
    Aggregation Start Time – 2016-07-22 22.30.00
    Aggregation End Time – 2016-07-23 00.06.30
    Time taken for whole process – 5790047
    Aggregation Start Time – 2016-07-21 22.30.00
    Aggregation End Time – 2016-07-22 01.10.59
    Time taken for whole process – 9659321
    Time taken for whole process – 9659337
    Aggregation Start Time – 2016-07-20 22.30.00
    Aggregation End Time – 2016-07-20 23.35.56
    Time taken for whole process – 3956665
    Aggregation Start Time – 2016-07-19 22.30.00
    Aggregation End Time – 2016-07-19 23.30.55
    Time taken for whole process – 3655256

    Is there a way, I need a line gap for each and every 'Time taken for whole process'?

    my code:
    sc VIS_PRD_Data_Aggregation_Log.txt (gc final1.txt | % {$l = 1} {if ($l++ % 3) {$_} else {$_,"}})

    -Kalyan

    #48198
    Profile photo of Venkata Kalyan
    Venkata Kalyan
    Participant

    Thanks all.
    Modified the script a bit & all seems to be working.

    -Kalyan

Viewing 5 posts - 1 through 5 (of 5 total)

You must be logged in to reply to this topic.