Author Posts

July 25, 2016 at 11:49 am

Hi,
I have a text file withe below data & I am trying to delete duplicate content.
Aggregation Start Time – 2016-07-23 22.30.01
Aggregation End Time – 2016-07-24 00.07.15
Time taken for whole process – 5833684
Aggregation Start Time – 2016-07-23 22.30.01
Aggregation End Time – 2016-07-24 00.07.15
Time taken for whole process – 5833684
Aggregation Start Time – 2016-07-22 22.30.00
Aggregation End Time – 2016-07-23 00.06.30
Time taken for whole process – 5790047
Aggregation Start Time – 2016-07-22 22.30.00
Aggregation End Time – 2016-07-23 00.06.30
Time taken for whole process – 5790047
Aggregation Start Time – 2016-07-21 22.30.00
Aggregation End Time – 2016-07-22 01.10.59
Time taken for whole process – 9659321
Aggregation Start Time – 2016-07-21 22.30.00
Aggregation End Time – 2016-07-22 01.10.59
Time taken for whole process – 9659337
Aggregation Start Time – 2016-07-20 22.30.00
Aggregation End Time – 2016-07-20 23.35.56
Time taken for whole process – 3956665
Aggregation Start Time – 2016-07-20 22.30.00
Aggregation End Time – 2016-07-20 23.35.56
Time taken for whole process – 3956665
Aggregation Start Time – 2016-07-19 22.30.00
Aggregation End Time – 2016-07-19 23.30.55
Time taken for whole process – 3655256
Aggregation Start Time – 2016-07-19 22.30.00
Aggregation End Time – 2016-07-19 23.30.55
Time taken for whole process – 3655256
Aggregation Start Time – 2016-07-17 22.30.00
Aggregation End Time – 2016-07-17 23.32.02
Time taken for whole process – 3722827
Aggregation Start Time – 2016-07-17 22.30.00
Aggregation End Time – 2016-07-17 23.32.02
Time taken for whole process – 3722827
Aggregation Start Time – 2016-07-18 22.30.00
Aggregation End Time – 2016-07-18 23.30.56
Time taken for whole process – 3656366
Aggregation Start Time – 2016-07-18 22.30.00
Aggregation End Time – 2016-07-18 23.30.56
Time taken for whole process – 3656366

Code:
$hash = @{} # define a new empty hash table
gc C:\Opscripts\Kalyan\Vision_Data_Aggregation\filter.txt | %{if($hash.$_ -eq $null) { $_ }; $hash.$_ = 1} > C:\Opscripts\Kalyan\Vision_Data_Aggregation\final.txt

But output is coming as below:
Aggregation Start Time – 2016-07-21 22.30.00
Aggregation End Time – 2016-07-22 01.10.59
Time taken for whole process – 9432
Time taken for whole process – 9432
Aggregation Start Time – 2016-07-20 22.30.00
Aggregation End Time – 2016-07-20 23.35.56
Time taken for whole process – 3863
Aggregation Start Time – 2016-07-19 22.30.00
Aggregation End Time – 2016-07-19 23.30.55
Time taken for whole process – 3569
Aggregation Start Time – 2016-07-17 22.30.00
Aggregation End Time – 2016-07-17 23.32.02
Time taken for whole process – 3635
Aggregation Start Time – 2016-07-18 22.30.00
Aggregation End Time – 2016-07-18 23.30.56
Time taken for whole process – 3570

Can someone please help me, where I am missing?

-Kalyan

July 25, 2016 at 12:07 pm

(gc .\text.txt | group).name

July 25, 2016 at 1:20 pm

Get-Content "C:\Opscripts\Kalyan\Vision_Data_Aggregation\filter.txt" | 
Select-Object -Unique | Out-File "C:\Opscripts\Kalyan\Vision_Data_Aggregation\final.txt"

July 25, 2016 at 3:54 pm

Hi,
Thanks. Tried the same & it worked fine.

Aggregation Start Time – 2016-07-24 22.30.00
Aggregation End Time – 2016-07-25 00.12.13
Time taken for whole process – 6133243
Aggregation Start Time – 2016-07-23 22.30.01
Aggregation End Time – 2016-07-24 00.07.15
Time taken for whole process – 5833684
Aggregation Start Time – 2016-07-22 22.30.00
Aggregation End Time – 2016-07-23 00.06.30
Time taken for whole process – 5790047
Aggregation Start Time – 2016-07-21 22.30.00
Aggregation End Time – 2016-07-22 01.10.59
Time taken for whole process – 9659321
Time taken for whole process – 9659337
Aggregation Start Time – 2016-07-20 22.30.00
Aggregation End Time – 2016-07-20 23.35.56
Time taken for whole process – 3956665
Aggregation Start Time – 2016-07-19 22.30.00
Aggregation End Time – 2016-07-19 23.30.55
Time taken for whole process – 3655256

Is there a way, I need a line gap for each and every 'Time taken for whole process'?

my code:
sc VIS_PRD_Data_Aggregation_Log.txt (gc final1.txt | % {$l = 1} {if ($l++ % 3) {$_} else {$_,"}})

-Kalyan

July 25, 2016 at 5:06 pm

Thanks all.
Modified the script a bit & all seems to be working.

-Kalyan