Parse Log.txt

This topic contains 19 replies, has 7 voices, and was last updated by Profile photo of Curtis Smith Curtis Smith 7 months, 3 weeks ago.

Viewing 15 posts - 1 through 15 (of 20 total)
  • Author
    Posts
  • #35032
    Profile photo of 37mm
    37mm
    Participant

    I have a program that dumps a log file that I need to parse.

    2016-02-08 18:58:33  INFO: [EDCAE484] id: 3, time: 2016-02-08 18:58:31, lat: 41.05376, lon: -74.28774, speed: 0.0, course: 0.0
    2016-02-08 18:58:45  INFO: [A62ED546] connected
    2016-02-08 18:58:45 DEBUG: [A62ED546: 5055  00.192.67.18] HEX: 485454502f312e3120323030204f4b0d0a0d0a
    2016-02-08 18:58:45  INFO: [A62ED546] disconnected
    2016-02-08 18:58:45  INFO: [A62ED546] id: 3, time: 2016-02-08 18:58:42, lat: 41.05376, lon: -74.28774, speed: 0.0, course: 0.0
    2016-02-08 18:58:56  INFO: [93D08D5C] connected
    2016-02-08 18:58:56 DEBUG: [93D08D5C: 5055  00.192.67.18] HEX: 485454502f312e3120323030204f4b0d0a0d0a
    2016-02-08 18:58:56  INFO: [93D08D5C] disconnected
    2016-02-08 18:58:56  INFO: [93D08D5C] id: 3, time: 2016-02-08 18:58:53, lat: 41.05376, lon: -74.28774, speed: 0.0, course: 0.0
    2016-02-08 18:58:58  INFO: [0ABBA09E] connected
    2016-02-08 18:58:58 DEBUG: [0ABBA09E: 5055  00.192.67.18] HEX: 485454502f312e3120323030204f4b0d0a0d0a
    2016-02-08 18:58:58  INFO: [0ABBA09E] disconnected
    2016-02-08 18:58:58  INFO: [0ABBA09E] id: 3, time: 2016-02-08 18:58:57, lat: 41.05376, lon: -74.28774, speed: 0.0, course: 0.0
    

    I need the line with ID to update and dump to a text file with the Date Time Lat Long. I only need the last unique value.

    Help??

    #35033
    Profile photo of Dan Potter
    Dan Potter
    Participant

    What have you tried?

    #35035
    Profile photo of 37mm
    37mm
    Participant
    $t = Get-Content 'C:\Program Files (x86)\Traccar\logs\tracker-server.log' 
    $t[0]
    

    for instance. There are hundreds of lines in the text document.

    The problem is I don't know how to tell powershell where to look, and how to grab the latest unique entry. I appreciate your time!

    #35036
    Profile photo of Wilfredo Perez
    Wilfredo Perez
    Participant

    You might have to use regex or match a string in your script

    #35037
    Profile photo of Dan Potter
    Dan Potter
    Participant

    gc .\log.txt | ? {$_ -match 'id:'} | select -Last 1

    #35038
    Profile photo of 37mm
    37mm
    Participant

    Dan Thanks alot, the only thing is there is multiple ID's so in the example I posted there is only ID 3 but there will be multiple ID's. ie ID 1 , ID 2

    How do I grab those?

    #35042
    Profile photo of random commandline
    random commandline
    Participant

    Get-Content may use large amounts of memory depending on number of lines and/or readcount use.

    # Get Log
    $log = Get-ChildItem -Path .\log.txt
    
    # Parse log for matching ids
    $col = {@()}.Invoke()
    $log | foreach {$file = $_.OpenText()
    while ($file.EndOfStream -eq $false){
    $line = $file.ReadLine() ; If ($line -match 'id'){
    $col.Add($line)}}}
    
    # Get unique ids
    $col2 = {@()}.Invoke()
    $col | Select-Object -Unique | foreach {
    $split = $_ -split ',' ; $id = $split[0] -match "id: (?'id'\d*$)"
    $dump = [pscustomobject]@{
    Id = $Matches.id
    Date = ($split[1] -split '\s')[2]
    Time = ($split[1] -split '\s')[3]
    Lat = ($split[2] -split ':')[1]
    Long = ($split[3] -split ':')[1]
    } ; $col2.Add($dump) } # End Foreach
    
    # Get last value for each id group
    $col3 = {@()}.Invoke()
    $col2 | Group-Object -Property id | Sort-Object -Property Name |
    foreach {$last = $_.group | Select-Object -Last 1 ; $col3.Add($last)}
    $col3 | Export-Csv -NoTypeInformation -Path .\last_unique_id.csv
    
    #35043
    Profile photo of 37mm
    37mm
    Participant

    Wow, Thank you soo much. I really appreciate it. Works great!

    #35044
    Profile photo of random commandline
    random commandline
    Participant

    Excellent, glad I could help.

    #35054
    Profile photo of 37mm
    37mm
    Participant
    Exception calling "OpenText" with "0" argument(s): "The process cannot access the file 'C:\Program Files (x86)\Traccar\logs\tracker-server.log' because it is being used 
    by another process."
    At C:\Users\jwall\Documents\JWallCreations\PullLog.ps1:6 char:17
    + $log | foreach {$file = $_.OpenText()
    +                 ~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
        + FullyQualifiedErrorId : IOException
     
    

    Will only let it run once then I get this. argh! I need to be able to cycle through this log every minute, rather I would like too

    #35056
    Profile photo of 37mm
    37mm
    Participant

    I should be able to force it, but -force doesn't make it work either. I can manually copy the file but powershell won't make a copy.

    #35057
    Profile photo of Bob McCoy
    Bob McCoy
    Participant

    The error stems from the fact that you never do a $file.Close(). You have to clean up after yourself before you try the next open.

    #35091
    Profile photo of Curtis Smith
    Curtis Smith
    Participant

    Just for fun, here's how I would do it.

    $hash=@{}
    Get-Content C:\temp\tracker-server.log | 
    Select-String "] id: (\d+), " | 
    ForEach-Object {
        $hash["$($_.Matches.Groups[1].Value)"] = $_.line
    }
    $hash.Values | ForEach-Object {
        $result = $_ -match "id:\s(?'id'\d+),\stime:\s(?'timestamp'\d+-\d+-\d+\s\d+:\d+:\d+),\slat:\s(?'lat'-*\d+.\d+),\slon:\s(?'lon'-*\d+.\d+)"
        [pscustomobject]@{
            Id = $Matches.id
            Timestamp = $Matches.timestamp
            Lat = $Matches.lat
            Long = $Matches.lon   
        }
    } | Export-Csv C:\Temp\export.csv -NoTypeInformation
    
    #35103
    Profile photo of 37mm
    37mm
    Participant

    Thank you Curtis, This method doesn't hang the file open!

    #35105
    Profile photo of random commandline
    random commandline
    Participant

    I was not able to reproduce your error. I was successful looping a single and multiple files. Also, while the loops were running, I was able to open the text file(s) manually. I guess I am overlooking something. I created loops using the following while loop and by piping an array (1..10). Also I changed $log variable to this “Get-ChildItem -Path . -Filter log*”.

     while ($true) {$count++; “Loop#: $count”; & “C:\path\to\pullLog.ps1”}

    PSVersion 3.0
    (.NET) CLRVersion 4.0.30319.34209

Viewing 15 posts - 1 through 15 (of 20 total)

You must be logged in to reply to this topic.