Parse Log.txt

This topic contains 19 replies, has 7 voices, and was last updated by  Curtis Smith 2 years, 4 months ago.

  • Author
  • #35032


    I have a program that dumps a log file that I need to parse.

    2016-02-08 18:58:33  INFO: [EDCAE484] id: 3, time: 2016-02-08 18:58:31, lat: 41.05376, lon: -74.28774, speed: 0.0, course: 0.0
    2016-02-08 18:58:45  INFO: [A62ED546] connected
    2016-02-08 18:58:45 DEBUG: [A62ED546: 5055] HEX: 485454502f312e3120323030204f4b0d0a0d0a
    2016-02-08 18:58:45  INFO: [A62ED546] disconnected
    2016-02-08 18:58:45  INFO: [A62ED546] id: 3, time: 2016-02-08 18:58:42, lat: 41.05376, lon: -74.28774, speed: 0.0, course: 0.0
    2016-02-08 18:58:56  INFO: [93D08D5C] connected
    2016-02-08 18:58:56 DEBUG: [93D08D5C: 5055] HEX: 485454502f312e3120323030204f4b0d0a0d0a
    2016-02-08 18:58:56  INFO: [93D08D5C] disconnected
    2016-02-08 18:58:56  INFO: [93D08D5C] id: 3, time: 2016-02-08 18:58:53, lat: 41.05376, lon: -74.28774, speed: 0.0, course: 0.0
    2016-02-08 18:58:58  INFO: [0ABBA09E] connected
    2016-02-08 18:58:58 DEBUG: [0ABBA09E: 5055] HEX: 485454502f312e3120323030204f4b0d0a0d0a
    2016-02-08 18:58:58  INFO: [0ABBA09E] disconnected
    2016-02-08 18:58:58  INFO: [0ABBA09E] id: 3, time: 2016-02-08 18:58:57, lat: 41.05376, lon: -74.28774, speed: 0.0, course: 0.0

    I need the line with ID to update and dump to a text file with the Date Time Lat Long. I only need the last unique value.


  • #35033

    Dan Potter

    What have you tried?

  • #35035

    $t = Get-Content 'C:\Program Files (x86)\Traccar\logs\tracker-server.log' 

    for instance. There are hundreds of lines in the text document.

    The problem is I don't know how to tell powershell where to look, and how to grab the latest unique entry. I appreciate your time!

  • #35036

    Wilfredo Perez

    You might have to use regex or match a string in your script

  • #35037

    Dan Potter

    gc .\log.txt | ? {$_ -match 'id:'} | select -Last 1

  • #35038


    Dan Thanks alot, the only thing is there is multiple ID's so in the example I posted there is only ID 3 but there will be multiple ID's. ie ID 1 , ID 2

    How do I grab those?

  • #35042

    random commandline

    Get-Content may use large amounts of memory depending on number of lines and/or readcount use.

    # Get Log
    $log = Get-ChildItem -Path .\log.txt
    # Parse log for matching ids
    $col = {@()}.Invoke()
    $log | foreach {$file = $_.OpenText()
    while ($file.EndOfStream -eq $false){
    $line = $file.ReadLine() ; If ($line -match 'id'){
    # Get unique ids
    $col2 = {@()}.Invoke()
    $col | Select-Object -Unique | foreach {
    $split = $_ -split ',' ; $id = $split[0] -match "id: (?'id'\d*$)"
    $dump = [pscustomobject]@{
    Id = $
    Date = ($split[1] -split '\s')[2]
    Time = ($split[1] -split '\s')[3]
    Lat = ($split[2] -split ':')[1]
    Long = ($split[3] -split ':')[1]
    } ; $col2.Add($dump) } # End Foreach
    # Get last value for each id group
    $col3 = {@()}.Invoke()
    $col2 | Group-Object -Property id | Sort-Object -Property Name |
    foreach {$last = $ | Select-Object -Last 1 ; $col3.Add($last)}
    $col3 | Export-Csv -NoTypeInformation -Path .\last_unique_id.csv
  • #35043


    Wow, Thank you soo much. I really appreciate it. Works great!

  • #35044

    random commandline

    Excellent, glad I could help.

  • #35054

    Exception calling "OpenText" with "0" argument(s): "The process cannot access the file 'C:\Program Files (x86)\Traccar\logs\tracker-server.log' because it is being used 
    by another process."
    At C:\Users\jwall\Documents\JWallCreations\PullLog.ps1:6 char:17
    + $log | foreach {$file = $_.OpenText()
    +                 ~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
        + FullyQualifiedErrorId : IOException

    Will only let it run once then I get this. argh! I need to be able to cycle through this log every minute, rather I would like too

  • #35056


    I should be able to force it, but -force doesn't make it work either. I can manually copy the file but powershell won't make a copy.

  • #35057

    Bob McCoy

    The error stems from the fact that you never do a $file.Close(). You have to clean up after yourself before you try the next open.

  • #35091

    Curtis Smith

    Just for fun, here's how I would do it.

    Get-Content C:\temp\tracker-server.log | 
    Select-String "] id: (\d+), " | 
    ForEach-Object {
        $hash["$($_.Matches.Groups[1].Value)"] = $_.line
    $hash.Values | ForEach-Object {
        $result = $_ -match "id:\s(?'id'\d+),\stime:\s(?'timestamp'\d+-\d+-\d+\s\d+:\d+:\d+),\slat:\s(?'lat'-*\d+.\d+),\slon:\s(?'lon'-*\d+.\d+)"
            Id = $
            Timestamp = $Matches.timestamp
            Lat = $
            Long = $Matches.lon   
    } | Export-Csv C:\Temp\export.csv -NoTypeInformation
  • #35103


    Thank you Curtis, This method doesn't hang the file open!

  • #35105

    random commandline

    I was not able to reproduce your error. I was successful looping a single and multiple files. Also, while the loops were running, I was able to open the text file(s) manually. I guess I am overlooking something. I created loops using the following while loop and by piping an array (1..10). Also I changed $log variable to this “Get-ChildItem -Path . -Filter log*”.

     while ($true) {$count++; “Loop#: $count”; & “C:\path\to\pullLog.ps1”}

    PSVersion 3.0
    (.NET) CLRVersion 4.0.30319.34209

  • #35106


    I will see, the file is being built by Its an open source project I am looking to query from.

  • #35132

    Graham Beer

    Hi Random Commandline,

    I was impressed with you code, wondering if you could help me understand it.
    If I do this :
    $a = {@()}
    Then look at the type name its,

    When I saw @() I assumed it would be an array.

    Would you mind explaining the following code in a bit more detail so I can learn it ?

    # Parse log for matching ids
    $col = {@()}.Invoke() – What is this invoking ?
    $log | foreach {$file = $_.OpenText()
    while ($file.EndOfStream -eq $false){
    $line = $file.ReadLine() ; If ($line -match 'id'){
    $col.Add($line)}}} – How is this working ?

    Thanks ! 🙂

  • #35138

    random commandline

    Ok, I understand how Curtis' $hash table works.
    If “$hash['key'] = value” is used to add to a hashtable, it overwrites the key's value (if key exists). If “$hash.Add('key','value')” is used an error occurs (if key exists).

    Exception calling "Add" with "2" argument(s):
    "Item has already been added. Key in dictionary: 'key' Key being added: 'key'"
    At line:1 char:1
    + $hash.Add('key','value')
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : ArgumentException

    Graham, this is what I like to do if I am parsing log files. In the past, I have parsed IIS log files that were MegaBytes in size and 100,000+ lines long. Using Get-Content was too slow and used alot of system memory.

    This is an example of how fast adding to collection is compared to array.

    # Array and Collection Speed Example col = 210 milliseconds; array = 4.6 seconds
    $arraytest = measure-command {$test = @(); 1..10000 | foreach {$test += $_}}
    $coltest = measure-command {$test = {@()}.Invoke(); 1..10000 | foreach {$test.Add($_)}}
    "Array Complete in {0}.{1} seconds" -f $arraytest.Seconds,$arraytest.Milliseconds
    "Collection Complete in {0}.{1} seconds" -f $coltest.Seconds,$coltest.Milliseconds 
    # Arrays have a fixed size and cannot be added to without using
    # the += assignment operator. Using this operator creates a new array and overwrites the # previous.
    # Converts array into a collection 
    $col = {@()}.Invoke()
    # Use .NET System.IO.StreamReader to open a file and read each line
    # Add matching lines to collection
    $log | foreach {$file = $_.OpenText()
    while ($file.EndOfStream -eq $false){
    $line = $file.ReadLine() ; If ($line -match 'id'){
  • #35140

    Graham Beer

    Thanks random command-line. So {@()} is a collection ? Does that differ much from a hash table ?

  • #35152

    Curtis Smith

    The advantage of the hash approach is that it is weeding out the unwanted records as it parses the log file. In the end there are only the 3 lines to parse into PSCustomObjects instead of all of the other records you don't really want anyway. You find the ones you want and just parse those.

You must be logged in to reply to this topic.