Select-String by pattern then if found actions

Welcome Forums General PowerShell Q&A Select-String by pattern then if found actions

This topic contains 22 replies, has 4 voices, and was last updated by

Pj
 
Participant
2 months, 3 weeks ago.

  • Author
    Posts
  • #108052
    Pj

    Participant
    Points: 0
    Rank: Member

    I am looking to search a log file for several patterns which I have working.  What I would like to do to expand upon that is when it finds pattern X THEN search UP in the text log for a new pattern, down to the same pattern searched UP for to establish which lines to grab.   A less technical summary would be this log has hundreds to thousands of operations in it and i want the log entries for an individual set of entries for one operation IF it contains an error.  Not sure how to present the exact information being worked with without information overload.

  • #108055
    Pj

    Participant
    Points: 0
    Rank: Member

    this is the script that finds the error entries for some context to what i want to add on to...

     

    
    $SawLog = Get-Content "C:\temp\run.log"
    
    $PrinterLog = "C:\temp\PrinterLog.txt"
    
    $Date = "2018-08-07"
    
    $PrintErrors = @()
    
    $SortMe = @()
    
    $NoDate = @()
    
    $Final = @()
    
     
    
    If (Test-Path $PrinterLog) {Remove-Item -Path $PrinterLog -Force}
    
    $SawLog | %{
    
    if ($_ -match "PRT(0004|0005|0009)|ENG(0029|0037)|OPR0078") {
    
    if ($Matches[0] -eq "PRT0009") {
    
    $LineIndex = $SawLog.IndexOf($_) + 1 # Get Next Line
    
    $Output = $_, $SawLog[$LineIndex]
    
    }
    
    elseif ($Matches[0] -eq "\[ID: ENG0037") {
    
    $LineIndex = $SawLog.IndexOf($_) + 3 # Get Next Line
    
    $Output = $_, $SawLog[$LineIndex]
    
    }
    
    else { $Output = $_ }
    
    $PrintErrors += $Output | Out-String
    
    }
    
    }
    
     
    
    $SortMe += $SawLog -match "\[PRT(0004|0005|0009)|\[ID: ENG(0029|0037)|OPR0078" | Select-String -pattern $Date | %{ $List = $_ -Split '\['
    
    $NoDate += $List[3] | Out-String  }
    
     
    
    $Final += $NoDate | Group-Object -NoElement| Format-Table -AutoSize | Out-String
    
     
    
    "Print Error Details `n`n" + $PrintErrors + "`n Error Summary For $Date `n" + $Final  | Set-Content -Path $PrinterLog
    
     
    
    Start-Process -FilePath 'Notepad++' -ArgumentList $PrinterLog
    
    
  • #108059

    Participant
    Points: 225
    Helping Hand
    Rank: Participant

    when it finds pattern X THEN

    search UP in the text log for a new pattern,

    down to the same pattern searched UP for to establish which lines to grab.

    Whaaaaat???

    Find X

    Match X

    Ignore X and replace with Y, starting from the index of X

    Search up for Y and extract Y

    Whaaaaat ???

    What happens when string are below the X index ????

    Color me really confused.

    Why not just group matches for the extract.

     

    • #108062
      Pj

      Participant
      Points: 0
      Rank: Member

      ok lets try this.  I want to select-string all lines from "Part Started" to "Part Complete" If between those two  -match "\[PRT(0004|0005|0009)|\[ID: ENG(0029|0037)|OPR0078" result is found.  does that clarify what im up to?  NOTE the number of lines between "Part Started" and "Part Complete" is not a static amount.

  • #108181
    Pj

    Participant
    Points: 0
    Rank: Member

    So what I have found is maybe a Select-String -Context but i need to ID the lines for a line number ID to make a Start point and for an end point.  Then maybe a Select-Object -Index to pull that group to an array to then say IF -match is true keep it and {do something} else if false discard and move to next.  This process would need to function repeatedly without duplication to process the log file maybe in a foreach loop.  Does this sound plausible?

  • #109066
    Pj

    Participant
    Points: 0
    Rank: Member

    So I have been working at an answer to this on and off and so far no luck.  Any -match or select-string statement works for the direct line with an error or desired piece of info but in trying to broaden the scope and gather relevant information to that error in the log with other entries the wall I run into is those other entries are an unknown number of lines from the error so I can't just say -context to a select-string.  Something I've used prior in a MDB code looks like maybe it's viable but i have not worked with it in other ways does the following code snip have potential to do what I'm trying to describe?

     

    $SawLog = Get-Content "C:\temp\Complete.log"
    
    $cache = @()
    
    $FindErrors = $SawLog -match "Part Started -"
    
    $FindErrors.MoveFirst()
    
    do {
    
    if ($Matches[0] -match "PRT(0004|0005|0009)|ENG(0029|0037)|OPR0078|WaitingForPrinterTrigger") {
    
    $Cache += NotSureHowToAddAboveLineMatchesToArray
    
    $FindErrors.MoveNext()}
    
    Else {
    
    $FindErrors.MoveNext() }}
    
    until($FindErrors.EOF -eq $True)
    
    
  • #109078

    Participant
    Points: 160
    Helping Hand
    Rank: Participant

    Get-Content is going to give us an array of lines, so we just need to flag when to start gathering data. If we use a basic example with just leveraging like to start a flag and break the loop:

    $test = @"
    data
    data
    data
    trigger
    data i want
    data i want
    data i want
    data i want
    data i want
    end trigger
    data
    data
    data
    "@
    
    $getData = $false
    
    $results = foreach ($line in ($test -split [environment]::NewLine)) {
    
        if ($line -like 'end tri*') {
            #We have all of the data, break loop
            break
        }
    
        if ($getData -eq $true) {
            #Return the line
            $line
        }
        
        if ($line -like 'trigg*') {
            #Start flag to gather data
            $getData = $true
            
        }
    }
    
    $results
    

    Output:

    $results
    
    
    data i want
    data i want
    data i want
    data i want
    data i want
    

    If you want the trigger lines, you can just change the order around

    $results = foreach ($line in ($test -split [environment]::NewLine)) {
     
        if ($line -like 'trigg*') {
            #Start flag to gather data
            $getData = $true
            
        }
    
        if ($getData -eq $true) {
            #Return the line
            $line
        }
        
        if ($line -like 'end tri*') {
            #We have all of the data, break loop
            break
        }
    }
    
    $results
    

    Output:

    $results
    
    
    trigger
    data i want
    data i want
    data i want
    data i want
    data i want
    end trigger
    

    Also, if you had 3 segements in a single log, just remove the break and set the flag to false and you should get all of the data.

  • #109156
    Pj

    Participant
    Points: 0
    Rank: Member

    So that is a good example and I like how that is written but I don't see how to handle the part that really has me hung up.  I will reuse your example code to try and better explain...

    $test = @"
    data
    data
    data
    Part Started Trigger
    data # of Unknown Lines with No Errors # i do not want this trigger to end trigger because there is no error between them
    Part Complete End Trigger
    data
    data
    Part Started Trigger
    data # of Unknown Lines with Error  # i want this trigger to end trigger because it contains an error
    Part Complete End Trigger
    data
    data
    data
    "@
    
    $getData = $false
    
    $results = foreach ($line in ($test -split [environment]::NewLine)) {
    
        if ($Line -like 'Part Started -*') {
            #Start flag to gather data
            $GetData = $True
        }
    
        if ($GetData -eq $True) {
            #Return the line
            $Line
        }
        
        if ($Line -like 'Part Complete*') {
            #We have all of the data, break loop
            $GetData = $False
           
        }
    }
    
    $results
    

    To do what I am after (from my very much still learning opinion) is to compound the foreach to run what this would get per trigger set that looks at a regex to determine if it contains an error and if it does send it to an array if it does not move to the next trigger set.

  • #109163

    Participant
    Points: 3
    Rank: Member

    PJ,

    Let me make sure I understand your problem.  Given the example data below you would want to return the array of all of segment 2 but NOT segment 1 because somewhere in segment 1 it contains an error.  Is that correct?

     

    $test = @"
    data
    data
    data
    Part Started Trigger
    data segment 1
    data error segment 1
    data segment 1
    Part Complete End Trigger
    data
    data
    Part Started Trigger
    data segment 2
    data segment 2
    Part Complete End Trigger
    data
    data
    data
    "@
  • #109166
    Pj

    Participant
    Points: 0
    Rank: Member

    Yes but backwards. I want the segment with the error to return to an array that compiles all found segments containing errors. If the segment does not contain an error i do not want it.

    • #109181

      Participant
      Points: 3
      Rank: Member

      OK.  With that the logic it is a little more complicated.  Here is what I came up with.  Someone may have a more eliquent solution, but this works.  BTW, [environment]::NewLine -ne "`n" on my system, so I can't use it in my example for the -split operator.  Maybe someone can explain why...

       

      $test = @"
      data
      data
      data
      Part Started Trigger
      data segment 1
      data error segment 1
      data segment 1
      Part Complete End Trigger
      data
      data
      Part Started Trigger
      data segment 2
      data segment 2
      Part Complete End Trigger
      data
      data
      data
      "@
      
      $getData = $false
      
      
      $data = $test -split '\n'
      $results = foreach ($line in $data) {
      
          if ($line -like 'Part Started *') {
              $GetData = $True
              $trap = $false
              $temp = @()
              continue #to not include the trigger line
          }
      
          if ($GetData -eq $True) {
              if ($line -like '*error*') {
                  $trap = $True
              }
              else { 
                  $temp += $line 
              }
          }
          
          if ($Line -like 'Part Complete *') {
              $GetData = $False
              if ($trap){
                  $temp[0..(($temp.Count) - 2)] # if you don't want to include trigger line otherwise just $temp
              }                    
          }
      }
      
      $results
  • #109178
    Pj

    Participant
    Points: 0
    Rank: Member

    So I modified the example to this and I think I'm close but my guess is the regex area is multiplying data and making it not finish the script?

    $SawLog = Get-Content "C:\temp\Complete.log"
    $Final = @()
    $FullSelect = @()
    
    #this part is to greatly reduce lines needing parsed as each log file is over 10k lines and 
    #is used to strip info down to the parts I want between triggers
    $FullSelect += $SawLog -match "Part( Started -| Complete)|PRT(0004|0005|0009)|ENG(0029|0037)|OPR0078|WaitingForPrinterTrigger"
    
    $GetData = $False
    
    $Results = foreach ($line in ($FullSelect -split [environment]::NewLine)) {
        
        if ($line -match 'Part Started -') {
            #Start flag to gather data
            $GetData = $True
        }
    
        if ($GetData -eq $True) {
            #Return the line
            $line
        }
        
        if ($line -match 'Part Complete') {
            #We have all of the data, break loop
            foreach ($line in ($Results -split [environment]::NewLine)) { 
            If ($line -match "PRT(0004|0005|0009)|ENG(0029|0037)|OPR0078|WaitingForPrinterTrigger") {
            $Final += $Results
            } Else {
            Continue}}
            $GetData = $False
    
        }
    }
    
    $Final
    
  • #109183

    Participant
    Points: 3
    Rank: Member

    As I am looking at my code again, it may read better to use an else if on line 38 to omit the trigger line rather than the clunky code on line 46.

    • #109186
      Pj

      Participant
      Points: 0
      Rank: Member

      I do want to show trigger lines so thats no worry. Working on trying what you posted will reply shortly.

  • #109187
    Pj

    Participant
    Points: 0
    Rank: Member

    So Mike this returns the same as

    $SawLog = Get-Content "C:\temp\Complete.log"
    
    $Results = @()
    $Results += $SawLog -match "Part( Started -| Complete)|PRT(0004|0005|0009)|ENG(0029|0037)|OPR0078|WaitingForPrinterTrigger"
    
    $Results
    

    I need to see the trigger lines as they contain data in reference to what part the error happened on but I do not want to see them if there is no error between start and complete

  • #109195
    Pj

    Participant
    Points: 0
    Rank: Member

    I think I got it, anyone see any issues with the following?...

    $SawLog = Get-Content "C:\temp\Complete.log"
    $ProcessArray = @()
    
    $FullSelect += $SawLog -match "Part( Started -| Complete)|PRT(0004|0005|0009)|ENG(0029|0037)|OPR0078|WaitingForPrinterTrigger"
    
    $GetData = $False
    
    $Results = foreach ($line in ($FullSelect -split [environment]::NewLine)) {
        
        if ($line -match 'Part Started -') {
            #Start flag to gather data
            $PartStart = $Line
            $GetData = $True
            $Trap = $False
        }
    
        if ($GetData -eq $True) {
            #Return the line
            if ($line -match "PRT(0004|0005|0009)|ENG(0029|0037)|OPR0078|WaitingForPrinterTrigger") {
                $trap = $True 
                $ProcessArray += $PartStart
                $ProcessArray += $line 
                } Else {
                continue}
        }
        
        if ($line -match 'Part Complete') {
            #We have all of the data, break loop
            $PartComplete = $line
            $GetData = $False
            if ($trap) {
            $ProcessArray += $PartComplete
            $ProcessArray
            }
            $trap = $False
            
    
        }
    }
    
    $ProcessArray
    
  • #109196
    Pj

    Participant
    Points: 0
    Rank: Member

    crap not quite that is missing part completes in the output

  • #109198
    Pj

    Participant
    Points: 0
    Rank: Member

    OK Now I think its working...

    $SawLog = Get-Content "C:\temp\Complete.log"
    $ProcessArray = @()
    
    $FullSelect += $SawLog -match "Part( Started -| Complete)|PRT(0004|0005|0009)|ENG(0029|0037)|OPR0078|WaitingForPrinterTrigger"
    
    $GetData = $False
    
    $Results = foreach ($line in ($FullSelect -split [environment]::NewLine)) {
        
        if ($line -match 'Part Started -') {
            #Start flag to gather data
            $PartStart = $Line
            $GetData = $True
            $Trap = $False
        }
    
        if ($line -match "Part Complete") {
            $PartComplete = $line
        }
    
        if ($GetData -eq $True) {
            #Return the line
            if ($line -match "PRT(0004|0005|0009)|ENG(0029|0037)|OPR0078|WaitingForPrinterTrigger") {
                $trap = $True 
                $ProcessArray += $PartStart
                $ProcessArray += $line 
                $ProcessArray += $PartComplete 
                } Else {
                Continue}
        }
    
        if ($line -match 'Part Complete') {
            #We have all of the data, break loop
            $GetData = $False
            if ($trap) {
            $ProcessArray
            }
        }
    }
    
    $ProcessArray
    
  • #109213
    Pj

    Participant
    Points: 0
    Rank: Member

    Ok its got quirks and I'm not sure how to deal with them. The script prior to this post works but I get a part started and part complete for each error. I would like it to be part started, all errors on that part, then part complete. My output looks like this and looks like the error is to the prior part started...

    2018-08-14 10:02:35,322 [SYS.Com.BoardStatus[Part Started – Board ID: 29 Part Seq: 8 Part XML ID: 0 Dsc: Job: 4402-J Remainder ID: 5
    2018-08-14 10:00:26,976 [SYS.Com.Machine[Machine Status has been updated from ActiveCutting to WaitingForPrinterTrigger.
    2018-08-14 10:02:51,445 [SYS.Com.BoardStatus[Part Complete – Board ID: 29 Part Seq: 8 Part XML ID: 0 Dsc: Job: 4402-J Remainder ID: 5

    2018-08-14 10:02:35,322 [SYS.Com.BoardStatus[Part Started – Board ID: 29 Part Seq: 8 Part XML ID: 0 Dsc: Job: 4402-J Remainder ID: 5
    2018-08-14 10:00:36,259 [SYS.Com.Machine[Machine Status has been updated from WaitingForPrinterTrigger to ReadyHomed.
    2018-08-14 10:02:51,445 [SYS.Com.BoardStatus[Part Complete – Board ID: 29 Part Seq: 8 Part XML ID: 0 Dsc: Job: 4402-J Remainder ID: 5

    2018-08-14 10:02:35,322 [SYS.Com.BoardStatus[Part Started – Board ID: 29 Part Seq: 8 Part XML ID: 0 Dsc: Job: 4402-J Remainder ID: 5
    2018-08-14 10:01:00,821 [SYS.Com.Machine[Machine Status has been updated from OneBoardActive to WaitingForPrinterTrigger.
    2018-08-14 10:02:51,445 [SYS.Com.BoardStatus[Part Complete – Board ID: 29 Part Seq: 8 Part XML ID: 0 Dsc: Job: 4402-J Remainder ID: 5

    2018-08-14 10:02:35,322 [SYS.Com.BoardStatus[Part Started – Board ID: 29 Part Seq: 8 Part XML ID: 0 Dsc: Job: 4402-J Remainder ID: 5
    2018-08-14 10:01:02,604 [SYS.Com.Machine[Machine Status has been updated from WaitingForPrinterTrigger to OneBoardActive.
    2018-08-14 10:02:51,445 [SYS.Com.BoardStatus[Part Complete – Board ID: 29 Part Seq: 8 Part XML ID: 0 Dsc: Job: 4402-J Remainder ID: 5

  • #109423
    Pj

    Participant
    Points: 0
    Rank: Member

    Can I do loop inside of the foreach? I cant seem to get it working but something like bellow to get a part started, all errors on that part, then part complete. Also last post disregard time stamp order issue, that was me messing with it to work at this modification.

    $SawLog = Get-Content "C:\temp\Complete.log"
    $ProcessArray = @()
    
    
    $FullSelect += $SawLog -match "Part( Started -| Complete)|PRT(0004|0005|0009)|ENG(0029|0037)|OPR0078|WaitingForPrinterTrigger"
    
    $GetData = $False
    
    $Results = foreach ($line in ($FullSelect -split [environment]::NewLine)) {
        
        if ($line -match 'Part Started -') {
            #Start flag to gather data
            $PartStart = $Line
            $GetData = $True
            $Trap = $False
        }
    
        if ($line -match "Part Complete") {
            $PartComplete = $line
        }
    
        if ($GetData -eq $True) {
            #Return the line
            if ($line -match "PRT(0004|0005|0009)|ENG(0029|0037)|OPR0078|WaitingForPrinterTrigger") {
                $trap = $True 
                Do{$ProcessArray += $line} until ($line -match 'Part Complete') #< this is there a way to repeat this until it sees part complete?
                $ProcessArray += $PartComplete 
                $ProcessArray += "`n"
                } Else {
                Continue}
        }
    
        if ($line -match 'Part Complete') {
            #We have all of the data, break loop
            $GetData = $False
            if ($trap) {
            $ProcessArray
            }
        }
    }
    
    $ProcessArray
    
  • #109432

    Participant
    Points: 3
    Rank: Member

    PJ,

    I don't have your log to test but, some of your logic is a little hard to follow.  Here's what I think should work.

     

    $SawLog = Get-Content "C:\temp\Complete.log"
    
    $FullSelect += $SawLog -match "Part( Started -| Complete)|PRT(0004|0005|0009)|ENG(0029|0037)|OPR0078|WaitingForPrinterTrigger"
    
    $GetData = $False
    
    $Results = foreach ($line in ($FullSelect -split [environment]::NewLine)) {
        
        if ($line -match 'Part Started -') {
            #Start flag to gather data, reset array and reset $trap flag
            $GetData = $True
            $ProcessArray = @()
            $Trap = $False
        }
    
        if ($GetData) {
            #Return the line and check for error to set trap
            if ($line -match "PRT(0004|0005|0009)|ENG(0029|0037)|OPR0078|WaitingForPrinterTrigger") {
                $trap = $True 
            }
            $ProcessArray += $line 
        }
        
        if ($line -match "Part Complete") {
            #reset the get data flag
            $GetData = $False
            if ($Trap) { $ProcessArray }
        }
    }
    
    $Results
  • #109472
    Pj

    Participant
    Points: 0
    Rank: Member

    That works. Seems I like to over itemize tasks in the thought process of what the pipe is doing (pluralsight vids preach one task at a time). Building scripts to perform command like functions seems easier but parsing strings is looping and pipe intensive. Definitely learned on this script I didnt have any idea I could flag a line in a loop like that and started off with thinking I needed to define a start and end post error string match. Any who thank you all for the assistance hope this base gets me what I need to parse away for my ever growing WPF script.

  • #109499
    Pj

    Participant
    Points: 0
    Rank: Member

    One last tid bit hopefully an easy one. The Foreach inverted the entry order as in the list is now newest on top instead of bottom. Is there an easy way to invert that?

You must be logged in to reply to this topic.