Group-object Powershell memory limitation

Welcome Forums General PowerShell Q&A Group-object Powershell memory limitation

  • This topic has 2 replies, 2 voices, and was last updated 3 weeks ago by
    Participant
    .
Viewing 1 reply thread
  • Author
    Posts
    • #270758
      Participant
      Topics: 3
      Replies: 8
      Points: 47
      Rank: Member

      Hi All,

      I have been using the following for sometime, however the logs I need to scan have grown immensely and it will no longer complete.  Powershell will use around 4GiB and  process endlessly.  Logs are roughly 3.6 GB in total, select-string would hit on 90% of the lines.  Ideas to optimize this would be greatly appreciated

      Linux works without any issues, and much quicker performance.  However this is not ideal for my automation.

       

      • This topic was modified 3 weeks ago by kvprasoon. Reason: Code formatting
    • #271114
      Participant
      Topics: 17
      Replies: 1869
      Points: 3,631
      Helping Hand
      Rank: Community Hero

      There is a lot of piping and for loops. Recommend providing example log and example expected output. Piping is expensive, with some example information others can provide alternate approaches. Group-Object should be returning Name and Count, so Select is not needed and is passing all of that data to another command to process as an example.

      • #271165
        Participant
        Topics: 3
        Replies: 8
        Points: 47
        Rank: Member

        There is a lot of piping and for loops. Recommend providing example log and example expected output. Piping is expensive, with some example information others can provide alternate approaches. Group-Object should be returning Name and Count, so Select is not needed and is passing all of that data to another command to process as an example.

        I tested a bit more seems it is hanging up on “group-object” command, “uniq -c” from linux can complete this in less than a minute with the same data.

        This is an example of what each row would contain, fields can be missing completely or empty

Viewing 1 reply thread
  • You must be logged in to reply to this topic.