Manipulating csv with storage counters

This topic contains 3 replies, has 2 voices, and was last updated by Profile photo of SF SF 2 years, 5 months ago.

  • Author
    Posts
  • #16949
    Profile photo of SF
    SF
    Participant

    Hi

    I was hoping to play and learn with a task in powershell that at least sounded pretty easy in my head, I guess it is pretty easy stuff but I'm obviously doing something very wrong. After a couple of hours of trial and error I'm pretty sure I'm approaching this the wrong way.

    I recently installed a tool to get some statistics/values from our storage system.
    The tool produces a bunch of csv files (attached one of them)

    I started with just trying to play with the first four columns: [i]Timestamp, MDisk, Read operations (IO/s),Write operations (IO/s)[/i]

    I'm trying to calculate the average read and write IOPS per MDisk.
    Wit the code I got now, it actually outputs a view I was kind of happy with, at least for a start.
    Problem is that the final $object only has one (the last) mdisk and its values in it.

    I kind of understand that I'm doing something stupid with the foreach statement that's causing this, and I've tried a couple of things like initializing an empty array to be able to use +=$object.

    My hope is to be able to use these csv files to parse the data in a nice way in the future, but as I got stuck right at the beginning of playing and started to get disgruntled I was hoping someone could guide me in the right direction.

    The code:

    $csv = Import-Csv .\Reports\StorageV7000.csv
    
    foreach ($mdisk in $csv | Select-Object -ExpandProperty MDisk -Unique)
    {
    
            $avgRead = $csv | Where-Object MDisk -eq $mdisk | Measure-Object -Property "Read operations (IO/s)" -Average -Maximum -Minimum
            $avgWrite = $csv | Where-Object MDisk -eq $mdisk | Measure-Object -Property "Write operations (IO/s)" -Average -Maximum -Minimum
    
    
    
           $object = [pscustomobject]@{
          
    
                    'MDisk' = $mdisk
                    'Average Read IOPS' = [System.Math]::Round($avgRead.Average,2)
                    'Average Write IOPS' = [System.Math]::Round($avgWrite.Average,2)
                    'Average Total IOPS' = [System.Math]::Round($avgRead.Average+$avgWrite.Average,2)
            }
    
    
    
            $object
    
    
    }
  • #16951
    Profile photo of Dave Wyatt
    Dave Wyatt
    Moderator

    Looks like you just need to gather up all of your objects into a collection. Try this:

    $csv = Import-Csv .\Reports\StorageV7000.csv
    
    $report = foreach ($mdisk in $csv | Select-Object -ExpandProperty MDisk -Unique)
    {
     
            $avgRead = $csv | Where-Object MDisk -eq $mdisk | Measure-Object -Property "Read operations (IO/s)" -Average -Maximum -Minimum
            $avgWrite = $csv | Where-Object MDisk -eq $mdisk | Measure-Object -Property "Write operations (IO/s)" -Average -Maximum -Minimum
     
           [pscustomobject]@{
                    'MDisk' = $mdisk
                    'Average Read IOPS' = [System.Math]::Round($avgRead.Average,2)
                    'Average Write IOPS' = [System.Math]::Round($avgWrite.Average,2)
                    'Average Total IOPS' = [System.Math]::Round($avgRead.Average+$avgWrite.Average,2)
            } 
    }
    
    $report
    
  • #16952
    Profile photo of Dave Wyatt
    Dave Wyatt
    Moderator

    On a side note, you may find this version slightly clearer:

    $csv = Import-Csv .\Reports\StorageV7000.csv
    
    $report = $csv |
    Group-Object -Property MDisk |
    ForEach-Object {
        $diskGroup = $_
        $avgRead = $diskGroup.Group | Measure-Object -Property "Read operations (IO/s)" -Average -Maximum -Minimum
        $avgWrite = $diskGroup.Group | Measure-Object -Property "Write operations (IO/s)" -Average -Maximum -Minimum
        
        [pscustomobject]@{
            'MDisk'              = $diskGroup.Name
            'Average Read IOPS'  = [System.Math]::Round($avgRead.Average,2)
            'Average Write IOPS' = [System.Math]::Round($avgWrite.Average,2)
            'Average Total IOPS' = [System.Math]::Round($avgRead.Average+$avgWrite.Average,2)
        } 
    }
    

    With your calls to Select-Object -Property MDisk -Unique, and later Where-Object MDisk -eq $mdisk, you were basically recreating the functionality of Group-Object in a less efficient way.

  • #16972
    Profile photo of SF
    SF
    Participant

    Thanks Dave, I'm gonna go with your last suggestion, haven't used Group-Object alot.
    I know there a lot of ways of doing this stuff in PS, but would you (or anyone else) say this is decent approach or are there any other (better) ways?

    This is what I got right now with your help:

    $csv = Import-Csv .\Reports\StorageV7000.csv
     
    $report = $csv |
    Group-Object -Property MDisk |
    ForEach-Object {
            $diskGroup    = $_
            $avgReadIO    = $diskGroup.Group | Measure-Object -Property "Read operations (IO/s)" -Average -Maximum -Minimum
            $avgWriteIO   = $diskGroup.Group | Measure-Object -Property "Write operations (IO/s)" -Average -Maximum -Minimum
            $avgReadB     = $diskGroup.Group | Measure-Object -Property "Read blocks (Byte/s)" -Average -Maximum -Minimum
            $avgWriteB    = $diskGroup.Group | Measure-Object -Property "Write blocks (Byte/s)" -Average -Maximum -Minimum
            $avgReadUSEC  = $diskGroup.Group | Measure-Object -Property "Read external response time (usec)" -Average -Maximum -Minimum
            $avgWriteUSEC = $diskGroup.Group | Measure-Object -Property "Write external response time (usec)" -Average -Maximum -Minimum
    
    
            [decimal]$avgReadMB  = $avgReadB.Average /1MB
            [decimal]$avgWriteMB = $avgWriteB.Average /1MB
            [decimal]$avgReadMS  = $avgReadUSEC.Average /1000
            [decimal]$avgWriteMS = $avgWriteUSEC.Average /1000
        
     
        [pscustomobject]@{
            'MDisk'       = $diskGroup.Name
            'Read IOPS'   = [System.Math]::Round($avgReadIO.Average,2)
            'Write IOPS'  = [System.Math]::Round($avgWriteIO.Average,2)
            'Total IOPS'  = [System.Math]::Round($avgReadIO.Average+$avgWriteIO.Average,2)
            'Read MB/s'   = [System.Math]::Round($avgReadMB,2)
            'Write MB/s'  = [System.Math]::Round($avgWriteMB,2)
            'Read MS'     = [System.Math]::Round($avgReadMS,2)
            'Write MS'    = [System.Math]::Round($avgWriteMS,2)
    
        } 
    }
    

    Thinking that next step somehow should be to include the Storage Pools.
    For example mdisk0-5 are in one Pool, so maybe create another object that just sums some of the values of those mdisks, that should be easy, let's see how that goes 🙂
    I'll keep this post updated if that's okay.

    Also, any feedback/suggestions on how to go forward with this, other angles or code cleanup are always appreciated.

You must be logged in to reply to this topic.