Piping or Export-Csv limit?

Welcome Forums General PowerShell Q&A Piping or Export-Csv limit?

Viewing 3 reply threads
  • Author
    Posts
    • #180830
      Participant
      Topics: 2
      Replies: 9
      Points: 3
      Rank: Member

      I have a bit of code that ends in an export-csv that is giving me blank files.

      $output | export-csv -Path 'c:\users\me\desktop\output.csv -NoTypeInformation

      $output when output to the screen is loaded with data I would expect to see

      $output.count says its 17102 items

      $output.capacity states 32768

      $output | out-gridview gives the same results as the export-csv, 17k blank lines

      PSVersion: 5.1

    • #180839
      js
      Participant
      Topics: 27
      Replies: 742
      Points: 2,030
      Helping Hand
      Rank: Community Hero

      Are they all the same type of object?

      • #180857
        Participant
        Topics: 2
        Replies: 9
        Points: 3
        Rank: Member

        Yes they are. Here's the full script:

        $namespaces = Import-Csv 'C:\Users\me\Desktop\namespaces.csv'
        $FolderCollection = New-Object System.Collections.Generic.List[System.Object]
        
        foreach($namespace in $namespaces){
        $childshare = Get-ChildItem -Directory -Path $namespace.Name #-Recurse -Depth 3 -Force
        For($i = 1; $i -le $childshare.Count; $i++){
        $sharecount = $childshare.Count
        Write-Progress -Activity "Gathering Permissions $i of $sharecount" -PercentComplete ($i/$childshare.Count*100)
        foreach($share in $childshare){
        $acl = Get-Acl -Path $share.FullName
        foreach($access in $acl.Access){
        $secgroup = ($access.IdentityReference).ToString()
        $secgroup = $secgroup.Substring(10)
        $ObjProp = [ordered]@{'This Share'=$share.FullName;'Grants these rights'=$Access.FileSystemRights;'To these groups'=$Access.IdentityReference;'Which consists of these members'=(get-adgroupmember $($access.IdentityReference.ToString().split('\')[1]) -Recursive -ErrorAction SilentlyContinue | select -ExpandProperty name ) -join '; '}
        $CollectionObject = New-Object -TypeName PSObject -Property $ObjProp
        $FolderCollection.add($CollectionObject) 
        }
        }
        }
        }
        
        $FolderCollection | export-csv -Path C:\Users\me\Desktop\allDFS_permissions.csv - -NoTypeInformation
      • #180860
        js
        Participant
        Topics: 27
        Replies: 742
        Points: 2,030
        Helping Hand
        Rank: Community Hero

        What's an example of one object that works and one object that exports to a blank line?

      • #181404
        Participant
        Topics: 2
        Replies: 9
        Points: 3
        Rank: Member

        In this version of the script, nothing exports, everything is blank. I have another script that does something similar and works just fine. That code is:

        $allshares = Import-Csv "C:\users\me\Desktop\allshares.csv"
        $FolderCollection = New-Object System.Collections.Generic.List[System.Object]
        foreach($share in $allshares){
        $Acl = Get-Acl -Path $share.Name
        
        
        foreach($access in $acl.Access){
        $ObjProp = [ordered]@{'This Share'=$share.Name;'Grants these rights'=$Access.FileSystemRights;'To these groups'=$Access.IdentityReference;'Which consists of these members'=(get-adgroupmember $($access.IdentityReference.ToString().split('\')[1]) -Recursive | select -ExpandProperty name) -join '; '}
        $CollectionObject = New-Object -TypeName PSObject -Property $ObjProp
        $FolderCollection.add($CollectionObject)
        }
        }
        $FolderCollection | export-csv -Path "c:\users\me\desktop\nonDFSpermissions.csv" -NoTypeInformation
      • #181695
        Participant
        Topics: 2
        Replies: 9
        Points: 3
        Rank: Member

        Any more insight? I'm really at a loss as to what is going on here.

    • #180854
      Participant
      Topics: 6
      Replies: 108
      Points: 304
      Helping Hand
      Rank: Contributor

      It sounds like $output isn't an object with multiple attributes, etc. Can you give us an example of the output you see at the console and maybe how you're creating the $output variable.

      • #181407
        Participant
        Topics: 2
        Replies: 9
        Points: 3
        Rank: Member
        PS C:\Windows\system32> $FolderCollection
        
        
        This Share Grants these rights To these groups Which consists of these members 
        ---------- ------------------- --------------- ------------------------------- 
        \\domain.local\appdata\Citrix Modify, Synchronize domain\FileChange P*-A**** P****; P*-D**** E****; P*-D**** V******; P*-Q***** N****; P*-M*** M****; P*...
        
        
      • #181692
        Participant
        Topics: 2
        Replies: 9
        Points: 3
        Rank: Member

        Any more insight? I've tried converting the list to an array but still getting blank output. I'm all out of ideas...

    • #180866
      Participant
      Topics: 2
      Replies: 511
      Points: 1,307
      Helping Hand
      Rank: Community Hero

      Judging by your code sample, $Output is a generic List collection.

      For these collections, Count is a count of the objects contained in the List currently. Capacity is the currently allocated space for more objects to be added before it has to re-allocate a new collection internally to store more items.

      Essentially, List is a front-end that keeps an array hidden internally with (often) more capacity than it really strictly needs to have, because by design it's expected that you'll be adding or removing items from the list, and having a bit of additional capacity is helpful to avoid expensive operations where it has to re-allocate an entire new array because it runs out of space.

      So you can think of Count as the size of the List, but Capacity is the size of the currently-allocated internal array where items are actually stored. Once Count reaches the Capacity value, the List will automatically re-allocate a new array the next time you .Add() to the List — if I recall correctly, the new array it allocates is typically double the size of the one whose capacity you've just exceeded. So, when adding many items to the list, you'll see its Capacity change as it allocates new arrays whenever it's needed.

      If you already know the approximate size of your List (within, say, an order of magnitude or thereabouts), you can actually specify its initial Capacity by either:

      1. Creating the list with a preset initial capacity with [System.Collections.Generic.List[object]]::new($Capacity) where $Capacity is an integer.
      2. Creating the list by passing in an existing array for it to copy. It will have an initial capacity equal to the size of the array, if I recall correctly: [System.Collections.Generic.List[object]]::new($array)

      So the fact that it has additional Capacity above and beyond your actual Count of objects is simple an implementation detail; you don't have additional items hiding in your List that aren't being exported.

      Your problem with blank exported files is coming from somewhere else. 🙂

Viewing 3 reply threads
  • The topic ‘Piping or Export-Csv limit?’ is closed to new replies.