Reading large text file

This topic contains 5 replies, has 5 voices, and was last updated by  Max Kozlov 1 week, 5 days ago.

  • Author
    Posts
  • #79168

    TeeStar
    Participant

    Hi All

    I have a large text file,over 6 million lines (A list of primes) and I am trying to import it into a array. Get-Content is way to slow so I have tried to use the DotNet to help out. This is what I have tried so far.

    $a = New-Object -ComObject Scripting.FileSystemObject
    $f = $a.OpenTextFile("$ENV:USERPROFILE\Documents\Primes\PrimeList.Cfg", 1)
    [Array]$Primes=$f.ReadAll()
    

    What I end up with this every element of the array is a single character. Part of the array looks like this.

    1

    2

    3

    5

    7

    1
    1

    What I want is

    1
    2
    3
    5
    7
    11

    Any help would be appreciated.
    Thanks
    Tim

  • #79184

    Curtis Smith
    Participant

    Hey,
    We would need to understand your input file.

    I created a file that contains the following

    1
    2
    3
    5
    7
    11
    13

    I then ran you code against it

    $a = New-Object -ComObject Scripting.FileSystemObject
    $f = $a.OpenTextFile("D:\input.txt", 1)
    [Array]$Primes=$f.ReadAll()
    $Primes

    Results:

    PS C:\Windows\System32\WindowsPowerShell\v1.0> 
    $a = New-Object -ComObject Scripting.FileSystemObject
    $f = $a.OpenTextFile("D:\input.txt", 1)
    [Array]$Primes=$f.ReadAll()
    $Primes
    1
    2
    3
    5
    7
    11
    13
  • #79186

    Kevyn
    Participant

    I did the same thing, Curtis, and got the same results you did. TeeStar, what do the contents of your file look like? Can you show us maybe the first 10 lines or so? You might open your .cfg file in something like notepad++ and see if there are any special characters that could be causing issues.

  • #79199

    Zohaib
    Participant

    There must be \r on each line of .cfg file you are trying to read.
    I'm very new to Powershell myself but it would happen in Python.
    Try to strip any \r and \n character.

  • #79379

    TeeStar
    Participant

    Thanks everyone

    My Notepad ++ output looks fine as well as Notepad. No special characters or anything.

    I have tried this

    $Primes=[System.IO.File]::ReadAllLines("$ENV:USERPROFILE\Documents\Primes\PrimeList.Cfg")

    and it seems to work. I don't know why this one and the other does not work, must be the way I saved it perhaps??

  • #79384

    Max Kozlov
    Participant

    Use Format-Hex for looking into your file.
    You can see any of "0D/0A" combinations, but only several of it correct 🙂

You must be logged in to reply to this topic.