Reading and saving file taking too long

This topic contains 23 replies, has 4 voices, and was last updated by Profile photo of Max Kozlov Max Kozlov 3 months, 1 week ago.

  • Author
    Posts
  • #60720
    Profile photo of Tony Antony
    Tony Antony
    Participant

    Hello,

    I have an INI file with about 1400 email address. How can I make this read faster? Currently it takes about 3 minutes to read and save file.

    I understand the reason why it's taking a long time is because it's looping through 1400 times doing the same thing over again. But, what's a better and faster way to do this? Is there a way to get the values of $Index outside of the for loop?

    $Users is the list of email addresses ~ 1400 emails.

    For($Index=0; $Index -lt $Users.length; $Index++)
        {          
            $Index_Plus_One = $Index + 1   
            $Email = $Users[$Index] -creplace '^[^=]*=', '' 
            $Re_Index = "$Index_Plus_One" + "=" + "$Email"   
            $Read_INI = $Read_INI | Where-Object {$_ -ne $Users[$Index]}
            $Read_INI | Set-Content $INI
        }  

    Thank you,

    Tony

  • #60721
    Profile photo of Olaf Soyk
    Olaf Soyk
    Participant

    Powershell is great for repetitive tasks. So it already know how about to iterate through a collection of items. This might push you in a better direction:

    Foreach($User in $Users){
        ... do something with $User
    }

    What do you actually want to do with your email addresses?

  • #60723
    Profile photo of Tony Antony
    Tony Antony
    Participant

    Thank you Olaf, but how would I get the array numbers using Foreach?

    I need the numbers to modify the email addresses

  • #60726
    Profile photo of Tony Antony
    Tony Antony
    Participant

    I tried the Foreach, and it's working, but I need get the numbers like I had in the for loop

  • #60727
    Profile photo of Olaf Soyk
    Olaf Soyk
    Participant

    Sorry for asking: What for? What do you want to do with your email addresses?

  • #60729
    Profile photo of Tony Antony
    Tony Antony
    Participant

    Right now, my email addresses look like this:

    1=mkelley@company.com
    2=jbruce@company.com
    3=oreynolds@company.com
    1438=alippard@company.com
    1439=klaw@company.com
    1440=dmarsh@company.com
    1441=jonathanj@company.com
    1442=aralls@company.com

    I need the index numbers so I can put them in order such as

    1=mkelley@company.com
    2=jbruce@company.com
    3=oreynolds@company.com
    4=alippard@company.com
    5=klaw@company.com
    6=dmarsh@company.com
    7=jonathanj@company.com
    8=aralls@company.com

  • #60732
    Profile photo of Olaf Soyk
    Olaf Soyk
    Participant

    hmmm ... ok. Assuming this 'order' provides an advantage for you, you can easely create a counter for your foreach loop like this.

    $Users = (1..10)
    $counter = 0
    Foreach($User in $Users){
        $counter++
        Write-Output "User_$User - Counter: '$($counter)'"
    }
    

    BTW: Powershell is able to sort you list if you want. 😉

  • #60735
    Profile photo of Tony Antony
    Tony Antony
    Participant

    Thank you, but the numbers of emails could change. It won't be always 1400. Any way to make it dynamic or is for loop the best option in my case?

    • #60736
      Profile photo of Olaf Soyk
      Olaf Soyk
      Participant

      Thank you, but the numbers of emails could change. It won't be always 1400.

      The numbers in front of your email addresses or the amount of email addresses in your file?? You confuse me a little bit. The amount does not matter. The foreach loop iterates over ALL items you have in a list or an array.

      Any way to make it dynamic or is for loop the best option in my case?

      Hhhhmmm I still didn't really get what you have to do with your email addresses.

  • #60739
    Profile photo of Tony Antony
    Tony Antony
    Participant

    Figured it out,

    But I think I'm still having the original problem with the for loop, let me try running it now.

    Thank you,

  • #60742
    Profile photo of Olaf Soyk
    Olaf Soyk
    Participant

    The proper property would be $users.count. Even if length works in this case. Length fits better for strings par example.

    But I think I'm still having the original problem with the for loop

    Why? Doesn't the foreach loop work for your needs?

  • #60745
    Profile photo of Tony Antony
    Tony Antony
    Participant

    The problem is that (same as the for loop problem) every time I add or delete an email address it reindexes the numbers starting from the beginning to end.

    I think the "fastest" way to do this is when I add someone, add them to the very last line, and when I delete someone, reindex the list after the delete email address instead of the whole email list.

    The only problem I see with this is that I need to create two functions? One for add and one for delete? The current functions works for both add and delete.

    Unless there's a faster way.

  • #60747
    Profile photo of Olaf Soyk
    Olaf Soyk
    Participant

    OK. I will only ask this one last time. What do you need to do wiht this email addresses. I believe I could help you if you'd tell me what you need. What is the situation before and what do you expect to be the situation after your script?

  • #60748
    Profile photo of Tony Antony
    Tony Antony
    Participant

    I have a list of about 1400 emails. Below is just 19 out of the 1400 emails as a sample.

    1=jbruce@company.com
    2=oreynolds@company.com
    3=JSTACK@company.com
    4=Dwu@company.com
    5=dsmith@company.com
    6=tantony@company.com
    7=jason.d@company.com
    8=jpettus@company.com
    9=plynch@company.com
    10=rhowe@company.com
    11=cconner@company.com
    12=cOberdalhoff@company.com
    13=emccomas@company.com
    14=fbrisco@company.com
    15=rmiller@company.com
    16=rburns@company.com
    17=jkohler@company.com
    18=Kfantom@company.com
    19=blevis@company.com
    so on ........

    What I'm trying to do is every time I add or delete an email, I want the numbers to be put in order, like above, this is working.

    The problem is that, it's taking about 2 minutes to put the numbers in order when I add and delete an email address.

    I think I can make it run faster when I add a person since I'm adding that email to the very last line.

    Example, if I add test@company.com

    1=jbruce@company.com
    2=oreynolds@company.com
    3=JSTACK@company.com
    4=Dwu@company.com
    5=dsmith@company.com
    6=tantony@company.com
    7=jason.d@company.com
    8=jpettus@company.com
    9=plynch@company.com
    10=rhowe@company.com
    11=cconner@company.com
    12=cOberdalhoff@company.com
    13=emccomas@company.com
    14=fbrisco@company.com
    15=rmiller@company.com
    16=rburns@company.com
    17=jkohler@company.com
    18=Kfantom@company.com
    19=blevis@company.com
    20=test@company.com

    But, I think depending on which email I'm deleting, the speed could vary because the email could be on top of the list or at the bottom

    If I delete jbruce@company.com it will take a longer time to put the list in order than if I'm deleting Kfantom@company.com

    I hope this is clear.

    Thank you,

  • #60750
    Profile photo of Olaf Soyk
    Olaf Soyk
    Participant

    OK. I got what you mean. Do you need this numbers (index) in front of your email addresses for some reason/purpose?

  • #60756
    Profile photo of Tony Antony
    Tony Antony
    Participant

    By changing where

    $Read_INI | Set-Content $INI -Force 

    is it's now running about 1 minute faster. Old way I had that inside the foreach, but by putting it outside the foreach, it's running more faster. Anything I can do?

    foreach ($User in $Users)
        {
            $Counter++
            $Email = $User -creplace '^[^=]*=', ''  
            $Index_Plus_One = $Counter
            $Re_Index = "$Index_Plus_One" + "=" + "$Email"          
            $Read_INI = $Read_INI | Where-Object {$_ -ne $User}    
            $Read_INI[$Users_Line_Num - 1] += "`r`n$Re_Index"
            $Read_INI | Set-Content $INI -Force 
        }    
    foreach ($User in $Users)
        {
            $Counter++
            $Email = $User -creplace '^[^=]*=', ''  
            $Index_Plus_One = $Counter
            $Re_Index = "$Index_Plus_One" + "=" + "$Email"          
            $Read_INI = $Read_INI | Where-Object {$_ -ne $User}    
            $Read_INI[$Users_Line_Num - 1] += "`r`n$Re_Index" 
        }    
        $Read_INI | Set-Content $INI -Force 

    The reason why I'm writing this script is because there's a GUI program, but doing it through the GUI takes 5 minutes to create and delete an email, so this is far more faster.

    GUI takes 5 minutes where as this takes 1 minute.

  • #60757
    Profile photo of Olaf Soyk
    Olaf Soyk
    Participant

    OK. To re-index your list you could do something like this:

    $AddressList = Get-Content -Path C:\sample\AddressList.txt
    $Counter = 0
    $NewList = Foreach($Address in $AddressList){
        $Counter++
        If($Address -match '^(\d+)='){
            $Address -replace '^(\d+)=',"$Counter="
        }
        Else{
            "$Counter" + "=" + "$Address"
        }
    }
    $NewList

    This should do it in seconds.
    To write this $NewList to a file you can pipe it to Out-File. Even to the original file if you want.
    How do you delete items or add items? Manually in an editor?

  • #60759
    Profile photo of Olaf Soyk
    Olaf Soyk
    Participant

    How do you think about changing your flat text list to a csv file. You could have one row with the index and one row with the email. I think it would make it easier to maintain and to work with.

  • #60762
    Profile photo of Tony Antony
    Tony Antony
    Participant

    Thank you, I'll try that recommendations. The file have to be in a .ini format for it to work properly with the software.

    Thanks for all your help.

    Tony

  • #60763
    Profile photo of Christian Sandfeld
    Christian Sandfeld
    Participant

    If you want to do the re-indexing at the same time when you delete address(es) you could do something like this

    # Sample data - replace with whatever you have to get your list of users
    $Users = @(
        '1=jbruce@company.com',
        '2=oreynolds@company.com',
        '3=JSTACK@company.com',
        '4=Dwu@company.com',
        '5=dsmith@company.com',
        '6=tantony@company.com',
        '7=jason.d@company.com',
        '8=jpettus@company.com',
        '9=plynch@company.com',
        '10=rhowe@company.com',
        '11=cconner@company.com',
        '12=cOberdalhoff@company.com',
        '13=emccomas@company.com',
        '14=fbrisco@company.com',
        '15=rmiller@company.com',
        '16=rburns@company.com',
        '17=jkohler@company.com',
        '18=Kfantom@company.com',
        '19=blevis@company.com',
        '20=test@company.com'
    )
    
    # E-mail address(es) to delete from your list
    $delete = @(
        'jbruce@company.com',
        'Kfantom@company.com'
    
    )
    
    # Initialize index variable
    $new_index = 0
    
    # Loop through all users and filter out those from delete variable
    $Users_Filtered = foreach ($User in $Users) {
        $index, $email = $User -split '='
        
        if ($delete -notcontains $email)
        {
            $new_index++
            Write-Output "$new_index=$email"
        }
        
    }
    
    # Final filtered list - insert whatever logic you need to save result to file
    $Users_Filtered
    

    I would expect that to be rather fast

  • #60766
    Profile photo of Olaf Soyk
    Olaf Soyk
    Participant

    Ah. OK. What a pitty. I will think about that a little more. Maybe we can streamline it a little.

  • #60769
    Profile photo of Christian Sandfeld
    Christian Sandfeld
    Participant

    Likewise to add to your list – assuming it is already ordered, you could do like this

    # Sample data - replace with whatever you have to get your list of users
    $Users = @(
        '1=jbruce@company.com',
        '2=oreynolds@company.com',
        '3=JSTACK@company.com',
        '4=Dwu@company.com',
        '5=dsmith@company.com',
        '6=tantony@company.com',
        '7=jason.d@company.com',
        '8=jpettus@company.com',
        '9=plynch@company.com',
        '10=rhowe@company.com',
        '11=cconner@company.com',
        '12=cOberdalhoff@company.com',
        '13=emccomas@company.com',
        '14=fbrisco@company.com',
        '15=rmiller@company.com',
        '16=rburns@company.com',
        '17=jkohler@company.com',
        '18=Kfantom@company.com',
        '19=blevis@company.com',
        '20=test@company.com'
    )
    
    # E-mail address(es) to add to the list
    $add = @(
        'johndoe@company.com',
        'janedoe@company.com'
    )
    
    # Initialize index variable
    $new_index = $Users.Count
    
    # Loop through addresses and add to list
    foreach ($email in $add)
    {
        $new_index++
        $Users += "$new_index=$email"
    }
    
    # Final list - insert logic to save to file
    $Users
    
  • #60778
    Profile photo of Max Kozlov
    Max Kozlov
    Participant

    tired to seeing as you walk on rake
    ... you write full ini on each of your 1400 users.

    here is my present. I spent 40 minutes but its work in second with 10000 users 🙂
    and it can be optimized more if ini group reading will be modified to use collections instead of hashes, because you want renumerations

    I will be glad if someone find it useful 🙂

You must be logged in to reply to this topic.