Powershell Core Linux – multi-core execution

Welcome Forums General PowerShell Q&A Powershell Core Linux – multi-core execution

  • This topic has 9 replies, 4 voices, and was last updated 3 months ago by
    js
    Participant
    .
Viewing 9 reply threads
  • Author
    Posts
    • #207288
      Participant
      Topics: 1
      Replies: 3
      Points: 19
      Rank: Member

      Hi,

      I am writing some powershell stuff (on linux) for Windows users.  I am reading and parsing large text files and inserting data into into SQL, etc..

      On linux, it only utilizes a single CPU core, and sometimes takes a rather long time.  I cannot seem to find any info on utilizing more than a single core to run the scripts.

      Does anyone know of a way to accomplish it using powershell core on Linux?  I have not fired up a windows virtual machine to see if the same scripts utilize more than a single core either, but I would definitely assume so considering the effort to make powershell multi-platform by MS.  If the answer there is no as well, I will need to convert all of this to Python, but that makes more setup work for me with the end users 🙂

       

      TiA,

       

      Greg

    • #207294
      Senior Moderator
      Topics: 8
      Replies: 1215
      Points: 4,335
      Helping Hand
      Rank: Community Hero

      PowerShell core scripts runs under pwsh process and AFAIK, there is no way where we can execute a script by utilizing multiple cores, I mean I don’t know if there a setting like that exist. I may be wrong here 🙂

    • #207321
      js
      Participant
      Topics: 29
      Replies: 813
      Points: 2,450
      Helping Hand
      Rank: Community Hero

      start-job will use multi processes (or put & at the end of a line), start-threadjob multi threads, powershell 7 also has foreach-object -parallel. There’s runspaces too but that’s more difficult. “get-content -readcount 0” is faster.

      • This reply was modified 3 months ago by js.
      • This reply was modified 3 months ago by js.
    • #207330
      Participant
      Topics: 1
      Replies: 3
      Points: 19
      Rank: Member

      Start-Job is not native in pwsh linux…

       

      [greg@dellxps SQLite]$ pwsh Start-Job -ScriptBlock { Get-Process -Name pwsh }
      The argument ‘Start-Job’ is not recognized as the name of a script file. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

    • #207333
      js
      Participant
      Topics: 29
      Replies: 813
      Points: 2,450
      Helping Hand
      Rank: Community Hero

      pwsh runs -file by default, not -command.

      pwsh -c "start-job {'hi'} | receive-job -wait -auto"
      
      hi
      • This reply was modified 3 months ago by js.
    • #207339
      Participant
      Topics: 1
      Replies: 3
      Points: 19
      Rank: Member

      I guess I am too newb at pwsh, but when I try this:

      pwsh -c “Start-Job { ‘routeDownload.ps1 importDat’ } | Receive-Job -wait -auto”

       

      I see it spawn, but there is no process running afterward.  I have a 14k line file which sleeps after each line for this test, pwsh immediately exits.  ImportDat is a function btw…

       

      When I run it native (pwsh routeDownload.ps1 importDat), it runs fine and remains in the process list until complete.

    • #207345
      js
      Participant
      Topics: 29
      Replies: 813
      Points: 2,450
      Helping Hand
      Rank: Community Hero

      You need to learn the basics. Plus the current directory of start-job is the home directory.

      pwsh -c “Start-Job { /users/me/routeDownload.ps1 importDat } | Receive-Job -wait -auto”
      • This reply was modified 3 months ago by js.
    • #207354
      Participant
      Topics: 1
      Replies: 3
      Points: 19
      Rank: Member

      No offense, but I am just trying to make a Windows user happy.

       

      Your latest comment did work when I put $PWD in front of the command, but it did still just utilize a single core (the origin of the post).

      Thanks to you both for the comments and help, but I will just proceed with python since I have never been a Windows user and will probably need a steep learning curve which is probably a lot longer than setting up the user’s machine with the python requirements.

       

      Thanks again though 🙂

       

    • #207357
      Participant
      Topics: 6
      Replies: 93
      Points: 427
      Helping Hand
      Rank: Contributor

      Well, I do know that in Windows, you can add -MTA to PowerShell.EXE for a “multithreaded apartment”. Not sure how you would accomplish this on Linux and or if it will use additional cores or not.

      I tried this option when I was having issues with a windows form calling functions and being locked up until the function completed, and by god, it worked. I did not however validate multiple cores being used. Your mileage will of course vary.

    • #207360
      js
      Participant
      Topics: 29
      Replies: 813
      Points: 2,450
      Helping Hand
      Rank: Community Hero

      Usually you would do multiple jobs, or a job while staying in the shell doing other things.

Viewing 9 reply threads
  • The topic ‘Powershell Core Linux – multi-core execution’ is closed to new replies.