Invoke-Parallel need help to clone the current Runspace

Welcome Forums General PowerShell Q&A Invoke-Parallel need help to clone the current Runspace

Viewing 9 reply threads
  • Author
    Posts
    • #15509
      Participant
      Topics: 10
      Replies: 18
      Points: 1
      Rank: Member

      Hi PowerShell Community!

      If I hand over the Invoke-Parallel Function to other peoples, they struggle about the extra level of abstraction.
      They do not know that each Runspace (Session State) has its own environment.
      So the Variables, Functions, Modules, Snapins und so on from the current Runspace are not accessible from inside an new Runspace.
      To make the use of Invoke-Parallel easier I Try to clone the current Runspace into the InitialSessionState.

      I present here my first working attempt.
      Do you have Ideas or do you find Bugs. What shall implement additional. Do I have forgotten something to clone?
      Some things you have to clone from $Host.Runspace some things you have to clone from $Host.RunspaceInitialSessionState some things you can get from current Thread (like the Apartment State)
      Which clone source is right which source is wrong?

      [/crayon]

      awaiting you comments
      Greets Peter Kriegel

    • #15512
      Member
      Topics: 9
      Replies: 2322
      Points: 0
      Rank: Member

      Be very careful with that approach. Adding Modules, Functions and Snap-ins to the new runspace is fine, but when you add the same object references for variables to another runspace, you’ll be creating potential race conditions and other threading bugs. Sharing objects between threads in this manner should be probably be a conscious decision by the caller, not something that just happens automatically.

      Also, using Get-Variable -Scope 1 to obtain the caller’s variables will work so long as the caller is in the same script or script module as Invoke-Parallel (or if it’s been dot-sourced from a ps1 file to the current session), but won’t do what you intend if Invoke-Parallel has been imported from a psm1 file. You can always get at caller variables via the $PSCmdlet variable in an advanced function (ex: $PSCmdlet.GetVariableValue(‘ErrorActionPreference’) ), but I’m not aware of a public API for enumerating all variables this way.

    • #15531
      Participant
      Topics: 10
      Replies: 18
      Points: 1
      Rank: Member

      Hi Dave!

      Thank you foryour comments!

      I know the bussines to do Thread save things, because i am a .NET developer too.

      If you fumble arround with Threads there is allways a risk!
      That is the reason because the powerShell Team has decided to use PowerShell Jobs as Processes not as Threads.
      I hope with Windows 8 we get an PowerShell-Async Module (for the WinRT API).
      If you know the pitfalls, you can use it savely. 😉
      The first pitfall to know is, that a clone is a Snapshot, a copy, of a current state.
      So the changes outside the child runspaces are not reflected inside them.

      I dont think the PowerShell Team offers a InitialSessionState.Variables.Add() Method to us, if there is a danger with Thread save calls.
      The Variables ar not handed over explicitly by Reference [Ref].
      I think the Variables are not Shared they are re-created (copied) in every Runspace. It is Expensive but thread save….
      I did not investigated in this. Do you realy know for shure, that the Variables are shared in a not thread save manner?
      Even the scopes of the Variables can get lost or go wrong.

      Thank you for theInformation that the Get-Variable -Scope 1 not work in every circumstances.
      That is even a Pitfall to know… 😉 I will put this on my Test List.

      (can somebody correct the typo in the Topic headline?)

      Thanks Peter

    • #15540
      Member
      Topics: 9
      Replies: 2322
      Points: 0
      Rank: Member

      I dont think the PowerShell Team offers a InitialSessionState.Variables.Add() Method to us, if there is a danger with Thread save calls.
      The Variables ar not handed over explicitly by Reference [Ref].
      I think the Variables are not Shared they are re-created (copied) in every Runspace. It is Expensive but thread save….
      I did not investigated in this. Do you realy know for shure, that the Variables are shared in a not thread save manner?

      I’m positive. I even wrote a Lock function for PowerShell once I realized it was possible to share object references across different threads / runspaces: http://gallery.technet.microsoft.com/scriptcenter/Lock-Object-Synchronize-725ef5e7

      That gallery post has some test code which demonstrates both the sharing of these objects and how to use the function to synchronize access to them.

      Edit: I should probably clarify something. You’re passing around .NET object references and creating PowerShell variables with those references as their value. You can modify the PowerShell variables within a runspace without causing problems, but if you modify the .NET object that the variable referred to, it’s not thread-safe.

      $someVariable = ‘New Value’ # is okay.
      $someVariable.SomeProperty = ‘New Value’ # is not.

    • #15580
      Participant
      Topics: 10
      Replies: 18
      Points: 1
      Rank: Member

      Hey Dave BIG BIG thanks to enlighten me!
      This is very good Information!
      You Lock Function is marvelous.

      Ok! I withdraw the clone process of all Variables and put in the ability for the User to chose some Shared Variables over a Parameter.
      DO you have some other suggestions how we can help the User to make this Function more simple to use?

    • #15615
      Participant
      Topics: 4
      Replies: 16
      Points: 0
      Rank: Member

      Hi Peter,

      Haven’t touched the code in a while, but that is exactly the issue I run into at work. Co-workers without much scripting or dev experience have trouble with the idea of each runspace being an independent environment.

      I provide basic example scripts they can borrow and tweak… eventually they make a change big enough to break something (i.e. reference a variable created outside the runspace) and I try to explain why this won’t work. They nod their heads, modify the code, and rinse/repeat a few months down the road : )

      If you’re comfortable sharing, would be interested to see the end result!

      Cheers!

    • #15637
      Participant
      Topics: 1
      Replies: 42
      Points: 0
      Rank: Member

      Kind of late to the party on runspace handlers, but I posted a possible solution for getting accurate start times to use for timing out stalled threads here: http://mjolinor.wordpress.com/2014/05/26/accurate-timeout-for-threads-in-runspace-pool/

      Looking over the available scripts for handling runspaces, I note that none seem to have any provisions for returning the stream information from the runspaces.

      A different approach to the problem might be to have a runspace handler that has provisions for collecting and returning that data to the user for diagnostics, so they can have a better idea what’s going on in those runspaces. With that they might be able to get their runspace environments set up with the minimum amount of stuff to get their script to run without throwing the kitchen sink (and the overhead that goes with it) at every runspace they open.

    • #21949
      Participant
      Topics: 0
      Replies: 1
      Points: 0
      Rank: Member

      Brilliant, I had the same desire yesterday, but I didn’t find this thread before @psCookieMonster replied.
      So I re-added it myself, here is code https://gist.github.com/vors/3c3cc7e08f37b1d824a1
      Checkout diff https://gist.github.com/vors/3c3cc7e08f37b1d824a1/revisions

      For me the whole idea of shared state is a way to enable simple late integration. Make it just work without rewriting a lot of code.

      Scenario:
      I wrote a script that provision remote VM and execute pester tests against it.
      Now I want to run it against few different VMs (with various OS versions).
      Provisioning take 15 minutes and most of this time I just wait.
      I want to make provisioning parallel with minimal changes.

      It’s essentially the same problem as here :

      Co-workers without much scripting or dev experience have trouble with the idea of each runspace being an independent environment.

      One of the goals is to enable positive first-time experience for developers.
      Literally, the experience that I want: replace “… | % {…}” by “… | Invoke-Parallel -ImportVariables -ImportModules -Script {…}”.

    • #21950
      Member
      Topics: 9
      Replies: 2322
      Points: 0
      Rank: Member

      You may still run into some headaches if Invoke-Parallel is being imported as part of a script module (Get-Variable will catch the global scope, but not the caller’s scope stack below that. As long as the caller of Invoke-Parallel is the interactive console and not a script, that won’t matter.)

    • #21951
      Participant
      Topics: 2
      Replies: 35
      Points: 116
      Team Member
      Rank: Participant

      Painful visiting old and sloppy code! Oh well. Sergei, awesome ideas, incorporated the idea and a piece of your implementation, updated on github and technet. Thank you!

Viewing 9 reply threads
  • The topic ‘Invoke-Parallel need help to clone the current Runspace’ is closed to new replies.