invoke-command takes 90 seconds to complete

This topic contains 4 replies, has 2 voices, and was last updated by Profile photo of David Lee David Lee 1 year, 7 months ago.

  • Author
    Posts
  • #23350
    Profile photo of David Lee
    David Lee
    Participant

    all servers I'm testing with is server 2008r2 with powershell 4. I have a text file with 100 servers that are verified online and with PS remoting enabled and working.

    When I run this command:

    # this takes about 90 seconds to complete
    Invoke-Command -ScriptBlock {get-date} -ComputerName (gc 100servers.txt) -ThrottleLimit 100

    I didn't get these delays when my local was running powershell 2 and executing against the same set of servers with PS4 on them. PS2 only took 2-3 seconds to finish the command if all were online. If I run the command in PS4 ISE, it seems to work fast as well but for some reason PS4 traditional shell (not ISE) gives me these delays when working with large set of servers and increasing throttle to 100

    I even run new shell with -noprofile switch then I get the same execution delays with PS4 shell. I even tried a different server with PS4 to run the command from and same results.

    # I even tried skipping the different checks with sessionOption below but same results

    Invoke-Command -ScriptBlock {get-date} -ComputerName (gc 100servers.txt) -ThrottleLimit 100 -SessionOption (New-PSSessionOption -SkipCACheck -SkipRevocationCheck -SkipCNCheck)

    I also tried running the same command from win8.1 with PS4 and it will hang for about 90 seconds before it spits out all the results at once.

    This seems to be a problem with large number of servers. Works fine with small set. Is anyone else having this problem? I know there's other ways of running scripts in parallel, but I can't figure out why this is not working. I'm also using the default pssessionoption below. I have full admin rights to all the servers. I tried changing my winrm settings. Any suggestions? I know of another guy in another company with the same problem too so it's not just my environment.

    Set-Item WSMan:\localhost\Shell\MaxProcessesPerShell 200
    Set-Item WSMan:\localhost\Plugin\Microsoft.PowerShell\Quotas\MaxProcessesPerShell 200
    Set-Item WSMan:\localhost\Plugin\Microsoft.PowerShell32\Quotas\MaxProcessesPerShell 200
    Set-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB 4096
    Set-Item WSMan:\localhost\Plugin\Microsoft.PowerShell\Quotas\MaxMemoryPerShellMB 4096
    Set-Item WSMan:\localhost\Plugin\Microsoft.PowerShell32\Quotas\MaxMemoryPerShellMB 4096
    Restart-Service winrm

    MaximumConnectionRedirectionCount : 5
    NoCompression : False
    NoMachineProfile : False
    ProxyAccessType : None
    ProxyAuthentication : Negotiate
    ProxyCredential :
    SkipCACheck : False
    SkipCNCheck : False
    SkipRevocationCheck : False
    OperationTimeout : 00:03:00
    NoEncryption : False
    UseUTF16 : False
    IncludePortInSPN : False
    OutputBufferingMode : None
    Culture :
    UICulture :
    MaximumReceivedDataSizePerCommand :
    MaximumReceivedObjectSize :
    ApplicationArguments :
    OpenTimeout : 00:03:00
    CancelTimeout : 00:01:00
    IdleTimeout : -00:00:00.0010000

  • #23402
    Profile photo of Don Jones
    Don Jones
    Keymaster

    Well, you've taken the same troubleshooting steps I would have taken – you've ruled out differences between the ISE and Console, for example, and discovered that only Console has the problem.

    I'm wondering what would happen if you scaled your throttling back to maybe 50. Keep in mind that each active connection is a thread of PowerShell, requiring memory and processor overhead. The console application may somehow be less efficient at managing the large number of threads you've asked for. I've not personally encountered a similar situation, but I tend to run at the default 32-thread throttle limit, and I'm just maybe a bit more patient in waiting for it to churn through the server lists I give it. But in customer situations, I've worked with sets well over 3,000 machines. I don't know that we would ever have considered 90 seconds a problem, though.

    I'm not sure it's accurate to describe this as "not working;" it [i]is[/i] working, albeit not as fast as you'd like.

    Keep in mind that v2 used less robust WS-MAN connectivity; I'm a little tempted to attribute at least some of the difference to the more robust session types introduced in v3, but that doesn't explain why you're only experiencing the slower performance in the console.

    You might use the PSDiagnostics module to start a detailed trace. Do one run in the console, and another in the ISE, and compare the log messages to see if there are any noticeable differences.

    I wouldn't mess with the WinRM settings; those same settings are used for both the console and the ISE, so they're clearly not going to be causing a difference in only one of those. Same thing with admin rights – that obviously wouldn't cause a problem in only one application. Let's try to stick with things that are different between the two applications, since you've indicated that the ISE works acceptably for you.

  • #25049
    Profile photo of David Lee
    David Lee
    Participant

    Sorry I didn't get an email notification and just found it in my junk mail. The from address was noreply@owershell.org/wp. So looks like they misspelled the domain so it was failing the SPF check. Just thought you might want to know Don 🙂 This was back in March so hopefully they fixed it by now.

    Thank you so much for replying. I haven't gotten anywhere in the other 3 sites/forums I've posted this to. You are correct. It does "work", but the hangs are really annoying. Especially coming from Myspace where one of our engineers (Dan Farino) created a server agent that does mostly the same thing as PSremoting, but our serveragent returns results almost instantly on healthy servers. And this was back in PS v1, before PSremoting came around. Then been using PSremoting in v2 since it's builtin. It was a little slower than our serveragent, but not bad.

    Shell1: I tried throttling back to 50 and it finished between 40-50 seconds. I ran it again in the same shell and it ran as fast as PS2 (2-3 seconds).
    Shell2: I tried throttle at 50 again, still took 40-50 seconds. After it finishes, I ran it again with throttle 100 and it finishes in 2-3 seconds. I switched back to throttle 50 and its still fast.
    Shell3: Throttle is default at 32 and it hangs for 20-30 seconds. I ran it again in same shell and it's fast (3 seconds). Then I throttle to 100 and got 20-25 second hangs. Then I went back to 32 and its still fast. Throttle to 100 and it takes 10 seconds. I retried 100 again and got 85 seconds.

    If I wait about 5 mins in the shell that is working fast, then the initial hangs come back. Sometimes when it hangs, I get small chunks of servers that do return their results but the overall hang time is the same for the entire set of servers. While they are hanging, I'm able to run the same command on another server with PS2 and it finishes within 2 seconds against the same set of servers so I'm sure the servers are healthy. 90 seconds doesn't sound like a long time to wait, but it's a big noticeable difference from PSv2 that only takes 2 seconds.

    Another interesting thing I found is that new-pssession has this same hang time when I run:
    new-pssession -ComputerName (gc 100servers.txt) -ThrottleLimit 100

    It makes sense that invoke-command and new-pssession shares the same methods to establish the session. I just wish I had access to source to see what is really going on under the hood. Since Don is a powershell Jedi master, could you please pass this along to the developers? I've already posted to connect.microsoft.com below but I have a feeling it won't be fixed before powershell 5 is released since there's only 2 votes 🙁

    https://connect.microsoft.com/PowerShell/feedback/details/1260624/invoke-command-throttle-100-hangs

    Let me know if you have anymore suggestions! Thank you so much for your time. I know you're busy. I'm checking out the new video of you and Jeffrey at Ignite now.

  • #25090
    Profile photo of Don Jones
    Don Jones
    Keymaster

    So, some of what you're seeing, I'd expect. Keep in mind that you're connecting to an executable, which has to load up a bunch of .NET stuff. When it ends, some of that will remain in memory, so if you reconnect quickly, it'll run back up faster. Eventually the server will reclaim that memory, so a later connection will force it all to load again. So that behavior is expected.

    Throttling won't affect any of that, but obviously the connections are affected by that. Throttling is simply a limit on the number of local threads PowerShell will spin up to make new connections. So if you're reconnecting quickly, I'd always expected second and later attempts to be faster, because of the above.

    But in terms of testing this, I think you're maybe throwing in unnecessary variables, or at least testing too many at once. You've got the delay between connection attempts, whether or not you're using existing sessions or spinning up new ones, and the throttling. Plus, the variable load that the servers are under, which will obviously impact. I think narrowing down so you can control for some of those variables will lead to test results that are a little easier to interpret.

  • #25095
    Profile photo of David Lee
    David Lee
    Participant

    Hi Don, yes I have noticed how it normally takes awhile on the initially call (even on PSv2) for the endpoints to respond. It makes sense that WinRM on the endpoints need to spin up and load everything similar to how IIS works if the app pool has reached idle timeout. I would like to find a way to increase those timeouts or even have it startup and load everything initially without waiting for the first caller. I think that should be in the WSMAN settings, but haven't had a chance to look deeper. That should probably be a separate forum thread anyway.

    But the problem still remains that after I spin up all those 100 servers using PSv2 and can get results from all those servers to return within 2 seconds in PSv2, I then immediately run the same command from a PSv4 server on the same set of servers but still get the 80-90 second hang if I use the -throttle 100 option. Once the PSv4 returns results 90 seconds later, I immediately run the same command and it still hangs 90 seconds. I've tried the same command 3 or 4 times in a row and it still hangs, even with or without me on PSv2 running the same command and getting results in 2 seconds during the PSv4 hangs. I do get better results if I try throttle 50, 32, or less servers. It hangs seem nonexistent with small set of servers (10 or less) and hangs become noticeable around 30-50 and sometimes hangs or not hang inconsistently. But at 100, it hangs every time as long as I keep the throttle at 100 everytime and starting from a new shell. So this is reproduceable. I have 2 other friends in 2 other companies that are able to reproduce this with 100 servers. But most people don't have this many servers to test with or are used to the hangs since they haven't compared it with PSv2 as in depth as I have. Others have resorted to using runspacepools for speed.

    Yes I agree, I threw in unnecessary variables in my testing, but it's mainly to rule out any possibilities and to show you I've done extensive testing and since you were asking about trying throttle 50 instead of 100. But narrowing down those numbers reveal that the issue is with large sets of servers throttled at a high number such as 100. It's like Newton's law of gravity holds true on earth with small speed and distance, but there's a bug in his equation that isn't noticeable until you apply them to planets much farther out in which Einstein fixed by including speed of light. hehe well you get my point 🙂 Just hoping we have an Einstein in the community willing to take on this challenge since it's beyond me.

    Thank you so much Don! If you ever get a chance to work with at least 100 servers, please test this out and let me know if you experience it too. If you have any buddies at Microsoft that could test and potentially fix it, please pass it along 🙂

You must be logged in to reply to this topic.