Command exist and does not exist at the same time

This topic contains 14 replies, has 7 voices, and was last updated by Profile photo of Jonas S. Jonas S. 2 weeks, 2 days ago.

  • Author
    Posts
  • #58012
    Profile photo of Paal Braathen
    Paal Braathen
    Participant

    So ... I encountered something very weird:

    I create a task that is running as a group managed service account:

    $Trigger = New-ScheduledTaskTrigger -Once -At (get-date (Get-Date).AddSeconds(2) -Format o)
    $Action = New-ScheduledTaskAction -Execute "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -Argument ("-File {0}" -f "c:\temp\do-stuff.ps1")
    
    $Principal = New-ScheduledTaskPrincipal -UserID gmsa-test$ -LogonType Password
    $Task = New-ScheduledTask -Action $Action -Trigger $Trigger -Principal $Principal
    
    $Task | Register-ScheduledTask -TaskPath test -TaskName 123
    

    The script that is run:

    try {
        $ErrorActionPreference_old = $ErrorActionPreference
        $ErrorActionPreference = "stop"
        $env:USERNAME | Out-File "C:\temp\do-stuff.txt"
        Get-Date -Format "o" | Out-File -Append "C:\temp\do-stuff.txt"
    
    } catch {
        $msg = $_
        $env:USERNAME | Out-File "C:\temp\do-stuff.txt"
        Get-Date -Format "o" | Out-File -Append "C:\temp\do-stuff.txt"
        $msg | Out-File -Append c:\temp\do-stuff.txt
        $ErrorActionPreference = $ErrorActionPreference_old
    }
    
    $ErrorActionPreference = $ErrorActionPreference_old
    

    The content of do-stuff.txt after execution:

    gmsa-test$
    2016-11-17T15:29:43.7208152+01:00
    Out-File : The term 'Out-File' is not recognized as the name of a cmdlet, function, script file, or operable program. C
    heck the spelling of the name, or if a path was included, verify that the path is correct and try again.
    At C:\temp\do-stuff.ps1:4 char:21
    +     $env:USERNAME | Out-File "C:\temp\do-stuff.txt"
    +                     ~~~~~~~~
        + CategoryInfo          : ObjectNotFound: (Out-File:String) [], CommandNotFoundException
        + FullyQualifiedErrorId : CommandNotFoundException
    

    So: The Out-File is failing because it does not exist ... and this error is written to a file by using Out-File(!).

    What could possibly be going on here?

    It's not easy to reproduce this for all scripts. Just minor changes to the script and it might run as expected.

    EDIT:

    I just confirmed that the same script might fail and might not fail when the task executes. This must be some kind of race condition in powershell.exe? It's almost as if the script is running before powershell is loaded..?

  • #58013
    Profile photo of Don Jones
    Don Jones
    Keymaster

    Does this go away if you're not running it as a managed service account?

    • #58016
      Profile photo of Paal Braathen
      Paal Braathen
      Participant

      @donj

      I haven't been able to produce this error when using other accounts. I don't know if you saw my update, but this looks like a race condition of some kind (so it might be possible to reproduce as non-GMSA).

  • #58018
    Profile photo of Don Jones
    Don Jones
    Keymaster

    Yeah, thats' kind of what I was wondering. It's almost like the execution environment is in restricted mode for a moment. I'm not sure that's something we're going to be able to help you resolve – except, I dunno, maybe adding a Start-Sleep into the beginning.

    • #58019
      Profile photo of Paal Braathen
      Paal Braathen
      Participant

      Well ... sadly:

      If I put start-sleep at the top it will fail with:

      "Start-Sleep : The term 'Start-Sleep' is not recognized as the name of a cmdlet, function, script file, or operable prog
      ram. Check the spelling of the name, or if a path was included, verify that the path is correct and try again."

      Something weird is going on here :\

  • #58022
    Profile photo of Don Jones
    Don Jones
    Keymaster

    Yeah. Let me ask around a bit.

  • #58030
    Profile photo of Ron
    Ron
    Participant

    I had some trouble recently with powershell and scheduled tasks. No matter what I did, it would not honor the working directory parameter for the scheduled task. I even had to hard code the location of the script in the powershell.exe parameters. I didn't want to hard code it into the script, but it needed to be set since I read and write files in the script directory. I don't know why this fixed it, but I created a .bat file (in the same directory as the script) to call the powershell.exe command (without hard coding the the script path), used the .bat file in the scheduled task, and everything started working as expected.

  • #58040
    Profile photo of Don Jones
    Don Jones
    Keymaster

    One good thing to check – at the top of your script, see if you can output what's in $env:PSModulePath. Some of us are wondering if the environment isn't populated, which means PowerShell wouldn't be able to find commands. If you're not able to get a command to run to output that, you may need to do it with a raw .NET class, since those don't depend on the environment existing.

  • #58054
    Profile photo of Dave Wyatt
    Dave Wyatt
    Moderator

    What version of PowerShell are you running? I remember a similar problem from a thread over on the TechNet forums, but it was supposed to be fixed in PowerShell 4:

    https://social.technet.microsoft.com/Forums/windows/en-US/edc4ae2d-b804-4361-94de-7482533a8084/intermittent-the-term-getdate-is-not-recognized-as-the-name-of-a-cmdlet-function-script-file?forum=winserverpowershell

  • #58067
    Profile photo of Sean Decker
    Sean Decker
    Participant

    This happens with V5. We ran into the same problem. We actually found it when restructuring a module used by the script the scheduled task calls. We think it's related to task scheduler not loading profiles. If you explicitly load the core modules and out put $env they are all the guest values instead of the MSA profile values. Weirdly importing our module automagically makes autoloading work. We found it when we switched to trying to use required modules for dependencies. It break if you specify version. Just module name and it works. All in all, there is something seriously borked here.

  • #58070
    Profile photo of Sean Decker
    Sean Decker
    Participant

    Also, it is not intermittent for us. It happens every execution of the scheduledTask when using MSA

  • #58091
    Profile photo of Don Jones
    Don Jones
    Keymaster

    If you can read https://twitter.com/concentrateddon/status/799265924603752448, there's a thread of useful replies. It may help to launch your script from a .BAT file.

  • #58121
    Profile photo of Paal Braathen
    Paal Braathen
    Participant

    I've only tested and experienced this with PSVersion 5.0.10586.117 on a Windows Server 2012 R2.

    I did some modifications to the script:

    try {
        $ErrorActionPreference_old = $ErrorActionPreference
        $ErrorActionPreference = "stop"
        "==TRY==" | Out-File "C:\temp\do-stuff.txt"
        $env:USERNAME | Out-File -Append "C:\temp\do-stuff.txt"
        $env:PSModulePath | Out-File -Append "C:\temp\do-stuff.txt"
        Get-ChildItem "env:" | Out-File -Append "C:\temp\do-stuff.txt"
        Get-Date -Format "o" | Out-File -Append "C:\temp\do-stuff.txt"
    
    } catch {
        $msg = $_
        "==CATCH==" | Out-File -Append "C:\temp\do-stuff.txt"
        $msg | Out-File -Append c:\temp\do-stuff.txt
        $env:USERNAME | Out-File -Append "C:\temp\do-stuff.txt"
        $env:PSModulePath | Out-File -Append "C:\temp\do-stuff.txt"
        Get-ChildItem "env:" | Out-File -Append "C:\temp\do-stuff.txt"
        Get-Date -Format "o" | Out-File -Append "C:\temp\do-stuff.txt"
        $ErrorActionPreference = $ErrorActionPreference_old
    }
    
    $ErrorActionPreference = $ErrorActionPreference_old
    

    The above script has both failed and run as expected when I've been running the task.

    These results might be interesting?

    Success:

    ==TRY==
    gmsa-test$
    C:\Users\gmsa-test$\Documents\WindowsPowerShell\Modules;C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules\;C:\Program Files\Microsoft Monitoring Agent\Agent\PowerShell\
    
    Name                           Value                                                                                   
    ----                           -----                                                                                   
    ALLUSERSPROFILE                C:\ProgramData                                                                          
    APPDATA                        C:\Users\gmsa-test$\AppData\Roaming                                                   
    CommonProgramFiles             C:\Program Files\Common Files                                                           
    CommonProgramFiles(x86)        C:\Program Files (x86)\Common Files                                                     
    CommonProgramW6432             C:\Program Files\Common Files                                                           
    COMPUTERNAME                   MY-SERVER01                                                                              
    ComSpec                        C:\Windows\system32\cmd.exe                                                             
    FP_NO_HOST_CHECK               NO                                                                                      
    LOCALAPPDATA                   C:\Users\gmsa-test$\AppData\Local                                                     
    NUMBER_OF_PROCESSORS           2                                                                                       
    OS                             Windows_NT                                                                              
    Path                           C:\Python27\;C:\Python27\Scripts;C:\Perl\site\bin;C:\Perl\bin;C:\Windows\system32;C:\...
    PATHEXT                        .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.py;.pyw;.CPL                     
    PROCESSOR_ARCHITECTURE         AMD64                                                                                   
    PROCESSOR_IDENTIFIER           Intel64 Family 6 Model 42 Stepping 2, GenuineIntel                                      
    PROCESSOR_LEVEL                6                                                                                       
    PROCESSOR_REVISION             2a02                                                                                    
    ProgramData                    C:\ProgramData                                                                          
    ProgramFiles                   C:\Program Files                                                                        
    ProgramFiles(x86)              C:\Program Files (x86)                                                                  
    ProgramW6432                   C:\Program Files                                                                        
    PSModulePath                   C:\Users\gmsa-test$\Documents\WindowsPowerShell\Modules;C:\Program Files\WindowsPow...
    PUBLIC                         C:\Users\Public                                                                         
    SystemDrive                    C:                                                                                      
    SystemRoot                     C:\Windows                                                                              
    TEMP                           C:\Users\GMSA-T~1\AppData\Local\Temp                                                    
    TMP                            C:\Users\GMSA-T~1\AppData\Local\Temp                                                    
    USERDNSDOMAIN                  EXAMPLE.COM                                                                                  
    USERDOMAIN                     EXAMPLE                                                                                     
    USERNAME                       gmsa-test$                                                                            
    USERPROFILE                    C:\Users\gmsa-test$                                                                   
    windir                         C:\Windows                                                                              
    
    
    2016-11-18T10:28:59.5750559+01:00
    

    Fail:

    ==CATCH==
    Out-File : The term 'Out-File' is not recognized as the name of a cmdlet, function, script file, or operable program. C
    heck the spelling of the name, or if a path was included, verify that the path is correct and try again.
    At C:\temp\do-stuff.ps1:4 char:17
    +     "==TRY==" | Out-File "C:\temp\do-stuff.txt"
    +                 ~~~~~~~~
        + CategoryInfo          : ObjectNotFound: (Out-File:String) [], CommandNotFoundException
        + FullyQualifiedErrorId : CommandNotFoundException
     
    gmsa-test$
    C:\Users\Default\Documents\WindowsPowerShell\Modules;C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules\;C:\Program Files\Microsoft Monitoring Agent\Agent\PowerShell\
    
    Name                           Value                                                                                   
    ----                           -----                                                                                   
    ALLUSERSPROFILE                C:\ProgramData                                                                          
    CommonProgramFiles             C:\Program Files\Common Files                                                           
    CommonProgramFiles(x86)        C:\Program Files (x86)\Common Files                                                     
    CommonProgramW6432             C:\Program Files\Common Files                                                           
    COMPUTERNAME                   MY-SERVER01                                                                              
    ComSpec                        C:\Windows\system32\cmd.exe                                                             
    FP_NO_HOST_CHECK               NO                                                                                      
    NUMBER_OF_PROCESSORS           2                                                                                       
    OS                             Windows_NT                                                                              
    Path                           C:\Python27\;C:\Python27\Scripts;C:\Perl\site\bin;C:\Perl\bin;C:\Windows\system32;C:\...
    PATHEXT                        .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.py;.pyw;.CPL                     
    PROCESSOR_ARCHITECTURE         AMD64                                                                                   
    PROCESSOR_IDENTIFIER           Intel64 Family 6 Model 42 Stepping 2, GenuineIntel                                      
    PROCESSOR_LEVEL                6                                                                                       
    PROCESSOR_REVISION             2a02                                                                                    
    ProgramData                    C:\ProgramData                                                                          
    ProgramFiles                   C:\Program Files                                                                        
    ProgramFiles(x86)              C:\Program Files (x86)                                                                  
    ProgramW6432                   C:\Program Files                                                                        
    PSModulePath                   C:\Users\Default\Documents\WindowsPowerShell\Modules;C:\Program Files\WindowsPowerShe...
    PUBLIC                         C:\Users\Public                                                                         
    SystemDrive                    C:                                                                                      
    SystemRoot                     C:\Windows                                                                              
    TEMP                           C:\Windows\TEMP                                                                         
    TMP                            C:\Windows\TEMP                                                                         
    USERDNSDOMAIN                  EXAMPLE.COM                                                                                  
    USERDOMAIN                     EXAMPLE                                                                                     
    USERNAME                       gmsa-test$                                                                            
    USERPROFILE                    C:\Users\Default                                                                        
    windir                         C:\Windows                                                                              
    
    
    2016-11-18T10:29:12.7784602+01:00
    

    I guess that i might use bat files as a workaround, but I'd prefer if this just worked as expected.

  • #58156
    Profile photo of Jaykul
    Jaykul
    Participant

    The fact that it sometimes works for you is the only part of that which differs from what I found in testing.

    When the environment isn't set right, all sorts of PowerShell things fail, but most importantly:

    • Add-Type doesn't work if $Env:Temp is set to C:\Windows\Temp
    • Module Autoloading doesn't work, so you have to explicitly import modules — even the "default" Microsoft ones.

    A few other oddities vanished when we started setting the Temp folder and importing modules explicitly, but I can't begin to list all the weird ways things break 😉

  • #58162
    Profile photo of Jonas S.
    Jonas S.
    Participant

    Edit: Got a little too excited and didn't read all the other answers. This solution is already posted above.

    I'm seeing the exact same issue. I am switching my scheduled tasks from traditional service accounts (users) to gMSA and the scripts fail with the same errors.

    What I found is that it seems that the standard modules doesn't always get loaded in time. My workaround is to put the following at the top of my scripts:

    Import-Module Microsoft.PowerShell.Utility
    Import-Module Microsoft.PowerShell.Management
    

    You may have to load other modules as well. Microsoft.PowerShell.Utility covers the Out-* cmdlets and also Get-Date and others.

You must be logged in to reply to this topic.