Schedule rolling reboots with service check & email?

This topic contains 6 replies, has 4 voices, and was last updated by Profile photo of Tore Groneng Tore Groneng 3 years, 2 months ago.

  • Author
    Posts
  • #13082
    Profile photo of Jake Sully
    Jake Sully
    Participant

    Ive been banging my head on the desk trying to take a batch script and get the functionality in powershell.

    I need to schedule reboots of servers and have the servers reboot one at a time according to the order of the file the server is in. I need to check a service or two, and then capture a log file for all servers in that specific scheduled reboot time, then email it out.

    I am basically lost on this one entirely as I've tried all the options and I can schedule a single reboot, but nothing in a rolling manner, I can't get the logs or service checks to go, or the email.

    Can anyone help me out? The batch file is old and it isn't great, but I'm also trying to learn more. All servers don't have powershell on them, and it's a mix of 2003-2012.

  • #13093
    Profile photo of Vern Anderson
    Vern Anderson
    Participant

    Do the services on the 1st server listen on a specific TCP port? You could have the second server do an while loop or do until those ports are back up and listening from the first server being rebooted and then trigger each dependent server in that manner.

    emailing will only work if the SMTP server you're using alows the IP Address of the server sending the email to relay.
    http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/58f05ef9-55a3-42b3-9f57-27fdc8723b8a.mspx?mfr=true

    Also if the batch files work maybe you can post the batch file code here so we can look at it and show how it might be done with PowerShell.

    -VERN

  • #13321
    Profile photo of Jake Sully
    Jake Sully
    Participant

    Howdy –

    Thank you for the response, I'm still working at this. The service I'm checking now is the RPC server and I found a better "batch file script" online than what I had so I am trying that, but it's still the same process basically and I think it could be better. I'd like the emails to be formatted so it's the server name – reboot start time – time server came back online, and anything else useful which I could add. I'm still trying to figure out how to do this in powershell and not having luck at all. One issue I have with this script is that I often get the "hang on reboot" alert emails and if I increase the time limit then it takes a long time to reboot the list of servers as some reboot quicker, I know I can't fix that as it's based on the server, but maybe a way to not have the hung reboot alerts unless it's really hung? I can send email from the system that I run this on.

    echo off

    REM ********************************
    REM Set the time limits for the items below

    REM Set the max time here for the time for the server to shutdown
    set ST=1200
    REM Set the max time in SECONDS for a server to complete POST and start booting
    set BT=200
    REM Set the max time in MINUTES for a server to start after POST
    set CT=10
    REM Set the name of the mail server:
    set MLSVR=mail.testdomain.local
    REM Set the TO: email address:
    set TOEML=Jake@testdomain.local
    REM Set the From: email address:
    set FRMEML=jake@testdomain.local

    REM End of Tunable Parameters
    REM ********************************

    REM Finding the Local Timezone
    for /f "tokens=1,2*" %%K in ( ' reg query HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\TimeZoneInformation ' ) do if %%K==StandardName set timezone=%%M

    REM Creating the temporary Log file
    echo ***Logging Started on %DATE% %TIME% %TIMEZONE% *** > log.txt
    echo %USERNAME% on %COMPUTERNAME% has initiated the script at %CD% >> log.txt

    REM Checking if the list of servers for reboots is available.
    if NOT EXIST computers.txt goto nocomputers

    REM Listing the servers that will be rebooted.
    echo The following Servers will be rebooted: >> log.txt
    echo The following Servers will be rebooted:
    type computers.txt >> log.txt
    type computers.txt
    echo. >> log.txt
    echo.

    :routinestart
    REM Parse file and call the reboot subroutine with the server name as argument.
    for /F %%i in (computers.txt) do (call :restartscript %%i)

    REM Subroutine to rename the temporary log file and then email the final log.
    :cleanup
    for /f "tokens=1-5 delims=/ " %%d in ("%date%") do (
    echo. >>%USERDOMAIN%-%%g-%%f-%%e.txt
    type rebootlog.txt >> %USERDOMAIN%-%%g-%%f-%%e.txt
    del /F rebootlog.txt
    bmail -s %MLSVR% -t %TOEML% -f %FRMEML% -h -a "%USERDOMAIN% Server Reboot Log" -m %USERDOMAIN%-%%g-%%f-%%e.txt
    )
    goto eof

    REM The actual reboot script.
    :restartscript
    REM Initializing the Log file.
    set COMP=%1
    echo. >> log.txt
    echo.
    echo %DATE% %TIME% *** Log for %COMP% >> log.txt
    echo %DATE% %TIME% *** Log for %COMP%
    echo %DATE% %TIME% Checking if %COMP% is up >> log.txt
    echo %DATE% %TIME% Checking if %COMP% is up

    REM Checking if server is up if it is not up, skip this server and go to the next one.
    ping %COMP% > nul
    if %ERRORLEVEL% NEQ 0 goto offline
    goto startreboots

    REM Rebooting the server.
    :startreboots
    echo %DATE% %TIME% %COMP% is Up >> log.txt
    echo %DATE% %TIME% %COMP% is Up
    echo %DATE% %TIME% Rebooting %COMP% >> log.txt
    echo %DATE% %TIME% Rebooting %COMP%
    shutdown -r -f -m \\%COMP% -t 0
    echo %DATE% %TIME% %COMP% is rebooting... please wait... >> log.txt
    echo %DATE% %TIME% %COMP% is rebooting... please wait...

    for /L %%a in (1,1,%ST%) do (
    ping -n 2 -w 1 %COMP% > nul
    if ERRORLEVEL 1 goto down
    )
    set CAUSE=has hung on shutdown
    goto computerhung

    :down
    echo %DATE% %TIME% %COMP% is down, waiting for server to come up >> log.txt
    echo %DATE% %TIME% %COMP% is down, waiting for server to come up

    for /L %%a in (1,1,%BT%) do (
    ping -n 2 -w 1 %COMP% > nul
    if NOT ERRORLEVEL 1 goto start
    )
    set CAUSE=has hung during or after POST.
    goto computerhung

    :start
    echo %DATE% %TIME% %COMP% is starting up... please wait for a minute... >> log.txt
    echo %DATE% %TIME% %COMP% is starting up... please wait for a minute...

    for /L %%a in (1,1,%CT%) do (
    ping -n 40 -w 1 %COMP% > nul
    echo %DATE% %TIME% Checking the RPC Service status... Pass %%a of %CT% >> log.txt
    echo %DATE% %TIME% Checking the RPC Service status... Pass %%a of %CT%
    sc "\\%COMP%" query RpcSs | FIND "RUNNING" > nul
    if NOT ERRORLEVEL 1 goto online
    echo %DATE% %TIME% The RPC Service has not started... >> log.txt
    echo %DATE% %TIME% The RPC Service has not started...
    )
    set CAUSE=has hung on Boot
    goto computerhung

    :online
    echo %DATE% %TIME% %COMP% is up – RPC Service is running. >> log.txt
    echo %DATE% %TIME% %COMP% is up – RPC Service is running.
    goto eof

    :computerhung
    bmail -s %MLSVR% -t %TOEML% -f %FRMEML% -b "%USERDOMAIN% – %DATE% %TIME% %TIMEZONE% %COMP% %CAUSE%" -h -a "%USERDOMAIN% – %COMP% %CAUSE%"
    echo %DATE% %TIME% %COMP% %CAUSE%. Skipping to next server. >> rebootlog.txt
    echo %DATE% %TIME% %COMP% %CAUSE%. Skipping to next server.
    goto eof

    :offline
    echo %DATE% %TIME% %COMP% not reachable... skipping >> log.txt
    echo %DATE% %TIME% %COMP% not reachable... skipping
    goto eof

    :nocomputers
    echo Make sure the file "%CD%\computers.txt" exists and is readable >> log.txt
    echo Make sure the file "%CD%\computers.txt" exists and is readable
    echo Stopping Script execution now. >> log.txt
    echo Stopping Script execution now.
    goto cleanup

    :eof

  • #13339
    Profile photo of Richard Siddaway
    Richard Siddaway
    Moderator

    look at using restart-computer with the -wait parameter – – that makes the process wait until the machine has rebooted.
    If you add the -For parameter as well you can tell the reboot process to check for specifics

  • #14499
    Profile photo of Tore Groneng
    Tore Groneng
    Participant

    hi,

    you can start with this?


    $servers = @("server1","server2")
    $Body = "server was rebooted at"
    $mailParam = @{
    To = "Dude@contoso.com"
    Subject = "Server rebooted"
    From = "noreply@contoso.com"
    smtpserver = "mail.contoso.com"
    }
    $servers | foreach {
    # Wait 60 seconds maximum for server to complete the reboot
    Restart-Computer -ComputerName $_ -Wait -for WinRM -Timeout 60
    if (get-service spooler -ComputerName $_).status -eq "running"
    {
    # good to continue, send email?
    $Mailbody = "$_ $body $([datatime]::now)"
    Send-MailMessage @mailParam -Body $mailbody
    }
    else
    {
    Write-error "Bail out, server is not up"
    break # this will exit the foreach loop
    }
    }

    You could even throw in some test-connection statements etc

    Cheers

    Tore

  • #13327
    Profile photo of Vern Anderson
    Vern Anderson
    Participant

    I'll need some time to analyze the whole batch file but this first response I will just try to give you the PowerShell equivalents to some of your commands.

    # TimeZone #
    Get-WMIObject Win32_TimeZone | Format-Wide caption

    # append to a file instead of ECHO > #
    "The Following Servers will be rebooted" | Out-File log.txt -Append

    # The for loop with content from a file #
    ## Keep in mind this assumes your currently logged on user is admin of those remote machines ##
    $computers = Get-Content .\computers.txt
    foreach ($computer in $computers) { Restart-Computer -ComputerName $computer }

    # echo the date and time into your log file #
    Get-Date | Out-File .\log.txt -Append

    # Sending email #
    # Whatever bmail is may still work but if you know your SMTP relay server info #
    help Send-MailMessage -Full # actually best to read this help doc (since I've never done this in production)

    # Checking is server is up # PING still works in PS #
    # Test-Connection is the CMDLET name ping is an alias #
    help Test-Connection -Full # it has some cool stuff so you should read the entire help doc #

    # I showed rebooting remote servers above #

    # interesting for /L # I am not familiar with what is in your %%a parameter #
    # I showed PowerShell foreach loop above #

    # GOTO computerhung #
    ## In PowerShell you create functions and then call the function at that point in the script instead of saying GOTO ##
    ## Functions once defined stay in memory and behave like an actual command, so you just run it at that point in your script but define it prior to ##

    # ECHO text to user #
    $datetime = [datetime]::now
    "$datetime $computer is starting up. . . please wait a minute. . ."

    # Query a service # tragically not as powerful as SC #
    Get-Service RpcSs

  • #14405
    Profile photo of Jake Sully
    Jake Sully
    Participant

    Hi Vern,
    I'm trying to get back to this, but if you can help me I would appreciate it. I did try to work on this using the equivalent commands, but I'm not making any progress.

You must be logged in to reply to this topic.