Building a Desired State Configuration Infrastructure

This is a the kickoff in a series of posts about building a Desired State Configuration (DSC) infrastructure. I'll be leveraging concepts I've been working on as I've been building out our DSC deployment at Stack Exchange.

The High Points

I'm starting today with the general overview of what I'm trying to accomplish and why I'm trying to accomplish this. The what and why are critical in determining the how

The Overview

Goal:

All systems have basic and general purpose roles configured and monitored for drift via Desired State Configuration.

Reason:

System configuration is the one of the silent killers for sysadmin (yes, I prefer sysadmin to IT Pro - deal with it). In the case where deployments are not automated, each system is unique, a snowflake that results from the our fallibility as humans.

The more steps involved that require human intervention allow for more potential failure points. Yes, if I make a mistake in my automation, then that mistake can be replicated out. But as Deming teaches with the Wheel of Continuous Improvement (Plan, Do, Check, Act),  we can't correct a process problem until we have a stable process.

Deming Cycle

Every intervention by a human adds instability to the equation, so first we need to make the process consistent. We do that by standardizing the location(s) of human intervention.  Those touch points become the areas that we can tweak to further optimize the system.  I'm getting a bit ahead of myself though.

Let's continue to look at how organizations tend to deploy systems.  Organizations tend to have several levels of flexibility in their organizations about how systems are built and provided for use.  The three main categories I see are:

  • Automated provisioning from a purpose built image
  • Install and configure from checklist
  • Install and configure on demand

Usually, the size of the organization tends to indicate to what level they've automated deployments, but that is less true today.  Larger organizations tend to have more customized and automated deployments.  It's mainly been a matter of scale.  With virtualization and (please forgive me) cloud infrastructures, even smaller organizations can have ever increasing numbers of servers to manage, with admin to server ratios of 1 to hundreds being common and where the number of servers starts to overtake the client OS count.

If we aren't in a fully automated deployment environment, each server has the potential to be subtly (or not so subtly) unique.  Checklists and scripts can help with how varied our initial configurations can start out, but each server is like a unique piece of art (or a snowflake).

Try to make more than one of me...

That's kind of appealing to sysadmins who like to think of themselves as crafters of solutions.  However, in terms of maintainability, it is a nightmare.  Every possible deviation in settings can cause problems or irregularities in operations that can be difficult to track down.  It's also much more work overall.

What we want our servers to be is like components fresh off the assembly line.

Keeping it consistent

Each server should be consistently stamped out, with minimal deviations, so that troubleshooting across like servers is more consistent.  Or, even more exciting, if you are experiencing some local problems, refreshing the OS and configuration to a known good state becomes trivial.  Building the assembly line and work centers can be time consuming up front, but pays off in the long haul.

My Situation:

At Stack Exchange, we are a mix of these categories.  All of our OS deployments are driven by PXE boot deployments.  For our Linux systems, we fall into the first group.  We can deploy an OS and make the addition to our Puppet system, which will configure the box for the designated purpose.  For our Windows systems, we operate out of the second and third groups.  We have a basic checklist (about 30-some items) that details the standards our systems should be configured with, but once we get to configuring the server for a specific role, it's been a bit more chaotic.  As we've migrated to Server 2012 for a web farm and SQL servers, we've began to script out our installations for those roles, so they were kind of automated, but in a very one-time run way.

Given where we stood with our Windows deployments and the experience we had with Puppet, we looked at using Puppet with our Windows systems (like Paul Stack - podcast, video) and decided not to go that route (why is probably worthy of another post at another time).  That was around the time that DSC was starting to peek it's head out from under the covers of the Server 2012 R2 preview.  Long story made short, we decided to use DSC to standardize our Windows deployments and bring us parity with our Linux infrastructure in terms of configuration management.

Proposed Solution: Desired State Configuration

DSC offers us a pattern for building idempotent scripts (contained in DSC resources) and offers an engine for marshaling parameters from an external source (in my case a DSC Pull Server, but could be a tool like Chef or some other configuration management product) to be executed on the local machine, as well as coordinating the availability of extra functionality (custom resources).  I'm building an environment where a deployed server can request it's configuration from the pull server and reduce the number of touch points to improve consistency and velocity in server deployments.

Next up, I'm going to talk about how I've configured my pull server, including step by step instructions to set one up on Server 2012 R2.

Posted in:

19 Comments

  1. PowerShell on Linux exists to help Windows admins to help them transition away from Windows Server. But In a decade, when > 75% of workloads running on Azure are Linux, the tools used to manage those workloads will be designed for and meet the needs of the Linux administrator mindset. Linux administrators have a great set of mature tools and it isn't entirely clear why they'd want to transition to using PowerShell. Linux has won. The growth of Linux on Azure shows that. It's time for Azure to be a Linux First environment - and PowerShell on Linux doesn't really fit because it's primarily trying to graft a new and unproven Microsoft way of doing things onto successful practices and culture. Sure they can do it - but they'd be better off retraining everyone to "Think Linux".

    If Microsoft has (apparently) given up on Windows Server and understand that only a few legacy holdouts will want to run it in Azure in 10 years time (with the majority of workloads being Linux), they are wasting resources trying to reinvent how Linux administrators do things as a way of placating legacy Windows administrators. Legacy Windows administrators should bite the bullet and go "all-in" and adopt existing successful open source administration paradigms. The clock is ticking on their relevance and if they spend precious time investing in a nascent administration technology instead of fully transitioning to an open source mindset, they'll be less employable in future.

    PowerShell on Linux would be a neat idea if Windows Server had a future. It'll remain around in the same way that mainframes are still with us - but Microsoft has no interest in making a compelling case for organizations to choose their product over the free alternative. The future of Windows Server is the current reality of Windows Phone.

    • Thank you SO much for contributing your perspective! I do think - and I'm not a Microsoft fanboy per se - that you're misinformed, or at least under-informed. Time will tell, of course, but I suspect you've brought some personal bias to your viewpoint.

    • Interesting perspective. But I think you missed the point why Microsoft is actually open sourcing PowerShell. If you followed the talks that Jeffrey Snover did the last few years, it became clear that they want to be able to support heterogeneous environments. And for what I've seen now from Microsoft, and especially the PowerShell team, is that they don't have a hidden agenda. Microsoft is not the Microsoft anymore from let's say, 5-10 years ago.

      I agree with you, that there's a lot of Linux on Azure. But many, many companies I come are mainly Windows based infrastructures. Also guys that I know that work for other companies almost only see Windows based infra's.

      Yes, Linux has it's place in this world, but according for Microsoft there's no battle between Windows or Linux. They're citizens in IT which Microsoft wants to support best. And PowerShell is not a tool per se, it's meant to be a management framework. A framework that operates with built-in tools or in the case of Windows, the .NET framework.

      Windows Server has a big future, just look at the developments Microsoft is doing on Nano server.

    • Do you really think in 20 years well be dealing with "OS war" ?

      Do you think well see linux or windows in 20-30 years ? I dont.

      Do you think there is a loosing side or a winning side ? There are never winners in any war.

      From my POV, the shift towards lean kernels to accommodate the cloud, will get us eventually to a unified kernel of some sort getting the best of breed of all OSes, giving developers and IT the option to focus on the tools and the frameworks and less about the underlying layers.

      Running a business that creates OS is becoming very expensive. No one wants to be limited in the tools they want to use, thus the SQL on Linux is a huge huge thing in that sense and it will only get bigger with more such products going the same way. I have yet to see any major party offer anything similar things, coming from the Linux side because it takes money and effort very little companies have, so MS in that sense is helping transform the ecosystem again and its in a very good direction.

      Powershell on Linux exists so I, as a windows admin will have a lower barrier of entrance, if my boss decides one day to invest some our company assets on linux. If I can help my company get the right decisions that will save it money and achieve more and if that means going with a non MS way, guess what, I can still use my skills from the windows side without the hassle of the learning curve.

      For a long time I've been an advocate of learning both windows and Linux, no matter what I do mostly in my work time, as they are just tools to make the job, means to achieve a goal..they are not the goals themselves, and the movement to the cloud just emphasize it even more.

      I think your notion of what open source and free means is what's leading you in the line of thought and that's where I think you were wrong, imho.
      Not saying that my notion of what open source and free means is better, but its somewhat less biased. Nothing is free. Open source doesn't mean security (look at the horrible OpenSSL hole that's been there for two years and only recently been closed, or support-when-you-NEED it, that will always cost money, either by support contracts or having devs that know that specific language to deal with the bugs internaly (which by itself is even more limiting with the amount of languages and frameworks popping every second day).

      As for hidden agendas, you need to remember this is still a business. There's always money involved and business opportunities to be made. MS along they years was always good in creating those opportunities for itself and its partners and it continues to do so, the bottom line will be the tools. If you have ones that do the job for you, keep using them. If MS puts money and effort to create better tools with the community, who's the winner ? Everyone.

      I've seen this in the heated arguments on the PS repo the second it went public. The lack of broader vision some of the Linux base audience showed, the arrogance, the "Its mine, dont touch it" is somewhat alarming. I'm just happy to know that the sysadmins in 20 years, the ones born today well have a different starting point where they will choose the tools and be told what to use by old retiring sysadmins that are trying to hold to their precious seats instead of embracing change and supporting it in the evolving it world.

  2. If I have to copy files from installation folder to destination and I have to do exception handling because everything is automated, then how to do that? What all errors may arise and how to recognize and handle them? Please help.

  3. powershell ought to get the credit it deserves for enabling a developer to rapidly create rich output handling complex decisions based upon datasets gathered from various means and implementing a nearly infinite number of actions based on these. Simply put, it can be, and it is, much more than an admin tool, in the right hands.

  4. To start this is a thing of beauty in it's simplicity.
    Does anyone have experience with how much memory the results occupy and doing Get-job | Receive-Job at the end? If I run say a 1000 or 10,000 will this cause memory problems? I am thinking doing Get-Job -State Complete | Receive-Job & then |remove-job inside the loop (and logging it) would reduce the chance of running the host out of memory, or am I just over complicating it?

  5. a) Remoting
    The primary purpose of PS on *nix will be remoting to Win-Hosts, such as Bash on Windows vice versa.

    Due to the nature of *nix as document driven OS, an object based shell does not make that much sense. We're missing the API level. Jeffrey told us so, long ago.

    b) Religious affairs
    It's not about publishing the code (which is nevertheless great!).
    The GPL especially is the denial of the biz model that drives the revenue of Microsoft. So, indeed, haters will hate. Agree.

    But, in for a penny, in for a pound, PoSh is part of Windows which is an expensive, closed down product, increasingly incapacitating the user.

    c) The role of Community
    Sorry to say that, but the PS community is so much more than the few "Get-Expert -wellknown | Get-Random" MVPs. I know it's hard to see that inside the bubble.

    PoSh itself is gorgeous but - at the end - just a shell, such as Korn, C, Z and all the others.

    Far more important: the promise of a datacenter abstraction layer beyond the borders of specific vendors, automation and the refusal of a click UI.

    In this sense, publishing the underlying code is a statement which can't be exaggerated!

    Great Post, Don!

  6. Thanks, Don. This article saved me tons of frustration. I was writing a fairly simple script that would iterate through a list of servers and grab some WMI information. However, using Get-Content, I found that once the file went beyond some threshold, my script would no longer work properly. Implementing your method fixed my problem, and the script works perfectly.

  7. Do some one know about DSC using SQL database progress? I checked that Microsoft.Powershell.DesiredStateConfiguration.Service.dll is using OLEDB connection as an option for ESEN but when I configured to use OLEDB trough a SQL database it's showing an error "Registration of Dsc Agent with AgentID = b994f3ae-6888-11e6-8121-005056be155c failed. The underlying error is : Must declare the scalar variable "@agentId" , checking the ReadRegistrationRecordFromDatabase(string agentId, string databaseProvider, string connectionString) function is using @AgentId and I think that scalar variables with OLEDB need to be invoked using "?" simbol.

    Example:
    Query:
    dbCommand.CommandText = "Select * from RegistrationData WHERE [email protected]";
    Replace for:
    dbCommand.CommandText = "Select * from RegistrationData WHERE agentId=?";

    thanks in advance.
    Cesar

    • you cant use SQL Server as the database backend to DSC.

      ESENT is the default connection. IT did support OLEDB back in WMF 4.0 and I think it still does out of backward compatibility but going forward, I think they will stick to ESENT, at least until they fully open source DSC and then the DB schema will be open to use what ever ODBC driver you want. At least one can hope they do go OSS 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *