How to have an Optional Configuration

This topic contains 8 replies, has 2 voices, and was last updated by  Raymond Piller 2 months ago.

  • Author
    Posts
  • #96213

    Raymond Piller
    Participant

    What do I mean?

    I want the DSC LCM to be fine if it can't fine a configuration on the server that it's looking for. Unfortunately, if it encounters a configuration that doesn't exist (from say two partial configuration blocks), the DSC LCM will throw a warning (which isn't an error) and fails to process the other configuration that it did fine. Here's the warning:

    WARNING: [VM-WIN10-1607]: [] Executing Get-Action on partial configuration 'VM-WIN10-1607' Failed. Please ensure the parital configuration and its checksum exist on the server. Check the DSC event logs for further details.

    Note: I didn't spell "partial" wrong in the warning message; that was a copy/paste.

    What am I trying to accomplish?

    I want to pre-configure each computer to look for it's group configuration and it's individual configuration: GroupName, ComputerName. However, I only want to create a ComputerName configuration when it's needed.

    Alternate Option

    Can I use the main DSC Pull configuration to tell the LCM on the node to start pulling another configuration?

  • #96231

    Raymond Piller
    Participant

    I see a lot of flaws in this system ...

    • I can't target a configuration if that configuration doesn't exist. A warning will be generated, that should be an error. If one config is not found, none are processed by the LCM.
      • Also, putting that configuration back in place doesn't fix the LCM. It does stop warning/erroring, but doesn't apply any configurations.
    • In order to get a specific node to have a unique configuration, I can put another *Node* block in the configuration with that computer's name.
      • This would become a monolithic configuration with the configuration details for the entire infrastructure inside of it.
      • Too keep this manageable, I think I would basically need to have Jenkins create the config.
      • If a system got hacked, they would have the layout of the entire castle. Even if you encrypt the MOF files, those keys have to be sent to the node so the LCM can decrypt the MOF to read it. A decent hacker could pull the key from the cert store and decrypt the MOF.
    • In order to get a group of computers node to have a unique configuration, I would have to (likely with Jenkins) build the monolithic configuration to add those Resoruces to the *Node* block for each computer in the group.

    Possible Solutions to a monolithic config being sent out to all systems:

    • We would go back to the 4.0 method of using a ConfigurationID and having Jenkins generate a configuration that's unique to each system.
    • We use configuration names, and we somehow log and fix any 404s on IIS for machines requesting configs that aren't there.

    This is getting really complex for something that should just be handled. I'm still new to DSC so if I'm way off base here, please explain to me how I could be doing it better. Thanks.

  • #96234

    gael
    Participant

    Hi,

    Lots of questions there, here's a first stab at some of them:

    1. I'd recommend to stay away from Partial Configurations, the reasons have been discussed in the forum, and here's a good source of info for one of the reasons why: https://stevenmurawski.com/2016/03/dsc-partial-configurations-are-the-devils-workshop/

    2. When you explain what you are trying to accomplish, you seem to express the difficulty to compose the configuration in a manageable an flexible way. Shameless plug, I wrote about that here: https://gaelcolas.com/2018/01/29/the-dsc-configuration-data-problem/

    3. When you say "Can I use the main DSC Pull configuration to tell the LCM on the node to start pulling another configuration?". The short answer is that out of the box, it's not possible (AFAIK). The LCM settings aren't managed by the configuration but by the Meta Configuration, which can be set 'out of the LCM process'.

    4. In the first flaw you describe in your second post, that sounds like one of the fun things about partials, I'm afraid I can't help.

    5. Then what you describe a 'Monolithic' configuration, seems to assume that you need to put everything into one big file for all nodes.
    What you're trying to accomplish seems to be 'composing configurations' in way that's a bit more dynamic than the out of the box experience.
    I also wrote about composing roles (sorry, another self-promotion): https://gaelcolas.com/2018/02/07/composing-dsc-roles/

    6. When you say: "I would basically need to have Jenkins create the config", that's not mandatory, but it follows the best practices covered in The Release Pipeline Model anyway, so yes. A change to the source (be it a Node file, role definition or something else, should trigger a compilation of the MOFs.

    7. When you say "If a system got hacked, they would have the layout of the entire castle", that's more complex as you need to look at the components and artefacts throughout the pipeline:
    – It's true that the sources (configuration Data + Configurations + Resources) will show the detailed configuration of your whole infrastructure (that's the key benefit of Infrastructure as code). So if they have access to this, they have access to the whole definition (layout) of your infra (but is it that secret for someone's who already in your environment?).
    – The MOFs are encrypted at rest only one the managed Nodes, if you want to encrypt them at rest on the Pull server you could plug that into Tug for instance. When a Computer requests its MOF you can make TUG decrypt the MOF and pass it along via the standard protocol. Once on the managed node, it'll be encrypted. What you can encrypt within the MOFs are the Credentials, using asymmetric keys, which means the (private) key to decrypt credentials will only be stored on the managed node, and it will be Unique per Node. Encrypting those credentials requires the public key during MOF compilation time (i.e. on your Jenkins build agent).

    8. Finally, if you follow my approach to role composition, whether you use Configuration Name or ConfigurationID is nearly transparent to you.

    I am presenting this concepts at PS Summit next month: https://powershelldevopsglobalsummit2018.sched.com/event/Cpp7/an-opinionated-dsc-solution-with-tooling
    I am also co-delivering a workshop at PS Conf EU, where this will be covered: Workshop: Bring Existing Infrastructure under Code Control.

    To make this easy to manage I am writing a module called Datum (https://github.com/gaelcolas/Datum/), which also has some documentation (although, to little so far).

    Hope that helps, and feel free to reach out on twitter @gaelcolas.

    Gael

  • #96413

    Raymond Piller
    Participant

    The information you've provided is fantastic. I've spent the weekend (around family time) processing it all. I'm going to be looking at Datum (thanks for sharing) this week, and will comment more as I put it all together. I just need some time to flesh it all out.

    Thanks so much for you feedback! 🙂

  • #96840

    Raymond Piller
    Participant

    I'd love to start using and contributing to Datum, but I'm having some issues getting through the (as you said) limited documentation. I sent you a tweet, but thought I'd drop more details here so I could have some more room to breath...

    • I've not figured out how to get the $Datum to be sent to RootConfiguration.
    • I've not figured out what $ConfigurationData actually is.

    More Info ...

    From the DEV branch of DscInfraSample ...

    PS > $Datum = New-DatumStructure -DefinitionFile .\DSC_ConfigData\Datum.yml
    PS > $Datum
    
    Name                           Value
    ----                           -----
    Environments                   FileProvider
    __Definition                   {ResolutionPrecedence, DatumStructure}
    SiteData                       FileProvider
    AllNodes                       FileProvider
    Roles                          FileProvider
    
    
    PS > $Datum.AllNodes
    
    DEV          PROD
    ---          ----
    FileProvider FileProvider
    
    
    PS > RootConfig -ConfigurationData $Datum -Out "BuildOutput\MOF\" -Verbose
    ValidateUpdate-ConfigurationData : ConfigurationData parameter property AllNodes needs to be a collection.
    At C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules\PSDesiredStateConfiguration\PSDesiredStateConfiguration.psm1:1999 char:30
    + ...  $dataValidated = ValidateUpdate-ConfigurationData $ConfigurationData
    +                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : InvalidOperation: (:) [Write-Error], InvalidOperationException
        + FullyQualifiedErrorId : ConfiguratonDataAllNodesNeedHashtable,ValidateUpdate-ConfigurationData

    I assume that the environments are messing it up, but I also assume that I'm missing a crucial step. Maybe I'm just missing it an it is documented, but I've searched for $ConfigurationData in both repos and haven't had an luck.

    Thanks so much for your time.

    Update

    Ohhhhh ... there it is: ConfigData.build.ps1#L68-L71. I had previously searched for $ConfigurationData, not ConfigurationData. 😀

  • #96957

    Raymond Piller
    Participant

    Status update ... I'm working through issues with the RootConfiguration.ps1 that you have. The issue is that it doesn't just work for me when I just run it (maybe I'm doing it wrong):

    configuration RootConfig {
        Import-DscResource -ModuleName PSDesiredStateConfiguration
        # Import-DscResource -ModuleName SharedDscConfig -ModuleVersion 0.0.3
        # Import-DscResource -ModuleName Chocolatey -ModuleVersion 0.0.46
    
        Node $ConfigurationData.AllNodes.NodeName {
            (Lookup 'Configurations').Foreach{
                $ConfigurationName = $_
                $Properties = $(lookup $ConfigurationName -DefaultValue @{})
                Get-DscSplattedResource -ResourceName $ConfigurationName -ExecutionName $ConfigurationName -Properties $Properties
            }
        }
    }
    
    $Environment = 'DEV'
    $Datum = New-DatumStructure -DefinitionFile "${PSScriptRoot}\Datum.yml"
    
    $RootConfig = @{
        'ConfigurationData' = @{
            'AllNodes' = @($Datum.AllNodes.($Environment).PSObject.Properties | ForEach-Object { 
                $Node = $Datum.AllNodes.($Environment).($_.Name)
                $Node.Add('Environment', $Environment)
                if(!$Node.Contains('Name') ) {
                    $Node.Add('Name', $_.Name)
                }
                (@{} + $Node)
            });
            'Datum' = $Datum
        }
        'Out' = "${PSScriptRoot}\BuildRoot\BuildOutput\MOF\"
    }
    Write-Verbose "RootConfig: $($RootConfig | ConvertTo-Json -Depth 3)"
    
    RootConfig @RootConfig

    Doing that, I get the following error:

    PSDesiredStateConfiguration\node : The lookup returned a Null value, but Null is not specified as Default. This is not allowed.

    It looks to me like it doesn't like the Lookup command, so I confirmed:

    PS > $Datum = New-DatumStructure -DefinitionFile "Datum.yml"
    PS > Lookup 'Configurations'
    WARNING:  No Datum store found for DSC Resource
    The lookup returned a Null value, but Null is not specified as Default. This is not allowed.
    At C:\Program Files\WindowsPowerShell\Modules\datum\0.0.30\ScriptsToProcess\Resolve-NodeProperty.ps1:106 char:9
    +         throw "The lookup returned a Null value, but Null is not spec ...
    +         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : OperationStopped: (The lookup retu...is not allowed.:String) [], RuntimeException
        + FullyQualifiedErrorId : The lookup returned a Null value, but Null is not specified as Default. This is not allowed
       .
    

    Alright, so how about this:

    PS C:\Users\VertigoRay\CloudStation\Temp\Git\ECS> Lookup 'Configurations' -DatumTree $Datum
    Shared1
    SoftwareBase

    Much better. So, I tweaked that line to look like this:

    (Lookup 'Configurations' -DatumTree $ConfigurationData.Datum).Foreach{

    Now, I get a different error:

    VERBOSE:  Result found for Configurations
    PSDesiredStateConfiguration\Node : The term 'Shared1' is not recognized as the name of a cmdlet, function, script file,
    or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try
    again.

    Now, I need to do some includes. It's taking me a bit to get my head wrapped around things cause ... well ... definitely need more documentation.

  • #96983

    gael
    Participant

    Hey, not sure here's the best place to discuss/document this.
    GitHub issues on the project are probably better, and a bit easier to track for me...
    (I don't want to 'pollute' the forum).

    That said, because of the lack of documentation (sorry) you're reverse-engineering the project a bit.
    Give me a couple of days and I'll document the 'how to get started'.

    If you can't wait, have a look here at how I load the data.

    Resolve-Datum is what's called under the hood by the 'Lookup' alias. In short, it automatically use the variable $ConfigurationData, and $Datum from a parent scope (could be Global as well).

    In theory, the DscInfraSample project, you don't need to do any of that, it's already in place for you. You should be able to do just this:

    ./.build.ps1 -ResolveDependency

    This will start by pulling all the dependencies for the project (into BuildOutput\modules, DSC_Configurations, DSC_Resources), before kicking off the build (using Invoke-Build) and should create the MOFs, MetaMOFs, and Package-up the modules containing the resources for the Pull server in BuildOutput.
    There's a couple of problems in DscInfraSample right now (versions of modules/Dsc Resources) which means it won't work out of the box.

    Let me push a quick fix and I'll update this post shortly.

  • #96989

    gael
    Participant

    All fixed. You'll need a git pull (or better, rebase).
    Quick screen recording available on my blog, at the very end of 'Composing DSC Roles'.

    Gael

  • #96995

    Raymond Piller
    Participant

    I agree GitHub is a better place for stuff like this. I also prefer typing this stuff up in markdown than this forum. 🙂

    Thanks so much for the update. I've just started the build and will keep you updated in #8.

You must be logged in to reply to this topic.