Be an Azure Consultant for PowerShell.org!

So, after our nearly-2-day outage, which was due to a still-not-fully-explained Apache fail, we're looking to make some changes. We need to migrate PowerShell.org to a different Azure subscription anyway, so this is a good time to change the kind of service we're using.

First, using Azure is non-negotiable. If your expert opinion is to use something else, please just don't ;). Update: This might be changing. AWS could be an option.

Second, our current environment is a classic VM running CentOS 6 (yeah, it's old), WordPress, and MySQL. WordPress and MySQL are also non-negotiable, this isn't about switching CMSs or anything. We use VaultPress for to-the-minute backups, but their restore process is a beast and has never been easy or reliable.

What we WANT is the ability to more or less push a button and re-deploy the entire site from backup, ideally automated through some OMS trigger that senses when the site has crashed.

Now, some caveats.

  • Our budget is $200/mo. We take about 150k-200k visitors per month, and WordPress is a reasonably demanding piece of software. We have about 500MB in files and about 300MB (currently) in data, and we grow about 75-ish MB a year.
  • We would ideally like the data backups to be distinct from the site itself. That is, if we could simply kill an old server and deploy a new one, and then drop the data onto it (all automatically), that'd be ideal. This is distinct from simply backing up an entire VM image, since the OS, files, and data would all be one chunk.
  • It'd be lovely if, instead of having to patch and upgrade the OS, we can just kill the current server, deploy a new one with all the new hotness in versions, and drop the data on it.
  • The more of this that lives in Azure (e.g., Backups), the more likely - we feel - we'll be able to automate this entire kill-and-deploy process in OMS or something.

Staying on Linux is fine. It also isn't a pre-req. WordPress (and MySQL, since WordPress doesn't play well with much else) are the main requirements. We'd like to be on a modern (6+) version of PHP, as well.

Update: So, let me outline the kind of thing we're thinking. So far we've gotten a lot of suggestions on which OS to use, or which DB to use, and that wasn't really the question so much as the architecture. For example:

  1. Run a small, on-demand staging instance where patches (WordPress and plugins) are applied to the site. The site's folder is under Git, and after applying updates and testing, we push to a private (because configs contain passwords) GitHub repo. This instance isn't backed up - the important bits are in GitHub.
  2. Use ____ to deploy the actual instance(s) that people will use. This deployment is a la Elastic Beanstalk, where you just push a base OS image and it sucks down your GitHub repo to populate the files of the site. Again, not backed up - GitHub is the backup.
  3. Except for the WordPress Uploads folder, which you redirect to another, simpler instance that serves these as static files. This is a bit complex, because WordPress needs file-based access to this for uploading, while it also needs to be exposed as a web server for downloading. Simple backups to ensure we have copies of the files handy, and we don't need to retain a backup history because there's no code that could break and need to be rolled back.
  4. Data lives on a distinct hosted RDBMS. That's probably MySQL, as it's what's supported with WordPress. We're aware of Namiproject, but unless that's moving in lockstep with the base WP releases and is 100% guaranteed to work with all the plugins we need.... The RDBMS is backed up separately.

A concern with #4 in Azure is that they only offer this (for MySQL) through ClearDb, and I've seen latency and persistency problems. I'm ideally wanting everything hosted in one datacenter/region to reduce that problem. And I'm aware of Namiproject, but we have something a bit more complex than a stock WP install, so someone's going to have to convince me that SQL Server's a safe choice.

So... any suggestions for a full-stack? Please, be serious and complete - if your suggestion is, "just run VMs," that's not helpful and it'll likely be deleted. But if you've got ideas for a mix of services that you think will do the job - by all means, please, speak up!

Posted in:
About the Author

Don Jones

Don Jones is a Windows PowerShell MVP, author of several Windows PowerShell books (and other IT books), Co-founder and President/CEO of PowerShell.org, PowerShell columnist for Microsoft TechNet Magazine, PowerShell educator, and designer/author of several Windows PowerShell courses (including Microsoft’s). Power to the shell!

29 Comments

  1. Have you considered just getting rid of the VM altogether? WordPress should run just fine directly in azure websites with
    No VM necessary. I don't know all the details of backup and restore off hand, but it's probably not to bad to do via Automation/OMS or some other built in mechanism. Only question then is PHP version.

    • It does run fine; the problem is in the database layer. Azure uses an outsourced MySQL provider and it's not without stability problems. There's also the concern about maintaining a deployable base image given how WordPress sucks down updates, and in dealing with the WordPress Uploads folder.

  2. Hi Don,

    While not an expert in WordPress or php, i do have some ideas on what your best approach might be.

    I would start by:
    1. Configuring your base VM needs with Azure Resource Manager template, include adding the PS DSC add-in to the VM
    2. Script in Powershell the DSC configuration for installation and configuration of the PHP , MySQL, WP (I am in fact working on this particular part as a Proof Of Concept for a colleague ) and have this applied to the VM.

    That will give you your base install and configuration for the site.
    there are a few different ways you can automate Backup and recovery of the MySQL db as Azure has pretty good backup options.
    Once back and recovery is deided on Azure automation can easily be plugged in.
    I use azure for all my VM's but not for installing WordPress on a VM as i use the Azure Web App with a WP install and i connect the recovery site vault.

  3. So, like you saw App Service has MySQL in preview right? You could run your site and MySQL on the same App Service Plan (this is the thing you are billed for). Though it's in preview you'd want to talk to the product group to see about maybe helping you help them 😉 App Services can be backed up daily or with premium plans up to 50 times daily. Upgrades would be cake; create a deployment slot cloning the production app+db, do your upgrade on the slot, if it works you can either swap the slot if you had new content locked or go and do the upgrade on the real production one (i'd create a fresh slot copy then do the upgrade that way you could just swap the slot to rollback).

    So say you don't want to muck around with a preview feature, I'd recommend using an App Service to host the website and then just a right sized Linux VM to host MySQL. When MySQL in-app is GA you'd then just migrate to that and reduce a little bit of spend or shift the spend from the Linux VM to a larger App Service Plan.

    https://azure.microsoft.com/en-us/blog/mysql-in-app-preview-app-service/
    https://blogs.msdn.microsoft.com/appserviceteam/2016/08/18/announcing-mysql-in-app-preview-for-web-apps/

    • Yeah, the problem is it's in-instance, which doesn't achieve the separation of data. If I redeploy the instance due to a failure, I have to re-do the data as well, which is difficult and time-consuming. And I'm not looking to maintain an OS (e.g., in a VM) if I don't have to - I'd much rather the cloud handle that for me. That's kinda the point.

  4. You might want to take a look at Project Nami, http://projectnami.org/. Its a project to get WordPress working with SQL and by extension Azure SQL. I've only used it in limited fashion and mostly small personal projects with Azure Web Apps and Azure SQL. None of my projects were on the scale or scope of PowerShell.org but I was impressed with its capability and completeness. Getting into the PaaS side of Azure and off the IaaS side might help with the scale and backup side of things. The OS updates are a non-issue on Azure Web Apps and Azure SQL. I have used the deploy from GitHub for Azure Web Apps and its pretty smooth process. For WordPress the bulk of your data/config is going to be in the DB. My specific knowledge in Azure SQL is far far less, but I image it has some backup/snap/restore process as its PaaS offering so the expectation is you get a knob or a button as apposed to a "good luck, roll your own" mentality of IaaS. Seems like Azure Automation (and OMS) could glue it all together and give you the "One" button, "One" view that you want.

    • Yeah, and that's my concern with Nami. But we're going to look into it, and maybe set up a test environment. But your thinking is aligned with ours right now.

      • With the load you describe above, you're probably looking at a single Medium Web App environment and anywhere from an S0 to S2 for the DB depending on how many users you have in the backend at any given time. Might even get by with a Small and a Basic DB after you test with it.

        The secret sauce for handing public traffic is our Blob Cache platform. Works with either PN or vanilla WP on Azure Web Apps.
        http://projectnami.org/blob-cache/

  5. Do you have any numbers of peak and average compute resource utilization of the server currently running they website, to get an idea of what will be required.

    • We do, although at the architecture stage I'm not sure it's needed. We know we only need a single Webserver instance to handle peak traffic, and we can size that once we figure out the architecture.

      • Based on the updated information in the original post, I would suggest for a docker implementation. Something like https://azure.microsoft.com/en-us/marketplace/partners/docker/wordpressplusmysql-arm/

        You said the data is stored in github so in your Dockerfile you should include a step to install git and perform a git pull to automate the processes needed.

        The base image uses ubuntu 16.04 which is very stable now and has plenty of support including support for PHP 7.0 which was one of your points. I have run wordpress on PHP 5.5 and 7.0 and there is reduced overhead using PHP 7.0.

        The system could be easily redeployed by using the base image and the small customized Dockerfiles within minutes.

  6. There is a possibility to dockrized both MySQL and WordPress and host them within Azure container service.

    That would make the solution easier to maintain - clone/baclup/restore
    Using standardized methods allready in use with docker and Azure, so no need to reinvent the wheel.

    Dockrized Php is supported as well
    And 3rd party components can be integrated via Iaas (separate vm) or paas services APIs.

    The Data and configuration files can stay in the container data store or be separated into different data store in a dedicated storage account.

    For governance I would use OMS combined with Azure runbook /worflow for the backup/restore automation

    • The primary problem for deploying a small site on Azure Container Service is that there is a minimum of three VMs, one of those must be the Master (at a cost of $84/Month) which would leave the site running on 2 D1v2 VMs - the alternative is deploying docker without ACS, at which point you might as well deploy a full VM.

  7. The first thing I would suggest doing here would be to gear the solution to something that is open sourced. To be improved and adapted as required.

    I would split the solution into five steps. 1. MySql Deployment 2. HTTP server deployment. 3. Backup processes. 4. Restore / Redeployment processes. 5. Orchestration

    Steps 1 & 2 could either be run by a locally installed CM engine client (Puppet, Ansible etc) or via a simple script that runs when the VM is deployed.

    The easiest / quickest way to perform backups would be to dump them directly to Azure FIles / Blob storage. When a new server is deployed it would suck those backups back in and continue as before. for the size of data you have this would be <1 min

    Orchestration is where the options get more complex. I would look at having the actual Azure build code (probably ARM template, but could be PS code) located in either Azure Automation or Azure Functions. This could be called with Parameters to deploy / redeploy / deploy test etc. If you have your DNS in Azure DNS, it could simplify failover.

    You run TeamCity so you could use that as a CI/CD engine, to test deployments and be sure that everything does as expected (especially useful if you make it a fully open sourced solution)

    I don't think It isn't an overly complex solution, at its very simplest it could be a single ARM template residing in Automation called from a webhook. Anything above that should be aimed towards adding flexibility / resilience to the solution.

    I run a Slack Channel at - http://azured.io - feel free to drop by and we can all help figure the best solution out.

    • Thanks for the detail, there, Michael - I've added some detail above. The MySQL bits remain my concern; I'd rather not own the VM per se, and rather buy a hosted instance, which Azure doesn't natively do yet.

      • I think there is the problem of trying to change too much in one go. With a good CM/CI/CD pipeline you can make incremental changes much more reliably. By separating the logic that deploys the DB and WordPress you can split them into separate solutions at some point in future if that is what is required. At the current stage though I would suggest moving things over and making sure that things fail reliably (in way that can be detected prior to deployment)

        The underlying Azure architecture is rapidly moving, the primary focus should be creating a solution that can be easily adapted and changed. When it all works, it would make investigating things like Docker, Nami etc. You could then deploy both side by side and push a small percentage of traffic to the other to see how it works.

        My personal recommendation would be to pursue reliability, use what works, what you know, that has the ability to self-heal. Unfortunately, in Azure that means running a VM for MySQL (I'm aware that it runs in App Service, but relatively new and untested)

  8. I hate managing MySQL and a WordPress site. I went to a fully hosted WordPress using Pagely and never looked back. Being fully hosted does limit somewhat to customization but you can still put whatever them, plugins, etc on your site as you wish. You can also work with them to do any more customization. They also perform all the backups for you and manage the restore if things go awry.

    I've got DevOps stuff and PowerShell to do; not tinkering with WordPress.

    If Pagely is too expensive, I'd look at other hosted solutions instead of rolling your own.

    • I totally <3 wpengine for my blog, for many of the many pain points outlined above and after trying various solutions with aws and azure over the years (read about it on my blog if ya want). I would recommend wpengine for $99 per month with cloudflare in front of it to reduce visits to the actual site by serving most up through the CDN.

      https://wpengine.com/plans/

      (I set myself up an affiliate link with wpengine recently so if you do find you're interested let me know - I can save a bit on my own hosting that way)

    • We're -really- heavily customized, or a hosted solution would be ideal. My goal is to move to a PaaS model, which is about halfway between what we're doing now and fully-hosted. But I'll definitely revisit Pagely with all this in mind.

      • I know you would prefer Azure and AWS, and there have been several other WordPress hosting options presented - but I highly suggest that you take a look at SiteGround.com.

        They are one of 4 web hosts recommended by WordPress.org for WordPress hosting. Here's a link "straight from the horses mouth".

        https://wordpress.org/hosting/

        They have a fully managed cloud hosting option which uses Linux containers and is backed by SSD drives. You can auto-scale to meet traffic spikes if desired, and they also offer free CDNs from multiple locations.

        https://www.siteground.com/cloud-hosting.htm

        I've been in the Windows hosting business for nearly 20 years, and I can honestly say that its extremely rare that you come across a web host with this level of expertise on both the hardware and the software side. IMHO, they know Linux hosting like you know PowerShell.

        Of you talk to one of the SiteGround support engineers, tell then who you are and what you are looking for, I expect they will very likely be able to provide you with a turn-key solution that meets your needs.

        I know this probably sounds like a commercial, but I assure you that my only reason for writing this is so that you can stop worrying about hosting your WordPress site, and get back to teaching the world how to use PowerShell. WordPress is a PITA to host, and these guys take all the pain away.

        Hope this helps, and thanks for sharing all the great PowerShell knowledge.

        Get-WebHost -Provider SiteGround -Worry $false
        Write-Verbose "The puppy lives"

  9. Noting that AWS is now an option, this would allow you to use native services that are perhaps a bit more mature than the Azure offerings are currently.

    The backend database could be handled by RDS or Aurora, which allows you to bring in a level of HA that you would be lacking if the database was stored on a single VM. Backups can be scheduled, and you could also back up to S3 with a retention policy if that is something that you feel you need.

    In terms of the separation of files from the WordPress installation, this could again be achieved by using the S3 object store for your data. The WordPress install itself could be deployed to two instances in separate availability zones that are reached via a load balancer.

    Those instances could be placed in an auto-scaling group, which would automatically replace an unhealthy vm with your latest AMI, with your CM tool of choice doing any extra config that you may need.

    The main advantages to this, as I see it, are that:

    1. Your environment is deployed entirely with the same provider with no external dependencies
    2. You're taking advantage of native services that the provider will manage for you, which removes an element of administration
    3. It provides you a level of HA that you may not currently have

    If you wanted to, you could also use CloudFormation as an option to store the infrastructure as code for an automated re-deployment if required (bearing in mind that this would also enable you to deploy a test environment quickly and painlessly to test updates on, that you can then just tear down afterwards to save on ongoing costs).

    • I've been very reluctantly coming to more or less the same conclusion. All of this will be much easier in AWS. I'm doing some load testing in Azure right now, and the Web App performance isn't stellar for the price :(.

      • If you're set on running the infrastructure yourself, which I presume you are, then at the moment AWS is definitely the way to go.

        Having said that, if you can make your provisioning processes as provider-agnostic as possible, then transitioning back to Azure once there's a compelling reason to while not being trivial shouldn't be too difficult.

  10. Project Naminis rock solid, been running it for.over a year now and sizes to what you decide you need. It has a great compatibility with plug ins (haven't found one I needed yet that didn't work) and ditches your underlying infrastructure because it is PaaS. This buys you your independence from OS layer (as you already know) and gives you scale.
    You hook it up to git for deployment, run a second instance for staging environments, do A/B testing if you like and gets updated and redeploys from your git.
    For an update and sync with WP, I have seen on average less than 24 hr sync with release cadence for the past year - they are on top of it.
    I use a caching layer with one plug in built for Nami and blob services /CDN layer and I can size my site up or down (and SQL) as I see fit. I migrated mine by doing an wp export from site, cleats a new Nami site and import, then ftp'd my wp content folder over. Then tweaked my CSS and plugins to match old site.
    Done and done, won't go back.
    I hated crappy MySQL 3rd party service and love my SQL scaleable options now.

    And the 5k azure 503(3) option will go a long way running services for ya.

  11. I am not a WordPress expert but I do believe there is a WordPress extension that will allow you to store media in Azure Storage. Once you offload your media to Storage and your database/config to a separate database service, that will give you some flexibility for the actual WordPress site.

    At that point you can host the WordPress site in a PaaS service that handles backup/restore automatically for you since it's simply cloning an application and not any data/configuration.

    Good luck with your migration!