1.5 Setting up a local environment to mimic production

Setting up working environments with the magic of DevOps.

Posted by Dcycle on April 20, 2015
Home > Class 1: Basics > 1.5 Setting up a local environment to mimic production

The challenge of local development

Recall the challenges posed by local development:

  • Keeping track of different versions of the same project.
  • Making sure the project behaves the same way on each environment.

We had a glimpse of the first challenge in the previous lesson, as we looked at using Git. We’ll look more into Git in a future lesson.

The second challenge has long haunted the development community, but a solution is in sight, and we’ll start using it in this lesson.

Different types of websites

OK, I know that you know how a website works, but bear with me: your browser sends a request to a server, and in return gets html, css, and javascript files. Together, these make up what is displayed to you.

There are three main ways to deliver these files:

  • Option one: The first is to write them directly to text files yourself, and then save them to a webserver, or even to your desktop. Sites built in this way can be viewed directly on your computer without any special software, by opening them in your browser directly. Non-trivial sites are never built this way because the technique is error-prone and not very powerful.

  • Option two: You can also write your sites in languages other than, or in addition to, html, css, and javascript. These sites need to be built, that is, transformed into files your browser understands; the resulting files can then be viewed in your browser. This is the technique we’ve used in the blog we created in the first lesson: it uses a technology called Jekyll to take files which your browser does not understand, and transform them into files it does. These files are not built every time you visit a page; rather, they are built at a set interval. For example, GitHub builds your site every time it detects a change to the underlying code (changing a site’s title, for instance).

For example, Jekyll takes yml files such as _config.yml which contain information like:

title: Clean Blog
description: "A Clean Blog Theme by Start Bootstrap"

Every time a change is made to a file such as the above, Jekyll can transform that information into HTML like:

<div class="site-heading">
    <h1>Clean Blog</h1>
    <hr class="small">
    <span class="subheading">A Clean Blog Theme by Start Bootstrap</span>
</div>

Don’t worry if you don’t fully understand how HTML works; the important thing is: your browser understands it, and can display it.

  • Option three: The third option, used by Drupal, which is written in PHP, interprets source code every time a request is made, and uses software to build HTML on the fly.

For example, a PHP file like:

<?php
$rand = rand();
print ('<h1>This is a random number: ' . $rand . '</h1>');
?>

Will be interpreted as:

<h1>This is a random number: 83748</h1>

The next time you load your page, you might see:

<h1>This is a random number: 38292</h1>

In all cases, your browser sees html, or other formats it understands, and does not care about underlying technology.

If you are using option two or three, your site will be not be visible on your computer because you don’t have the required software to translate it into html (PHP, webservers, Jekyll, etc.). Creating a local environment requires that software.

Three approaches to creating a local environment

Assuming you are not modifying code directly on the production environment, here are three approaches to local development:

  • The worst thing you can do is try to install that software directly on your local computer! If you are developing a PHP application like Drupal, software packages like MAMP and XAMPP might work at first, but you will soon run into software mismatches, which can eat up days or weeks of your time. You do not want to deal with permissions issues, software version mismatches, and other weird issues. If you want to build your Jekyll site, you might try to install Jekyll on your Mac, for example, and it might work, but your colleague who has Windows might hit a wall. There has to be a better way! And there is…

  • The next approach is to use virtual machines. So: if you are not installing software on your laptop, perhaps you can create virtual machines which contain the required software. We already know how to use Vagrant, we’ve used it to set up a virtual machine which runs the CoreOS operating system. Let’s say your website is designed to use the Ubuntu operating system; now that you know how to use Vagrant, you can set a VM with Ubuntu. What about the software? You might have heard of configuration management tool such as Puppet, Chef, or Ansible that define what should be on your machine: For example, PHP 5.3, Apache, MySQL are required to run Drupal. To be honest, I do not recommend this because virtual machines are expensive. We already have a CoreOS VM, and we don’t want to bog our computer down with a bunch of virtual machines, one for each project we’re working on.

  • My personal favorite, containers: what if we could have the benefit of virtual machines, but with the speed of a local machine, all without having to bother with learning Puppet, Chef, or Ansible. Containers do exactly that. What’s wrong with VMs? VMs replicate a complete operating system (OS) as a guest, living on a host. So: a lot of what makes your OS work and take up resources is replicated on each virtual machine. This is inefficient. Containers, on the other hand, share resources with the host, which makes them much more nimble.

Remember how Vagrant is a command-line tool to easily manage VMs? Well, it turns out there is a tool which does the same thing but for containers: Docker. Not only is Docker easy to use to manage containers, it uses caching very well. This means that when you build a container a first time time, it can take a while (still shorter than building a VM); when you build it a second time, it takes about one second.

If you are still confused, or want to learn more about containers vs. VMs, I highly recommend the article Getting Started with Docker, on Servers for Hackers (March 20, 2014).

OK, so if containers are the best thing since sliced bread, then why did we install CoreOS as Virtual Machine, rather than a container? Containers rely on an underlying technology which is not available on Mac OS or Windows. The basic premise of CoreOS is to make an operating system as small and efficient as possible, include necessary software, and work with containers. That’s why we installed CoreOS: so we all have the same starting point to run containers via Docker.

Exercise: setting up a local version of the site you’re on right now

Dcycle U. is an open-source site; you can copy its structure and content, and do what you want with it, as long as you mention its source and don’t claim it as your own.

Go to the Dcycle U. code page on GitHub and locate the clone URL:

Note that because you don’t the permission to modify the Dcycle U. code directly (although you can fork it if you want), you don’t see the SSH clone method, as you do for your own projects. Keep the HTTPS method, which is fine for our purposes. The difference is that you don’t need an SSH key to use it, and you won’t be able to push changes from your computer to GitHub.

Now go to your CoreOS machine, and clone Dcycle U:

core@core-01 ~ $ cd share
core@core-01 ~/share $ git clone https://github.com/dcycleproject/dcycleu.git
Cloning into 'dcycleu'...
remote: Counting objects: 163, done.
remote: Total 163 (delta 0), reused 0 (delta 0), pack-reused 163
Receiving objects: 100% (163/163), 5.46 MiB | 1.25 MiB/s, done.
Resolving deltas: 100% (40/40), done.
Checking connectivity... done.
core@core-01 ~/share $ cd dcycleu
core@core-01 ~/share/dcycleu $

Understanding the files

Dcycle U. has a few files which are required to set up a local environment:

  • Dcycle-Dockerfile-jekyll, a file which describes how to build a Jekyll server.
  • Dcycle-Dockerfile-jekyll-test, same as above but tailored for testing.
  • Dcycle-Dockerfile-nginx, a file which describes how to build a webserver.
  • dcycle, a folder with scripts needed to deploy these two containers and make them talk to each other.

Don’t worry if you don’t understand every file. The important thing is that these file contains everything needed to set up the servers necessary for Jekyll: one server is used to build your website (transform the Jekyll files into HTML files); the second server is used as a webserver to publish your new site. It contains Nginx, a webserver good a serving HTML files. For our purposes, it replaces Apache, another webserver you might have heard of.

With this recipe, Docker will be able to completely create a Jekyll-ready server cluster as containers—the equivalent of having two servers at your disposal—and destroy them when it is done with them.

./dcycle/deploy.sh is a script file, and it contains a series of commands. Typing the name of this script is the equivalent of typing its contents in the command line.

./dcycle/deploy.sh will then call other scripts, the most important one being ./dcycle/lib/build-dev.sh. Before creating our server, let’s look at what our scripts are doing. Open ./dcycle/lib/build-dev.sh and look at the code. Here are the basics:

  • Use docker, the tool we use to manage containers and which is included in CoreOS, to build two new images based on the Dockerfiles which we’ve seen above. Give our images a unique name. Think of an image as a hard disk which is not plugged into a computer.
  • Once we have our images, create new containers with them. Think of a container as a running computer with which we can interact. -p $PORT:80 makes sure that we can access our Nginx container using a port we can specify. Port 80 is the default port used by web browsers, so we’ll use that one in the following examples; if port 80 is not accessible, web browsers will not be able to fetch web pages from a server.

Notice that, using -v, the Jekyll container has access to $(pwd), that is, the directory which holds all of your code, and shares that directory with its internal path /srv/jekyll/. The Jekyll container’s job is take all your code and transform it into a website, which it puts into a subdirectory called _site (that’s how Jekyll works).

The Nginx server does not need access to the entire code of your site, so -v $(pwd)/_site:/usr/share/nginx/html/ only gives it access to the _site subdirectory, and puts it into /usr/share/nginx/html/ on the container, so it may be published.

Let’s recap how files are shared. In lesson 1.3. we put our vagrant machine at ~/Documents/coreos-vagrant on our laptop.

  • ~/Documents/coreos-vagrant/dcycleu/README.md on our laptop is the same file as ~/share/dcycleu/README.md on our CoreOS VM.
  • Once we have a running Jekyll container, ~/share/dcycleu/README.md will be the same file as ~/Documents/coreos-vagrant/dcycleu/README.md on our laptop, which will be the same file as /srv/jekyll/README.md on the container, which itself is running on top of CoreOS.
  • ~/share/dcycleu/_site will be the same directory as ~/Documents/coreos-vagrant/dcycleu/_site on our laptop, which will be the same directory as /srv/jekyll/README.md on the Jekyll container, will be the same directory as /usr/share/nginx/html/ on the Nginx container.
  • The files are actually the same, they are not copies, but the same file: Changes to README.md in one place will change the file on all locations.

Running a local environment

Start by checking which containers are currently running by using the docker ps command.

core@core-01 ~ $ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
core@core-01 ~ $

That’s normal: you have no running containers yet. Time to run the script!

core@core-01 ~ $ cd ~/share/dcycleu/
core@core-01 ~/share/dcycleu $ ./dcycle/deploy.sh -p80

Depending on your internet speed, this might take a while the first time you run it. When it’s done, check your running containers again. You should have one.

core@core-01 ~/share/dcycleu $ docker ps
CONTAINER ID        IMAGE                              COMMAND                CREATED             STATUS              PORTS                         NAMES
0fd2d9d3ba10        docker-dcycleu-dev-nginx:latest    "nginx -g 'daemon of   About an hour ago   Up About an hour    0.0.0.0:80->80/tcp, 443/tcp   berserk_ptolemy
8ec35228fce8        docker-dcycleu-dev-jekyll:latest   "/usr/bin/startup /b   About an hour ago   Up About an hour    4000/tcp                      serene_morse

Because your Nginx container is accessible on port 80 (the port that websites use), you will be able to open a browser on your computer, and visit http://172.17.8.101 (that’s the internal IP address of your CoreOS VM).

Are you seeing the DcycleU website? Great! If not, file an issue.

OK, time to see one of the main advantages of containers: speed. Start by killing the Nginx and Jekyll containers. This will not touch your code, it just kills the container where it is being run. First, find the ID of your running containers. In this example, 0fd2d9d3ba10 and 8ec35228fce8. Then kill them with:

core@core-01 ~/share/dcycleu $ docker kill 0fd2d9d3ba10 8ec35228fce8

You can also use 0fd instead of 0fd2d9d3ba10 if you don’t like copy-pasting. If you visit http://172.17.8.101 or run docker ps now, you won’t see that container anymore. Try building the container again and count how long it takes:

core@core-01 ~/share/dcycleu $ ./scripts/deploy.sh -p80

About one second, right? Keep in mind that we are doing the equivalent of starting up two servers, booting two operating systems, and starting all relevant software. That’s the speed of Docker.

Make a change to Dcycle U.

On your computer now, find Dcycle U.’s _config.yml file, make a change to the site’s title using your text editor, and save it.

Changes should be reflected in your environment within a few seconds.

Exercise: changing your own site locally

OK, now that you have made to the Dcycle U. site, you can delete that folder.

Also, make sure you kill the Dcycle U. container, or else no other container will be able to exist on port 80, and you will get the error Cannot start container ...: Bind for 0.0.0.0:80 failed: port is already allocated.

Recall the files within the Dcycle U. that allowed us to create containers and update our site:

  • Dcycle-Dockerfile-jekyll.
  • Dcycle-Dockerfile-jekyll-test.
  • Dcycle-Dockerfile-nginx.
  • dcycle (a folder).

Dcyclebox is a project I have set up to house all these files independently from particular projects. To get the latest versions of the above files, download the Dcyclebox zip file, unzip it, and copy all the contents from move-to-your-project-root/jekyll to your project.

  • Navigate into your folder using cd ~/startbootstrap-clean-blog-jekyll-gh-pages, on the CoreOS VM.
  • Run dcycle/deploy.sh -p80.
  • Make any changes you want directly on your computer.
  • See them reflected on your local environment after a few seconds.

Exercise: push your changes live

If all went well, your local environment at http://172.17.8.101 should contain your changes, but not the site as it is hosted for the public on GitHub.

Changes are pushed via Git. Because we added files to our Git repo (dcycle/deploy.sh, for example), we need to add them before committing, using Git’s add command. (Note that adding a folder, as we will do with dcycle, will also add all the files within that folder.) Here’s is what to do:

core@core-01 ~/share/startbootstrap-clean-blog-jekyll-gh-pages $ git add Dockerfile-dcycle-jekyll
core@core-01 ~/share/startbootstrap-clean-blog-jekyll-gh-pages $ git add Dockerfile-dcycle-jekyll-test
core@core-01 ~/share/startbootstrap-clean-blog-jekyll-gh-pages $ git add Dockerfile-dcycle-nginx
core@core-01 ~/share/startbootstrap-clean-blog-jekyll-gh-pages $ git add dcycle
core@core-01 ~/share/startbootstrap-clean-blog-jekyll-gh-pages $ git commit -am 'Added Docker support'
core@core-01 ~/share/startbootstrap-clean-blog-jekyll-gh-pages $ git push origin gh-pages

If you add files to your local repo, Git will ignore them until they are added with git add. To see if Git is ignoring anything you can type git status.

After less than a minute, the changes you made locally should appear on your live, or production, site—the one hosted on GitHub.

Recap

It is useful to take a minute and understand all we have accomplished, as this solid base will prevent countless headaches in the future:

  • All our changes can now be under version control. This prevents confusion about what is the most up-to-date version of our code.
  • We are using a Dev-Production workflow (later on, we’ll use dev-stage-production). This allows not just you, but anyone to whom you give access, to make local modifications on their computer, and push them live.
  • We are using a DevOps workflow. This allows anyone on any computer on any OS to start developing right away. All that is needed is Vagrant, VirtualBox and CoreOS VM. If you have worked on teams which did things before DevOps, you know the colossal amount of time you will be saving by not having to debug Tom’s development environment because he does not have PHP mb_string extension installed, or Sylvia’s environment because she does not have the permission to chmod _sites to 755, your own instance of Jekyll because you just upgraded Mac OS and it broke everything, and innumerable other things you don’t care about.

Before moving on:

Before looking at the next lesson, you should now be familiar with:

  • Adding files to git using git add.
  • How to list Docker containers running on your CoreOS VM.
  • How to kill a Docker container.
  • What people mean when they say DevOps: embedding the server information with the project code to be able to create and maintain standardized environments.
  • Two DevOps tools, what they do, and how they differ: Vagrant, and Docker.