As the tools we use become increasingly more powerful and extendable, with it comes the temptation to over exploit their functionality. This isn’t always a bad thing, but sometimes we get lured into using tools for purposes in which they were never intended. This can have adverse effects in the long term. Let’s look at those in turn:

As hard as we try, oftentimes we cant avoid having to use certain tools. Whether its because we don’t have the time or skills to build our own; sometimes we just have to buy. My first suggestion is whenever you adopt a new tool you should always think about an exit strategy. I’ve seen companies stuck in bed with vendors who’s license costs rise exponentially on an annual basis, and they find themselves debating whether they should use their £million maintenance money to move off a tool/platform. Mainframe anyone?? Don’t let a tactical decision end up as your strategic solution!

When you leverage a tool for functionality outside of the norm, you run the risk of those features being removed at any time. I once had a scenario where some colleagues had extended Jenkins to create a web form which would post rest calls to another back-end system. They did this through the use of a plugin that had a fairly small community for support. The problem here is when we needed to upgrade Jenkins to get the latest patch fix, this plugin was still lagging behind and had broken with the upgrade. In this instance a simple X on rails framework would have done the job and broken any dependencies on Jenkins.

Increasing tool load. Lets go back to the Jenkins example above. If we have thousands of people filling in this web form and submitting to backend systems, then we are placing additional load on Jenkins. Fact. Jenkins is a build system, and can often be fairly critical to delivery. If we’re placing unplanned load on the service and bring it down, then we run the risk of hindering other users who may choose to use the tool also.

These days we take dependency management for granted. We simply specify the dependencies we want, and our build tool does the rest. I’m as guilty as the next person of not investing the authenticity of my dependencies so long as my code works. This naivety is riddled with risks to your business, or worse, anyone who chooses to use your code.

I’m developing a super awesome framework that I eventually want everyone to use. I add a few dependencies and my code works perfectly. I publish the binary on Maven Central and my code is downloaded 100million times in the first 10 minutes. Awesome!

This looks harmless on the face of it. I would say most people work in this way, right? As a simple exercise to demonstrate just how risky this is, I want you to go to Maven Central and search for the most popular Java test framework JUnit. I just did it and got 399 entries returned. Which one do I chose? am I using the actual JUnit? The second problem is that adding a dependency tells us nothing about how much care those developers were taking from a security perspective. It could be riddled with security holes. Finally, and potentially the most dangerous… I don’t know what dependencies JUnit is also pulling in. These transitive dependencies could also be victims of all the above.

Dependency Check is a tool from the OWASP team to check the dependencies you are using for know risks. The core engine contains a series of analyzers that inspect the project dependencies, collect pieces of information about the dependencies (referred to as evidence within the tool). The evidence is then used to identify the Common Platform Enumeration (CPE) for the given dependency. If a CPE is identified, a listing of associated Common Vulnerability and Exposure (CVE) entries are listed in a report.

Once you have this information available you can make a more informed decision about the dependencies you want to carry on using.

I was so impressed with this tool that I decided to work on the Dependency Check Gradle Plugin. You can find this plugin in the Gradle Plugin portal, so feel free to give it a try and provide feedback.

This tutorial was prompted by recent blog posts; Tor for technologists and How to Route Traffic through a Tor Docker container.

The idea here is that we use a docker container to run the Tor client, then using Chrome and Proxy Switch Omega we can switch between secure and non-secure browsing easily.

When adding mobile support to your application you want to quickly see what it looks like on multiple devices. While we can use a real mobile device to test things out, this process can be quite cumbersome and time consuming. It’s best to find a plugin for our browser in order to switch the user agent settings automatically.

Whilst this is by no means the only solution, I have found it to be the easiest. User-Agent Switcher is an extension for the Chrome browser which lets you easily switch a browser to a mobile device.

If you read my series titled Vagrant, Amazon EC2, Docker and Microservices, you may have got to the end and shuddered. If you haven’t read it, let me explain; I was compiling the microservice on the desktop then using Vagrant to copy the package and the Dockerfile to the cloud instance to perform the docker build. From there I performed a docker run and tested the microservice. Quite clunky and slow - especially the copy accross.

After some thought, and a better understanding of Docker I settled on the following workflow which I uploaded to www.slideshare.net.

In this slideshow you will see much more thought has gone into the process. We begin in a development environment where a developer can build and run his docker image until he/she is satisfied. Once happy they push to their version control system of choice. At this point the build machine fires into action. It should validate the code and build the image. You may run a series of tests against the container at this point. Once satisfied you would push to your Docker repository (I use Bintray, but you can use DockerHub). The image is then ready to be consumed in other environments.

I had an issue today where I was working with Jenkins and my release package was given the same name as the Jenkins job. For those of you familiar with Jenkins and the Git plugin, you will know that the workspace is given the same name as the job name, and the source is downloaded into the workspace. What you may not have known is that Gradle infers the name of the project from the root dir name.

To overcome this issue its a good idea to set up a settings.gradle file to honour your projects name. Create settings.gradle in the root directory.

This will also protect you in the open source world with Git where some users may choose to clone into a repository with a different name to what you originally set.

You may also wish to do this with your sub projects too. I find it makes working in an IDE more pleasant.

This filter picks up any items that slip through the built in filter.

Gives you a list of tickets you have beed an assignee on since the start of the month.

For this tutorial I’m going to use a spring boot application that will help us prove the concepts behind this tutorial. There are loads of microservice frameworks to chose from, but for this tutorial we will use Spring Boot. Maybe in the future I will look at trying out some other popular frameworks. Links to the Spring guide are given at the bottom of this tutorial.

At this point we could cheat and do everything in Gradle. Here’s a handy tutorial showing how to perform your build and docker run using Gradle.

When I first started writing this article I had plans of going through a 101 for Docker. Since then I have found a fantastic youtube video called Docker 101: Dockerizing Your Infrastructure and I just don’t think I can add anything to that excellent tutorial. So, rather than going from the foundations I have decided to assume you have watched that video.

Now that we can do the basics with Docker lets start setting our requirements for the container. First, we need java installed so that we can start our microservice. Then, it would be good if we could do some monitoring once the service is started. This is especially important if we plan on running this in a production setting. For that reason we’re going to move away from ubuntu images, and use an extended version of the baseimage image by Phusion. The image is called flurdy/oracle-java7 and if we look at the Dockerfile for flurdy/oracle-java7 we can see this image is using phusion/baseimage:latest and bootstrapping it with Java.

Lets start building our Dockerfile. I have created mine in ${projectDir}/app/docker:

With the FROM keyword Docker will first look for the image locally, then look to the public repo if it doesn’t find it.

MAINTAINER simply tells the reader who the author of this Dockerfile is.

EXPOSE tells Docker that a port is to be exposed when the container is started. Now, lets add our build output to the container; we do that using the ADD keyword:

The ADD instruction basically takes a <src> and <dest>. If the <dest> path doesn’t exist, it is created along with the missing directories along its path.

Finally, we need to tell Docker what command should be run when the container is executed. We do that with the CMD keyword.

And thats it!

If you look in my github repository you will find 2 helper scripts - one to build the image and the second to run it.

Lets run through the steps:

At this point we could get Ansible to do all of these steps, but as this is a learning exercise its nice to go through the motions. If you were to add these steps, then we refactored earlier, so these would sit nicely in the Docker Playbook.

This tutorial series has been a whistle stop tour through a few different tools and technologies. Whilst the example was extremely simple, hopefully you can see how this could be applied to a Continuous Delivery pipeline for on demand test environments. For the small effort up front in writing your scripts (which in this case were super simple!), you can save costs from not having "always on" environments that aren’t being used. Also, because we provisioned the environments using a coded format, we can rest assured the environments are the same every time - we also get the luxury of being able to version control our scripts as a result.

If you have any questions or would like me to clarify anything in these tutorials then please feel free to add a comment below.

Resources:

After thinking about the hack I put in to get Puppet installed on the box before I could use it, I felt a little dirty, and decided that maybe Puppet wasn’t the best decision after-all.

So, the problem is that I need an agent installed before I can provision my box, but I’m trying to automate the provisioning - catch 22. Here’s where Ansible has really stepped up.

I suppose we could have continued using the shell provisioner, but Adam Brett raises a good point on this:

"Why not just use Bash scripts, then? Ansible has an edge over Bash scripts because of its simplicity. Ansible just uses a list of tasks to run in YAML2 format. Ansible also comes with idempotency out of the box. That means you can run the same operation numerous times, and the output will remain consistent (i.e. it won’t do something twice unless you ask it to). You can write Bash scripts this way, but it requires quite a bit more overhead."

There is one small caveat with Ansible - we have to install it on the machines that will be running the Vagrant script (Ansible call it the control machine). I’ve added a link to the docs down in the resources section.

Puppet has manifests, Chef has cookbooks and Ansible has playbooks.

As we did with Puppet, lets create an Ansible dev environment:

The tasks are the same as before, and in the playbook.yml we express them in the following way:

Now we need to tell Vagrant that we want to use the Ansible provisioner. Replace the previous provisioner blocks with the following:

You can now run vagrant up --provider=aws and you should see Ansible being used for the provisioning:

Again, we can ssh into the box and prove Docker is installed:

At the moment our playbook is really simple, so you could argue its not worth refactoring, but as this is a learning exercise I think its worth going through the motions.

If you have ever imported another build using apply from: "${rootDir}/gradle/publish.gradle" then you will appreciate how its a little difficult to know exactly what has been applied to your build by said apply action.

I often use this pattern when I want to clearly seperate the parts of my build. In my build scripts you may see something like:

This is very clear and works very well, but sometimes I just don’t need to be notified of all the tasks a build file imports.

In those cases a nicer solution may be to use the GradleBuild task type as shown below.

I think this is really clear, and if you run gradle tasks you should find all other tasks from that build file ommitted.

I was just watching a PuppetLabsInc video on YouTube which had Michael Stankhe presenting "Getting Started with Puppet". Its a great presentation and really thought provoking, but the bit I like most is what he said about patterns.

"I use [the term] patterns, I dont use the words best practice because it implies I know all possible options and all your variables. I dont! Patterns are generally good ideas - sometimes they wont work for you, you may be in a situation where one of these patterns fails miserably."

In the first part of this tutorial, we showed how to use Vagrant to automate and manage an Amazon EC2 instance. We defined a simple Vagrantfile to specify certain attributes for an instance to run, and got it running using Vagrant’s command line tools. In this part of the tutorial, we’ll be using Puppet to define and automate the configuration details for our instance. This way, whenever we start up the environment with vagrant up, it will be set up to run Docker without any additional manual configuration.

The documentation for Docker is very good. Let’s use that to drive the requirements of our puppet scripts:

Installing Docker on Ubuntu Trusty 14.04 (LTS) (64-bit)

Lets start by setting up our puppet dev environment:

First thing we will want to do on our newly created instance is ensure the apt-get package database is up to date. This can be achieved with the following block:

Once thats complete we will want to install the docker package:

At this point we have a bit of a chicken and egg situation. We want to run Puppet on our box to provision it, but Puppet isn’t currently installed. We can use the shell provisioner to solve that initial problem.

Since we have placed our puppet script in the default location all we need to do is add the following line to the Vagrantfile.

config.vm.provision :puppet

And thats it!! We can test this works by running vagrant ssh followed by a sudo docker run -i -t ubuntu /bin/bash. If all is well then you should see something similar to:

Resources:

Microservices are all the rage at the moment, but from my experience they just move the bottleneck. Yes, the speed of development increases massively, but it does so at the cost of an increased dependency on the Build and Ops guys.

This blog series is about using Docker to run a complete and fully functional microservice in the cloud using Vagrant, Amazon AWS and Docker. The goals are as follows:

The idea here is if we can automate the whole process, then we will quickly see the real benefits of using a microservice based architecture.

Lets introduce the tools…

Vagrant is a nice way to manage our EC2 instances. We can use Vagrant to create and instance and provision the box to a state we desire.

Once Vagrant is installed on your dev machine, to use the AWS provider type in Vagrant we will need to install the Vagrant AWS plugin. That can be done with the following command:

vagrant plugin install vagrant-aws

So, now lets create our project and the Vagrantfile. Run the following commands:

You should get a message something similar to:

If we follow the Vagrant AWS plugin docs, we can see the basic Vagrantfile should look as follows:

The guide suggests putting your access_key_id and secret_access_key in the Vagrantfile, which is fine if you have a private repository, but as I plan on making this public I will set them using environment variables.

After configuring my Security Group, selecting an AMI and sorting my private key, my Vagrantfile now looks like:

We can now start the VM using the command:

vagrant up --provider=aws

and once the instance is available we can connect using:

vagrant ssh

finally, when we want to stop the instance we can run:

vagrant halt

This concludes part one of the tutorial. We can now create and control the lifecycle of an EC2 instance, and in part two we will install Docker and any other dependencies.

The source can be found in the repository below: https://github.com/willis7/infra-n-app-automation

Be very careful with your Amazon details on the web. I have provided a solution above for removing them from your source code. For a more in depth example see, here.Dont end up like this poor fella: My $500 Cloud Security Screwup

Dependency management has come a long way over the past 10 years, but I believe it has some way to go before we can say the problem is solved.

Consider the scenario where you have developed a library which inadvertently introduced a severe security vulnerability. Because your organisation believes in reuse it has been used in many different projects. The Maven POM (Project Object Model) does a good job in providing us with meta-data about the modules which are suppliers to a project, but it doesn’t capture information about who the consumers are.

So, we have a dangerous library in the wild, but we cant say with any certainty who is consuming it. At this point the only solution is to trawl through every projects POM and look to see if you have declared this library as a dependency. This is going to make your day very unpleasant if you have more than a hand full of projects, and if you’ve moved in the direction of microservices then this is going to be hell!

As I mentioned earlier a Maven POM provides us with a way of describing what dependencies a project has. These are identified using a standard set of attributes; groupId, artifactId and version. There are other attributes, but we will ignore them for now.

Lets turn our dependency tree into a graphical representation:

Wouldn’t it be good if we could store all of these project graphs in a single location where they could establish relationships with other projects?

We’re already talking about graphs, so wouldn’t a graph database be a good place to start?

Ensure that user account running mongod has correct permissions for the directory:

$ mongod

Note: If you get something like this:

It means that /data/db lacks required permission and ownership.

Run ls -ld /data/db/

Output should look like this (willis7 is directory owner and staff is group to which willis7 belongs):

First of all a great thanks to Mac for steering me in the direction of JBake. His blog inspired me to spend the time and implement my own version.

I’m not going to spend a long time on this because there is a plethora of information online - rather I will add tips here and links to the resources I used.

Here’s a list of tools you will need to follow along:

Use GVM to install Gradle and JBake latest. This will save you a lot of manual installation. Sorry Windows people, but you will have to install manually unless you use Cygwin and GVM.

Follow the instructions on the Git website to install Git.