Sunday, 2 October 2016

Set up a VSO Build Agent on a Docker Container

In my last blog post, I wrote about setting up .Net core on an Ubuntu 16.04 machine. In this post, I will go a step forward and explain how to set up a container to run as your build agent.

Containers are brilliant in that they provide a rather lightweight mechanism of setting up desired software in your build agent without installing it on host machine or in a virtual machine. In this post, I will explain setting up a Docker container on a Windows 10 machine, install all the desired software  for build agent and running a build to compile an core application on it.

This post is split into following areas
  • Setting up Docker on a Windows 10 machine
  • Create a Docker image to run Build Agent.
  • Configure and run a Build Agent on Docker.
  • Run a Build on the newly setup Build Agent.

Set up a Docker container on a Windows 10 machine

Docker on windows uses Hyper-V to create a linux virtual machine, The Docker daemon is run on this virtual machine.This means that you need at least Windows 10 professional to run Docker. We are using Windows 10 Enterprise. 
To install Docker visit the site and install the .MSI by clicking the "Get Docker for Windows (stable)" button. The installation will setup Docker for you, by setting up a VM called MobyLinxuVM. It would also add Docker's bin directory to your path, so that the Docker command is available in your command window.
Every docker deployment has a "hello-world" image. To test docker, type in
docker run -it hello-world
You should see the text "Hello from Docker!" in your response.

Create a Docker image to run Build Agent

Now that docker we have a Docker instance running, lets set up a Docker image with all the software needed to run a build agent. The VSO build agent is Node.js based so installing Node.js and it's pre-requisites is a requirement. Also, since the build agent is going to build .Net Core core, we would also be installing .Net core.
To start with, we get a Docker image with the latest version of Ubuntu. At the time of writing this post, the latest version available was 16.04. So we start with getting this version. To do this run
docker run Ubuntu:16.04.
It would look for the instance locally and if it's not found download it from Docker hub. Once the command is complete, the Docker Ubuntu 16.04 image will appear in your “docker images” command

We now install Node.js, npm and vso agent onto the image and store it as another image. To do that the first step is to run the Ubuntu:16.04 image as a container. To do that, run the following command.
docker run -t -i Ubuntu:16.04
You will now be on command prompt of root. To run the desire software, run the following commands one after another
apt-get update
apt-get install npm nodejs
apt-get install nodejs-legacy
npm install vsoagent-installer –g
My Ubuntu 16.04 instance didn't have https protocol installed, so installed it by running
apt-get install apt-transport--https
We will also need to install git on it
apt-get install git
Now, to install .Net Core run the following
sh -c 'echo "deb [arch=amd64] xenial main" > /etc/apt/sources.list.d/dotnetdev.list'
apt-key adv --keyserver --recv-keys 417A0893
apt-get update
apt-get install dotnet-dev-1.0.0-preview2-003131
At this stage, the container contains all the software that you need to run a vso build agent.

Configure and run Build Agent

The only thing left to do now is to configure the build agent. The agent installer allows us to easily create multiple agents on a single server. Each agent will run inside its own folder. We create a folder called MyBuildAgent1 and uses it to run the agent. To do this run the following commands
mkdir MyBuildAgent1
cd MyBuildAgent1
All commands so far has been run using the root user. We don't want to run the build agent to run as root, so will create a user called buildagentuser and switch to it to use it. Below are the command that you need to run.

adduser buildagentuser
chown -R buildagentuser /MyBuildAgent1

The build agent is almost ready to run. At this stage we want to commit this container to save the image. This way we can use it again.

Step out of Docker container by pressing CTRL+P followed by CTRL+Q. You will be back to command prompt. Now type in "docker ps" to view the list of containers running. You will see a response like following

Here, the ubuntu:16.04 running container has the Id "aea7e12541d5". We will commit this container to create a new image. To do this, type in the following on your command prompt
docker commit -a "Hamid Shahid" -m "Basic vso .net core build agent" aea7e12541d5 basicvsogent:v1
If you are following instructions, please use your container id. You can verify it by running "docker images". You will see the new image in the list of images

Since, we just "stepped out" of Docker container we were working now, we will now reattach it and run the build agent. To do this type in
docker attach aea7e12541d5
Now start the build agent by running
node agent/vsoagent
You will be prompted about your VSO url and the credentials to connect to. I used the following options in the prompt

The user you specify must have the service "Service Account" role in the agent pool you specified. In my case, I had added them to the VSO group "Agent Pool 1 Service Account Users".

Now that my build agent is now running, I will create now create a simple build definition to run the build. We had already created a simple .Net core application.

Run a build on the new build agent

We will now create a very simple build definition and execute it on the new agent. Our build definition has three simple steps.
1) In the first step, invoke dotnet with the argument "restore".
2) In the second, invoke dotnet with the argument "build". Make sure, you set the working folder to the source directory
3) In the third step, set the contents to "**/*.dll".
Since, we used the Default Build Pool, set it to use the default build agent.  Now run the build and let's monitor the build agent command window. We should see messages regarding the statue of the build job.

This is so awesome. Now that we have the Docker image captured as well, we can start other build agents and distribute our build load across containers. This is far more efficient than running virtual machines as build agents.
The above would only work for dot net core applications. In my next post, I will write about how to set up VSO agents on Windows Containers, where we would also have the ability to build .Net applications.

Friday, 27 May 2016

Installing .Net Core on Ubuntu 16.04 LTS

The recent development by Microsoft in the ASP.Net core space is fascinating. With the introduction of ASP.Net Core 1.0 at the start of the year and then the recently released ASP.Net Core 1.0 RC2, I thought it would be a good idea to try it out on a Linux box.

The installation proved to be a bit trickier than I thought, so sharing my experience so as to help other out. The flavour of Linux I used was Ubuntu 16.04 LTS.

At the time of writing this post, the instructions present on Microsoft .Net Core website are for Ubuntu 14.04. Tried to follow the steps described on the website. However, execute dotnet failed with the following error on my machine
Failed to initialize CoreCLR, HRESULT: 0x80131500

So tried to proceed with some other steps. In general, installing .Net on Ubuntu require the following steps

1) Add .net repo to trusty sources list
2) Add key for the newly added trusted source
3) Install dotnet

To do the above, open up a terminal on your Ubuntu machine.

To add repo, run the following command
sudo sh -c 'echo "deb [arch=amd64] trusty main" > /etc/apt/sources.list.d/dotnetdev.list'
To add the key, run the following command
sudo apt-key adv --keyserver --recv-keys 417A0893
Once, the above done, it's bes to update everything by running
sudo apt-get update
 Now that everything is done, .Net can be installed by running
sudo apt-get install dotnet

The above command, however, didn't work for me. It failed with the following error

The following packages have unmet dependencies: dotnet : Depends: libicu52 (>= 52~m1-1~) but it is not installable E: Unable to correct problems, you have held broken packages.

The error is quite self-descriptive, and the answer is to install libicu package. Ran the commnad
sudo apt-get install libicu-dev

The above command installed the libicu55 on my machine, whilst .Net core has a dependency on libicu52. Fortunately, the package is available for download here.

Once downloaded, the package can be installed by running
sudo dpkg -i libicu52_52.1-3ubuntu0.4_amd64.deb 

Now that the pre-requisite is installed, installing dotnet is simply a matter of running the following
sudo apt-get install dotnet

The above worked for me this time around. To test that it installed correctly, just type dotnet new in a new folder. It will create files for your .net project.

Please Note: Make sure that you set permissions to execute downloaded files described in this post. 

Saturday, 3 October 2015

Shelveset Comparer now supports Visual Studio 2015

Last year, I created a visual studio extension that allow users to compare two shelvesets. The extension has proved to be quite popular. 

The visual studio 2015 version of the extension is out today and can be download from here

Please free free to use it and give your feedback.

Monday, 2 March 2015

“No tracking information on current branch” while doing git pull

Git continues to amuse me. There sheer options and versatility that you get in creating branches, merges and tracking commits, make it a such a powerful source code repository. It also means a steep learning curve for a lifelong TFSVC and VSS .

This is the first of a series of posts, I will write about some common start up mistakes I have done or seen others doing. I hope it would be helpful for the wider community.

To demonstrate my git repository clearly, I am using posh-git, which gives a view of my current git branch and also shows local, active and remote branches in different colours

Created branch but failed to created tracking information.

This is one of the most common mistakes. To describe it, I start with listing all my local and remote branch. To do that I type in “git branch –a” as shown


As you can see with the asterisk against master that my current local repository is “master”.

Now lets say that I want to work on the remote “TeamBuild/Staging” branch . To do that I created a new local branch naming it same as the server


The new branch is created successfully and I am switched on to the new branch as shown by running another “git branch –a”


You can see that the current branch is now switched to “teambuild/staging”. All good so far.

However, when I do a “git pull”, I got the following message.


So, what has gone wrong. A quick peek into the Git config file, tells me that the new branch is created without tracking it to the remote branch with the same name.


As you can see that there is no mention of the newly created branch.


The reason no tracking information was created is because Git is case sensitive. So, the branch“teambuild/staging” is not same as “TeamBuild/Staging”. For windows / TFS users, they look the same but Git would treat them as two separate branches.


You can add tracking information in a branch using the “git branch –u” command. So typing the following will add tracking information with the remote branch and you will be be up and running


You can see that the tracking information is added by peeking into the git config file. Notice the couple of lines in the end to see details of the /staging/teambuild branch added to git config.


Don’t try it at home

You will git into a bit of a pickle if are using Windows and you create two local branches that differentiate just by casing.  Since git creates a folder within the refs/head folder in the git directory. If the names vary only by casing, it would have two branches tracked on one folder which would confuse it. So be mindful with casing while working with Git branches.

Monday, 17 November 2014

Shelveset Comparer Updated

Shelveset Extension is a visual studio extension that I first published at the start of this year. The extension provides a functionality that is otherwise missing in both Visual Studio and Team Foundation Server that is to compare the contents of two shelvesets. I felt that need for it as our team used shelvesets to pass work around and tracking what has changed since the time a shelveset was taken was not always obvious.

There extension has proved popular I have been trying to keep up with comments and feedback on it. This update was due for some time. The view of the extension in team explorer has changed a bit to show options for typing in two users. This allows for comparing shelvesets between two users. However, unlike the first release of the extension, there is still one list to display all users. To separate out shelvesets of two users, an “Owner” column has been added. The column headers are made clickable as well and will sort the rows based on the clicked column.


Another feature added is the Options panel, allowing users to select whether they want to view the extension as a Team Explorer button or not. Another option is to hide the second user.



The options are there to allow users to customise the view as per their needs.


Apart from the new functionality, several fixes and performance improvements have been made


Going forward, there is going to be another release by the end of this year, where I will be adding feature to search on a shelveset name. There will be further optimisation in the performance when comparing the contents of two shelvesets.

Thursday, 15 May 2014

Feature Toggler – a Simple feature toggle library for .Net

So, you have decided to use Feature Toggling as your branching strategy. You don’t want the hassle of merging and branching and are confident that developers and testers can handle the additional complexity that comes with Feature Toggles. The next step is to decided how to go about using toggles. The simplest and most popular method of doing is to have feature toggles set in configuration files

Ideally, you would want a library that would take care of feature toggling. All you would need to do is to define the features and their toggle value in the configuration file and be able to check if a feature is available with a simple check. Some thing which for a configuration like below

    <add name="PrivateProfiles" toggle="on" />
    <add name="Photosharing" toggle="off" />
    <add name="Videos" toggle="1" />
    <add name="bookmarks" toggle="true" />

would allow having code like following

if (FeatureManager.HasFeature("PrivateProfiles")){


Having looked around, there were three libraries of note already available, which were

  1. NFeature
  2. FeatureToggle, and
  3. FeatureSwticher

This blog post gives a good comparison of them and their usability. Having used all three, I felt that all of them, though thorough, were overly complicated for the very simple scenario that I wanted to use. For example, NFeature requires you to create enumerations for all features added in configuration file.

I decided to create a new very simple feature toggling library.

The library is available as NuGet. Simply type “Install-package FeatureToggler” in the package manager window of your application. It will add references, add a configuration section in your configuration files and adds a few sample features in your configuration file.

Once you have the reference added, simply add features in the features collection and use them in your code using the FeatureManager.HasFeature(“”) method. Happy Coding!!

Technorati Tags: ,,

Thursday, 8 May 2014

Feature Toggles and their limitations


This month's MSDN magazine contains an article on feature toggles. The subject has been close to my heart in the last few weeks and I have been weighing up whether they would work for our projects or not .

For those, who are unaware of the term, here is a good post by Martin Fowler describing feature toggles and their merits. He is convinced that feature toggles is the way forward and should be used instead of feature branches. Here is another great blog post explains the differences and recommends to use Feature toggles.

I love the idea of having no features branches … makes life easier. However, my take is that feature toggles is not for everyone and every team. For someone like Plural soft who does continuous delivery (and they use feature toggles), the process is simple. Each release, results in some new "features" being added. The process is generally additive with software becoming more "feature rich" and there is control on the release pipeline.

Now, turn our attention to a simple "message broker" kind application that interface with multiple systems and has no UI. The application handles message say M1 from one applications, does something to it and pass on message M2 to another application. Now, let's say there is a change in message interface because the sending application is changing. We start with a feature to handle the new message interface. Since, the change is a few months away, we need to keep supporting the existing interface. In this case, if feature toggling is involved, we would have to create a parallel code path to handle the new message interface and direct to that code path with feature toggle. If it wasn't the case and we were using branching instead, the change in code would have been much simpler. So in essence, we have replaced the complexity of merging by having a more complex code change.

Take another example, this time we have to delete something from the application, let's say a web service from the system. The feature toggle mechanism would require us to modify it to error on invocation when the feature is on. Compare it with the alternative of removing the service altogether.

Similarly, let's consider a windows/web UI application. One of the features is to re-design of the screens. The redesign includes jigging around all the form controls and include some new graphics. With feature toggling approach, we will either have a condition on display of each of these changes or have a new form created altogether, choosing between the two based on toggle value.

These were only some of the scenarios where feature toggle wouldn't essentially simplify things in my opinion. Others might disagree and I would love to listen to them, so please post your comments if you have any.


Technorati Tags: