Running aspnet/JavaScriptServices on Docker

The aspnet/JavaScriptServices project is part of ASP.NET Core and allows you to use Modern Web App frameworks such as Angular, React, etc. for creating web user interfaces of your application, alongside using ASP.NET Core for your web services. If you are using VueJS, the project aspnetcore-Vue-starter does the same thing.

Since, it uses both node.js and .net core, any server hosting the website would need to have both the technologies installed on it. In addition to that, your application might be using node modules and nuget packages, which would need to be deployed on to the server. 

Creating Docker Image

When deploying your application in a docker container, typically you start with a base image. For instance, when deploying your .Net core application, your docker file would start with an instruction like 

FROM microsoft/aspnetcore

But if you are creating a node base application, your docker would start with an instruction like 

FROM node




So, what images should be used for the projects that use both node.js and .net core. The answer is to take one image and install the other library on it. We start with looking at the size of the image. The microsoft/aspnetcore is 345 MB and node:latest is 674MB. 

Both are pretty huge right? So, the first step is to find the slimmer versions of the containers which can be used for production. I found out that there is an alpine image of node, which is about 70 MB that we can use as a baseline and installed .net core on top of it.

We could do that but installing .net core on ubuntu takes far too many commands as compared to installing node.js on ubuntu. So an alternative would be to find an alpine image for dotnet core and install node on it. So, I look and yes there is certainly an alpine image for dotnet core and at about 162MB in size is just right for us.

This would do the trick. So our dockerfile to deploy node on top of dotnet would look like this 

FROM microsoft/dotnet:2.1.1-aspnetcore-runtime-alpine3.7 as baseimage

RUN apk add --update nodejs nodejs-npm

I actually added multi-stage build process in our docker file so that the compilation the application is compiled in it as well and a image produced at the end of the compilation process. The entire Dockerfile for the project produced by aspnetcore-Vue-starter project template looks follows

# Stage 1 - Restoring & Compiling

FROM microsoft/dotnet:2.1-sdk-alpine3.7 as builder

WORKDIR /source

RUN apk add --update nodejs nodejs-npm

COPY *.csproj .

RUN dotnet restore

COPY package.json .

RUN npm install

COPY . .

RUN dotnet publish -c Release -o /app/

# Stage 2 - Creating Image for compiled app

FROM microsoft/dotnet:2.1.1-aspnetcore-runtime-alpine3.7 as baseimage

RUN apk add --update nodejs nodejs-npm

WORKDIR /app

COPY --from=builder /app .

ENV ASPNETCORE_URLS=http://+:$port

# Run the application. REPLACE the name of dll with the name of the dll produced by your application

EXPOSE $port

CMD ["dotnet", "vue2spa.dll"]

PS: In case, you want to use a base image with .net core 2.1 and node already on it, you can use the base image https://hub.docker.com/r/hamidshahid/aspnetcore-plus-node/. Simply pull it by calling 

docker pull hamidshahid/aspnetcore-plus-node

Migrating ASP.NET MVC website to ASP .NET Core

I maintain an ASP.NET MVC website that I have been meaning to move to ASP.NET Core, but found the .Net Core 1.1 library rather limited. With the release of .NET Core 2.0 and ASP.NET Core 2.0, we decided to migrate the websites to the new framework. The site has been operational since October 2010 and was built using ASP.NET MVC 2.0. It has gone through various bouts of upgrades and is currently using ASP.NET MVC 5.2.0, which forms the baseline of this conversion. I had several discoveries along the way so thought to blog about them. 

In this post, I am going to write about prep and moving our "Model" to Entity Framework .Core 2.0. 

Background

The model of our website was built using Entity Framework code first. All database operations were performed using repository pattern. Our repository interface looks as follows

    public interface IRepository : IDisposable where TEntity : class

    {

        IQueryable GetQuery();

        IEnumerable GetAll();

        IEnumerable Find(Expression> predicate);

        TEntity Single(Expression> predicate);

        TEntity First(Expression> predicate);

        void Add(TEntity entity);

        void Delete(TEntity entity);

        void Attach(TEntity entity);

        void SaveChanges();

        DbContext DataContext { get; }

    }

We use interface inheritance to create repository for each of our model objects, so for a object "Token", the repository looks like following

    public interface ITokenRepository : IRepository

    {

    }

With the interface inheritance in place. our single generic repository class can the logic for database operations as shown below

public class Repository : IRepository where TEntity : class

    {

        private DbContext _context;

        private IDbSet _dbSet;

        private static string _connectionString = string.Empty;

        public Repository(IDataContextFactory dbContextFactory)

        {

            if (string.IsNullOrWhiteSpace(dbContextFactory.ConnectionString))

            {

                _context = dbContextFactory.Create(ConnectionString);

            }

            else

            {

                _context = dbContextFactory.Create();

            }

            

            _dbSet = _context.Set();

        }

        public Repository(DbContext context)

        {

            _context = context;

            _dbSet = _context.Set();

        }

        public DbContext DataContext

        {

            get

            {

                return _context;

            }

        }

        public IQueryable GetQuery()

        {

            return _dbSet;

        }

        public IEnumerable GetAll()

        {

            return GetQuery().AsEnumerable();

        }

        public IEnumerable Find(Expression> predicate)

        {

            return _dbSet.Where(predicate);

        }

        public TEntity Single(Expression> predicate)

        {

            return _dbSet.SingleOrDefault(predicate);

        }

        public TEntity First(Expression> predicate)

        {

            return _dbSet.FirstOrDefault(predicate);

        }

        public void Delete(TEntity entity)

        {

            if (entity == null)

            {

                throw new ArgumentNullException("entity");

            }

            _dbSet.Remove(entity);

        }

        public void Add(TEntity entity)

        {

            if (entity == null)

            {

                throw new ArgumentNullException("entity");

            }

            _dbSet.Add(entity);

        }

        public void Attach(TEntity entity)

        {

            _dbSet.Attach(entity);

        }

        public void SaveChanges()

        {

            _context.SaveChanges();

        }

        public void Dispose()

        {

            Dispose(true);

            GC.SuppressFinalize(this);

        }

        protected virtual void Dispose(bool disposing)

        {

            if (disposing)

            {

                if (_context != null)

                {

                    _context.Dispose();

                    _context = null;

                }

            }

        }

        public static string ConnectionString 

        {

            get

            {

                if (string.IsNullOrWhiteSpace(_connectionString))

                {

                    _connectionString = ConfigurationManager.ConnectionStrings["Rewards"].ConnectionString;

                }

                return _connectionString;

            }

        }

    }

The class above does all the heavy lifting for us. We just need to define classes that implement each of our models' repository interface. For our model Token, it would be

public class TokenRepository : Repository , ITokenRepository

    {

        public TokenRepository(IDataContextFactory dbContextFactory)

            : base(dbContextFactory)

        {   

        }

        public TokenRepository(DbContext dataContext) 

            : base(dataContext)

        {

        }

    }

Entity Framework Core 2.0 limitations

1. No Many-To-Many Relationship

The biggest issue we have encountered while migrating to .Net Core 2.0 is lack of resolution for Many-To-Many relationships. This is an open issue, which haven't been resolved yet. For us, it means a lot of re-work.

With the POCO way of working, you would start with writing your domain model and your write your business logic using models, without really thinking about relational database details. We have a lot of code where our LINQ queries were based on domain model relationships. Now, we need to re-work all those. 

This, in my mind, is a major issue and though ways to resolve this issue, it prevents Entity Framework .Core from being a true ORM tool. 

As an example, consider you have two entities Parent and Student in your model, where a student can have multiple parents and a parent can have multiple students. With Entity Framework 6, the model definition was sufficient to imply the correct type of relationship. If you have to do it explicitly, you could do it at the time of model creation like below

modelBuilder.Entity()

              .HasMany(p => p.Parents)

              .WithMany(r => r.Students)

              .Map(m =>

              {

                  m.ToTable("ParentStudents");

                  m.MapLeftKey("Student_ID");

                  m.MapRightKey("Parent_ID");

              });

You can then go on and work with defining a collection of Parents in Students class and a collection of Students in Parent class. The .WithMany() method  is not there in Entity Framework Core.

The lack of Many-To-Many feature in EF Core is hard to justify. POCO came out as a good model for domain driven development and not supporting many-to-many in a domain driven world is hard to justify. We didn't want to "dilute" the model with resolving entities, so we decided to "implement" the many-to-many resolution in our code. This series of post describes a good way of keeping domain relationship in our objects, so that there is no change in business logic in other parts of the application.


2. IDbSet Interface 

The IDbSet interface was removed in Entity Framework 6.0, because the team were looking to add new operations to it without defining a new set of interfaces. This is pretty well documented in EF 6.0 design decisions. I do not agree to this decision as it breaks the whole promise of interface as immutable being. The EF team wanted to avoid creating interfaces like IDBSet2, etc for more functions they decided to do away with it. However, the interface is still present in the in EntifyFramework 6.0 library, so our code still worked. Now we had to replace any use of IDbSet with the DBSet class. Also, meant our test code had to be re-written as we mocked IDbSet to for results from database.


3. No Lazy Loading

The entity framework does not support lazy loading as of yet. There is an open issue for it on github.  The feature request is in the backlog of EF team but there is no date of adding it yet. Lazy loading is the default behaviour of Entity Framework and would be there for you if you have the navigation property defined as virtual. This is another big way in which Entity Framework core breaks backward compatibility. 

The way around is to "Eager Loading" i.e. ensure that you use the .Include("") and .ThenInclude("") method in all places, where you are relying on Lazy loading. This is no simple as it's easy to miss it out at placed and the error is only manifested at run time. One way of go about doing it, is to find references of all virtual properties and add .Include("") where the object is "hydrated".


4. No GroupBy Translation

Entity Framework Core 2.0 doesn't support translate Group By to SQL. So, if your application is using GroupBy() method, you might need to take a look for alternatives. Fortunately, more support for Group By is getting added in EF Core 2.1.

The only way to resolve this issue without punitive performance impact is to move the logic to stored procedures. We were using GroupBy mostly in our reports, which were already a candidate to use stored procedures. So, although there was some work involved but the result was much better performance.


Final Words...

My experiences with migrating code from Entity Framework 6.0 to Entity Framework Core 2.0 would not have uncovered all pertaining issues in migration process but this post might help out someone who is looking to take the plunge. 

In my view, Entity Framework Core 2.0 is still a bit under cooked but if you are willing to take do the extra effort, it has enough functionality for you to move your model / data libraries to it.

Set up a VSTS Build Agent on a Docker Container

In my last blog post, I wrote about setting up .Net core on an Ubuntu 16.04 machine. In this post, I will go a step forward and explain how to set up a container to run as your build agent.

Containers are brilliant in that they provide a rather lightweight mechanism of setting up desired software in your build agent without installing it on host machine or in a virtual machine. In this post, I will explain setting up a Docker container on a Windows 10 machine, install all the desired software  for build agent and running a build to compile an ASP.net core application on it.

This post is split into following areas

• Setting up Docker on a Windows 10 machine
• Create a Docker image to run Build Agent.
• Configure and run a Build Agent on Docker.
• Run a Build on the newly setup Build Agent.

Set up a Docker container on a Windows 10 machine

Docker on windows uses Hyper-V to create a linux virtual machine, The Docker daemon is run on this virtual machine.This means that you need at least Windows 10 professional to run Docker. We are using Windows 10 Enterprise. 

To install Docker visit the site https://docs.docker.com/docker-for-windows/ and install the .MSI by clicking the "Get Docker for Windows (stable)" button. The installation will setup Docker for you, by setting up a VM called MobyLinxuVM. It would also add Docker's bin directory to your path, so that the Docker command is available in your command window.

Every docker deployment has a "hello-world" image. To test docker, type in

docker run -it hello-world

You should see the text "Hello from Docker!" in your response.

Create a Docker image to run Build Agent

Now that docker we have a Docker instance running, lets set up a Docker image with all the software needed to run a build agent. The VSO build agent is Node.js based so installing Node.js and it's pre-requisites is a requirement. Also, since the build agent is going to build .Net Core core, we would also be installing .Net core.

To start with, we get a Docker image with the latest version of Ubuntu. At the time of writing this post, the latest version available was 16.04. So we start with getting this version. To do this run

docker run Ubuntu:16.04.

It would look for the instance locally and if it's not found download it from Docker hub. Once the command is complete, the Docker Ubuntu 16.04 image will appear in your “docker images” command

We now install Node.js, npm and vso agent onto the image and store it as another image. To do that the first step is to run the Ubuntu:16.04 image as a container. To do that, run the following command.

docker run -t -i Ubuntu:16.04

You will now be on command prompt of root. To run the desire software, run the following commands one after another

apt-get update
apt-get install npm nodejs
apt-get install nodejs-legacy
npm install vsoagent-installer –g

My Ubuntu 16.04 instance didn't have https protocol installed, so installed it by running

apt-get install apt-transport--https

We will also need to install git on it

apt-get install git

Now, to install .Net Core run the following

sh -c 'echo "deb [arch=amd64] https://apt-mo.trafficmanager.net/repos/dotnet-release/ xenial main" > /etc/apt/sources.list.d/dotnetdev.list'

apt-key adv --keyserver apt-mo.trafficmanager.net --recv-keys 417A0893
apt-get update

apt-get install dotnet-dev-1.0.0-preview2-003131

At this stage, the container contains all the software that you need to run a vso build agent.

Configure and run Build Agent

The only thing left to do now is to configure the build agent. The agent installer allows us to easily create multiple agents on a single server. Each agent will run inside its own folder. We create a folder called MyBuildAgent1 and uses it to run the agent. To do this run the following commands

mkdir MyBuildAgent1
cd MyBuildAgent1
vsoagent-installer

All commands so far has been run using the root user. We don't want to run the build agent to run as root, so will create a user called buildagentuser and switch to it to use it. Below are the command that you need to run.

adduser buildagentuser
chown -R buildagentuser /MyBuildAgent1

The build agent is almost ready to run. At this stage we want to commit this container to save the image. This way we can use it again.

Step out of Docker container by pressing CTRL+P followed by CTRL+Q. You will be back to command prompt. Now type in "docker ps" to view the list of containers running. You will see a response like following




Here, the ubuntu:16.04 running container has the Id "aea7e12541d5". We will commit this container to create a new image. To do this, type in the following on your command prompt

docker commit -a "Hamid Shahid" -m "Basic vso .net core build agent" aea7e12541d5 basicvsogent:v1

If you are following instructions, please use your container id. You can verify it by running "docker images". You will see the new image in the list of images



Since, we just "stepped out" of Docker container we were working now, we will now reattach it and run the build agent. To do this type in

docker attach aea7e12541d5

Now start the build agent by running

node agent/vsoagent

You will be prompted about your VSTS url and the credentials to connect to. I used the following options in the prompt



The user you specify must have the service "Service Account" role in the agent pool you specified. In my case, I had added them to the VSTS group "Agent Pool 1 Service Account Users".


.
Now that my build agent is now running, I will create now create a simple build definition to run the build. We had already created a simple .Net core Asp.net application.

Run a build on the new build agent

We will now create a very simple build definition and execute it on the new agent. Our build definition has three simple steps.

1) In the first step, invoke dotnet with the argument "restore".

2) In the second, invoke dotnet with the argument "build". Make sure, you set the working folder to the source directory

3) In the third step, set the contents to "**/*.dll".

Since, we used the Default Build Pool, set it to use the default build agent.  Now run the build and let's monitor the build agent command window. We should see messages regarding the statue of the build job.

This is so awesome. Now that we have the Docker image captured as well, we can start other build agents and distribute our build load across containers. This is far more efficient than running virtual machines as build agents.

The above would only work for dot net core applications. In my next post, I will write about how to set up VSTS agents on Windows Containers, where we would also have the ability to build .Net applications.

Installing .Net Core on Ubuntu 16.04 LTS

The recent development by Microsoft in the ASP.Net core space is fascinating. With the introduction of ASP.Net Core 1.0 at the start of the year and then the recently released ASP.Net Core 1.0 RC2, I thought it would be a good idea to try it out on a Linux box.

The installation proved to be a bit trickier than I thought, so sharing my experience so as to help other out. The flavour of Linux I used was Ubuntu 16.04 LTS.

At the time of writing this post, the instructions present on Microsoft .Net Core website are for Ubuntu 14.04. Tried to follow the steps described on the website. However, execute dotnet failed with the following error on my machine

Failed to initialize CoreCLR, HRESULT: 0x80131500

So tried to proceed with some other steps. In general, installing .Net on Ubuntu require the following steps

1) Add .net repo to trusty sources list
2) Add key for the newly added trusted source
3) Install dotnet


To do the above, open up a terminal on your Ubuntu machine.

To add repo, run the following command

sudo sh -c 'echo "deb [arch=amd64] http://apt-mo.trafficmanager.net/repos/dotnet/ trusty main" > /etc/apt/sources.list.d/dotnetdev.list'

To add the key, run the following command

sudo apt-key adv --keyserver apt-mo.trafficmanager.net --recv-keys 417A0893

Once, the above done, it's bes to update everything by running

sudo apt-get update

 Now that everything is done, .Net can be installed by running

sudo apt-get install dotnet

The above command, however, didn't work for me. It failed with the following error

The following packages have unmet dependencies: dotnet : Depends: libicu52 (>= 52~m1-1~) but it is not installable E: Unable to correct problems, you have held broken packages.

The error is quite self-descriptive, and the answer is to install libicu package. Ran the commnad

sudo apt-get install libicu-dev

The above command installed the libicu55 on my machine, whilst .Net core has a dependency on libicu52. Fortunately, the package is available for download here.

Once downloaded, the package can be installed by running

sudo dpkg -i libicu52_52.1-3ubuntu0.4_amd64.deb 

Now that the pre-requisite is installed, installing dotnet is simply a matter of running the following

sudo apt-get install dotnet

The above worked for me this time around. To test that it installed correctly, just type dotnet new in a new folder. It will create files for your .net project.

Please Note: Make sure that you set permissions to execute downloaded files described in this post.