PART 2.1: WEB API PIPELINE – Building a Scalable App Environment with Infrastructure and Deployment

Using .NET, Angular, Kubernetes, Azure/Devops, Terraform, Eventhubs and other Azure resources.

image by author

This is one part of a series. So if you have not read the PART 0: OVERVIEW you can go there and read it to get an overview of what we will actually doing here …

Introduction

In the last PART 2: WEB API we have created our simple web api in .net core which provides a simple api for getting all messages or posting single messages. The sent messages endpoint will be sent the message to an event hub and from there it can be consumed by our workers ( which we build in the next parts!).

Now we will create a build and release pipeline (CI/CD) for our web api. The build pipeline will build the artifacts and upload the container in our azure registry.

We will create a release pipeline with two stages (acc and prd). The release pipeline will use our keyvaults and apply the secrets as environments vars to the app. We will write k8s deployment, service and ingress files to deploy the web api in our aks.

You can download the code from the git repository.

Deployment Files

Lets start with writing the needed deployment files…

Docker

Start by creating a file named “Dockerfile” in the “./.deploy” folder inside the webapi repository. Put then the following code into it.

This dockerfile is quite simple. It copies the required sources (WebApi and Common) and restore, build and publish the application. After then we define the container’s entry point. But one thing is something which could be important! I have been recognized that when I am using the default rendered dockerfile from visual studio it will not work. It is because the “runtime:3.1-buster-slim” will be used for runtime, but when running in aks it has to be the full sdk. If not then you will get an error on the aks like: “It was not possible to find any installed .NET Core SDKs”.

Kubernetes

To deploy our web api to our aks we need some deployment files. First we create a folder inside the “.deploy” folder and name it “k8s”.

We start by creating a file inside the k8s folder and name it “config.yaml”. Then put the following code into it.

We use this config file to push non secret environment variables to the application. At this time we only have one entry here for the “CORS”. Keep in mind that our web api replaces automatically values in “appsettings” with values from these environment variables.

Now we do the same with the secrets… Create a file named “secrets.yaml” besides the config.yaml. Then put the following code into it.

So here it is the same as with secrets. This variables will be pushed to the application as secrets. We have here the appinsights key, the storage connection string and the event hub connection string. All this secrets will be set inside the release pipeline later with the values from the the azure keyvault.

Now we came to the deployment file for the web api. Please create a file “deployment.yaml” and put the following code into it.

At least this deployment file pull the docker image from the azure container registry and pass our config and secret environment variables to the created pod. The values for the “ENVIRONMENT” and the “RELEASE_ARTIFACTS_BUILD_NOTIFIER_BUILDID” variables came from the release pipeline.

Then we need a k8s service, which let us connect to the application. To do that create a file and name it “service.yaml” and put the following…

We provide a service to connect to the pod over the port 8080. Now we have to provide an ingress which represents the routes for the web api in the aks. …create a file “ingress.yaml” and put the following…

This ingress will receive traffic from our public id which was created in the infrastructure part by creating the helm ingress controller. We can define here a lot, but to keep this simple, we only have one rule which routes all the traffic to our web api application on port 8080.

All this k8s files will be applied by the release pipeline, which we will see later in this post…

Build Pipeline

First we create the build pipeline and initialize terraform for our workspaces and validate them. Furthermore we create the artifact with the terraform files to later create the resources in the release pipeline.

Let’s got to azure devops now and navigate to your notifier project pipelines. The press “create pipeline” and choose Azure Repos Git for your source. Select the “Infrastructure” repository. Choose then “Starter pipeline” and click “Save and run” und yes commit it directly into the master branch. After the job has been successfully finished. Check out the repository changes from the origin. After that you should see “azure-pipelines.yml” file und your “Infrastructure” folder. From here we start to add our needed build stuff. So open the file in your editor and let’s go…

First thing to do is delete all stuff in there and then copy the following.

The trigger sets a branch where the pipeline will be automatically triggered. The repository source is to include the repositories which needed in the build. This is in our case only “self” (the repository which triggered the build). And the pool where we define our vmImage. We set it to latest ubuntu.

If we have this done, we can go to the steps. First delete all the code inside the steps section. (I have moved the pipeline file into a “.deploy” folder – if you want to do that to, you have to change the path from the yaml in the pipeline!) Lets start by setting the base configuration of the pipeline by putting the following code into the empty yaml file.

Here we set the trigger, resources, the pool and some variables. Pay attention to the resource that we include here the needed common lib repository. In the last part I describe the common lib. It is a lib which is used by web api and the workers… So it is an extra repository which you also need to create in you azure devops (if you do not have already done this!) You can download the code from the common lib here.

Now we come to the steps and tasks section… Put the following code directly under the variables section.

It begins with checking out the relevant repositories (self and common lib). In the first task we build the docker image for acc environment by our created docker file.

In the second task we push image to our container registry. The next two steps do the same for prod. You might be thinking why do we have to do the same for different environments… And yes this is only because we wanted to have every resource for every environment. But yes it is also a good idea to share some resources like the container registry. But the other has the advantage that we have real independent resources for every stage…

The next task is to copy our k8s files to the artifacts directory, so it can be uploaded in the last step as an artifact.

When you push your next changes in master branch the pipeline will be automatically triggered and should be run successfully.

Release Pipeline

For this release pipeline an hopefully already configured Azure ARM Service Connection is needed. We used this already in PART 1.1 Infrastructure Pipeline. I will here not go so much in detail of how creating the stages, adding artifacts and adding pre deployment conditions. (How to do this in detail is described in PART 1.1 Infrastructure Pipeline and can be easily adopted.)

Now let’s go to azure devops and create a release pipeline (azuredevops -> pipelines -> releases -> new) and name it “WebApi Release”. Then go to the pipeline in edit mode and start first by adding the web api artifact. At the end we want to have two stages (“acc” and “prd”). Set pre deployment conditions from “acc” to “After Release” and from “prd” to “After Stage -> acc”

Before we adding release tasks we go to the “Variables” tab and set some pipeline variables.

NameValueScope
DOMAIN_URL_BASE_NAMEnotifier.comRelease
DOMAIN_URL_SUFFIX-accacc
DOMAIN_URL_SUFFIX-prdprd
ENVIRONMENTaccacc
ENVIRONMENTprdprd
RELEASE_ARTIFACTS_BUILD_NOTIFIER_BUILDID$(Build.BuildId)Release
ApplicationInsights__InstrumentationKey$(ApplicationInsights–InstrumentationKey)Release
EventHubSettings__ConnectionString$(EventHubSettings–ConnectionString)Release
StorageSettings__ConnectionString$(StorageSettings–ConnectionString)Release

The values for app insights, event hubs and connection string comes from our key vault, which we provide through a task in the next step.

Now we start adding tasks for the release. Please go to the “Tasks” tab and select the “acc” stage. And start adding the first task “Azure Key Vault”. Set ARM Service connection and select the web api key vault. This tasks makes it possible to use the azure key vault inside the pipeline. The real cool thing here is that we have no password here it is all hidden in the key vault and the key vault access is via the service connection.

image by author

The second task is for replacing the variables in our kubernetes deployment files by the ones from our release pipeline. We choose here the “Replace Tokens” task. Set here the kubernetes folder as root directory. The default values for a replacement match is “startwith=#{” and “endswith=}#”. So we can leave it by default because

The last task is for deploying our web api app to aks! For this we need a further service connection. So go to “Project Settings” and select “Service Connections”. Then click create new service connection from type “Kubernetes” and select your azure subscription and wait for the login window. After then you should see your clusters. Select the acc cluster and give it a name “AKS Notifier ACC”. When you have done this you can go back to your pipeline release tasks and add the last task “Kubectl”.

image by author

Set here the Kubernetes Service Connection (in my case AKS Notifier – but in your case AKS Notifier ACC – because we have for every stage a k8s cluster). Then select the “apply” command set the path where the kubernetes deployment files are.

Then you should add the same tasks for the “prd” stage which I will not repeat here…

Verify

So ok if we now release the web api and the pipeline runs successfully then we could check if everything is working. To do that please make first a connection to the acc aks…

az aks get-credentials --resource-group notifier-resource-group-acc --name notifier-aks-acc

and then get the current services…

kubectl get services

You should now see the ingress controller which was created by infrastructure. This one should have an external ip. Then you should also see the webapi service connection running on port 8080.

Then you could check the pods…

kubectl get pods

When the pod is running, then all is good. I f you get an image loop backoff then please run the following command to get more information.

kubectl describe pod POD_NAME

I could guess that in our case it could be that the aks has no authorization to the container registry. When this is then please allow it by typing the following and create a new release.

az aks update -n notifier-aks-acc -g notifier-resource-group-acc --attach-acr notifiercontainerregistryacc

If all work up to here please call the ping (GET) endpoint (which does not uses other resouces like event hubs or something else).

GET http://YOUR_PUBLIC_IP/api/notifications/ping

If you get back “PING” then your ingress routing to your web api is working. Then go a step further lets try to get all notifications.

GET http://YOUR_PUBLIC_IP/api/notifications

When you get now an empty array. We are sure that our connection to our azurestorage is working. If this is not working then probably your connection string is wrong or was not set/replaced correctly in the pipeline.

And by posting to the endpoint a message should be sent to event hubs and the message should be saved in the azure table.

POST http://localhost:8080/api/notifications?message=hello this is my message

Conclusion

We created here a CI/CD pipeline for our web api, which is now running on an aks. The web api can be reached from outside the cluster and can be used for getting and sending notifications.

Preview

In the next chapter we are going to create the Notifier Workers (app insights, email) which can consume our notifications sent from the web api over the event hub.

Why I completely switched to VS Code

…also for C#/.NET development…

image by author

Foreword

First I have to say that I like Visual Studio very much. It is a good working IDE for development .NET stuff. I used it for over ten years now for mobile, desktop and web development. Yes of course sometimes visual studio can a bit annoying because of it’s performance and “magic” behind the scenes. But at least it is a good Program…

However some weeks ago I needed to reinstall windows and began installing all the required programs which I needed. Then I stopped after installing VS Code and said to myself: “Hm my msdn professional abo has been ended and the next Visual Studio Version (2020) will cost me some money. Let’s try rider? Hm no – before I want give VS Code a real try!” This is because I use VS Code a lot for all my other developments like angular, go, flutter, etc. and I know this should work with .NET.

So my first expectation was that I get some syntax coloring and all the compiling, tests, etc. will be running in terminal, or have to be done in a not so common way.

My Experience

So I knew that I can use also VS Code, but I always thought, that I have to dispense a lot of features. Most of the devs I know work with Rider or Visual Studio Professional/Enterprise, because they thought the same like me. But I was totally surprised that it worked so well. Until now I miss nothing really. It’s quite the opposite. I am enjoying the fast editor and the customizing for the individual needs without manipulating the source of the project.

I started by installing the “C# (powered by OmniSharp)” plugin. That plugin provides syntax highlighting, reference recognition, debugging, etc. I tried this on an existing solution (The root folder where the solution is located needs to be opened). If you want to work with full support by C# plugin you have to work with solution files. When you want to debug your project for the first time, you have to set a launch file. Here you can specify the startproject etc. With this installed you can debug your code, set breakpoints, view variables, add watch expressions and modify the code (like in the immediate window in Visual Studio) within “Debug Console” window.

By the “Visual Studio IntelliCode” plugin you get the same AI intelli sense experience like in Visual Studio. And to get a more powerful importing namespace experience use the “Auto-Using for C#” plugin. It provides importing when typing knowing types. And for a better overview you can install the “vscode-icons”.

If you need a kind of “gui” for the nuget packages you can install the “NuGet Package Manager” plugin, but of course you can also use the dotnet cli for it.

Then I thought ok that is really cool but what is with unit tests. Can I run tests inside VS Code or do I have to use the cli? And the answer is – yes it is very easy possible. So you need first the “TestExplorer UI” and then the “.NET Core Test Explorer” plugin. After installing these you have a new icon on the left which opens the test explorer. You might have to edit the settings to specify on which locations the plugin should be search for unit tests. And then you can go with it.

For creating new solutions or adding projects to the existing solution you have to use the dotnet cli. I thought – but while I wrote this, I have done a quick research and I came to the “vscode-solution-explorer”. With this installed you have the same experience like in Visual Studio. On the left side (activity bar) you have an additional Visual Studio icon. By pressing it you have the same view like in Visual Studio. You can create projects add existing projects, view all references, add nuget packages, etc.

There are thousands of plugins how you can optimize your dev environment for your needs. So but I am happy with the plugins I have addressed here.

Plugin List

  • C# OmniSharp
  • Test Explorer UI
  • .NET Core Test Explorer
  • Auto-Using for C#
  • NuGet Package Manager
  • vscode-solution-explorer
  • Visual Studio IntelliCode

There are so many more useful C# helper plugins, for code generating etc. But to get started very comfortable this is my recommendation.

Missing

May be there are some tools which you will not get. For profiling, code quality and that stuff I use other tools in anyway which are then part of the pipeline… I really found nothing which I would miss.

Conclusion

So if I work on projects which runs on .NET Core or 5 then I will definitely choose VS Code for now. It feels good to work with. I do not get it why some guys say that VS Code is not so much powerful etc. I tried it and I think I will not install Visual Studio again, except to that point when I have to change some old webforms code :).

Refactoring, debugging, testing and writing the code feels very great to me and the setup was very easy. May be the entry is a little more difficult for new unexperienced users, but I think in Visual Studio you have to know what you are doing to! So give it a try and tell me your thoughts of your experience!

Skeleton for Vertical Layered Web API in .NET CORE

image by Vecteezy.com

…with fluent validation and automapping…

Today I just want to share a basic skeleton web api written in .net core. This is a kind of basic set up which works fine for microservices as well as some bigger ones. So I am a very big fan of not only horizontal layering ( see post Horizontal and Vertical Layers in Software Development). The example also shows how an application layer validation could e applied and how to easily map the items through the layers. So may be this helps the one or the other…

First the link to the source: layered-net-core-app

Some Description

So this skeleton app shows how an application could be split into vertical layers. This has multiple advantages (again see Horizontal and Vertical Layers in Software Development). Furthermore it shows with one example the flow of the data and connections from top (controller/application layer) to bottom (data layer). The data layer is only pseudo code and does not persist stuff, to focus on the layering and communication stuff.

The application has three layers:

  • Application Layer
  • Business Layer
  • Data Layer

…and several projects:

  • WebApi (Contains startup, adding middleware, setups, controller routings, forming the output and at least hosting the application)
  • WebApi.Configuration (DI bootstrapping / connecting the layers together)
  • WebApi.Common (Common stuff for the web api)
  • Application (The implementation of the application layer)
  • Application.Contract (The access to the application layer from outside the application layer)
  • Business (The implementation of the business layer)
  • Business.Contract (The access to the business layer from outside the business layer)
  • DataLayer (The implementation of the data layer)
  • DataLayer.Contract (The access to the data layer from outside the data layer)

Application Layer

The application layer is used here in this example by the web api project (Controllers). But the Controllers has only access to the application layer contracts. This allows us to avoid unwanted direct connection to use the implementation directly or to access business entities which knows the application layer implementation, but should not possible to expose by the api to the “outer world”. The application layer implementation knows only about the business layer contracts to communicate with it.

What this layer will does depends on the call and the needs for it. But from my perspective it is responsible for providing services for getting data from business/domain side and/or triggers domain/business logic. Furthermore it orchestrate all this needed calls to business logic and for example third party services to get/set the wanted result.

Business Layer

The business layer knows only about the data layers contracts. The business lay exposes only interfaces to use the layer and the business entities.

This layer should contain all the business/domain logic. All business relevant calculations and manipulations should occur here. No manipulation of business entities should be done outside of this layer.

Data Layer

The data layer knows only about the database or storage where all the data is persisted. The contracts from the data layer should only be used from the business logic to make sure that persisting data follows the rules inside the business layer. It is the connection to database and or storage. If provides repositories to store and get items from the database/storge.

Comments

This splitting is for the most services to much, but it demonstrates very well how the layer can securely communicate to each other. This pattern fits best for situations where no object relation mapper is used, because in entity framework the business entities are the tables in the data layer. So this layer is mostly a little overdone.

Personal I think a layering in microservices for an application layer and domain layer (where all the domain logic and data storage is handled ) is the best compromise to securely work with domain logic and have not total overhead of bubbling through thousands of layers. But this depends like every time on the needs of the application.

PART 2: WEB API – Building a Scalable App Environment with Infrastructure and Deployment

Using .NET, Angular, Kubernetes, Azure/Devops, Terraform, Eventhubs and other Azure resources.
Image for post
image by author

This is one part of a series. So if you have not read the PART 0: OVERVIEW you can go there and read it to get an overview of what we will actually doing here …

Introduction

In the last PART 1.1: INFRASTRUCTURE PIPELINE we have finalized the infrastructure part with building our pipeline for it.

Now we are going further and start the web api for our notification application environment. We will build the web api in .net core. The web api provides two endpoints. One for creating a message and another for getting all messages. This is sufficient for our example. Furthermore we implementing the use of azure storage table, event hubs, application insights and key vaults.

Yes – I repeat myself, but all the code here is not really complete production ready (error handling, retries, tests, etc.). Maybe I will put some more features (if there is anyone interested in) to all parts of this application, when we are ready with our base version. But our base versions has already a lot of stuff in there, so lets start…

Prerequisites

We only need an editor or development environment for .net and the .net core 3.1 framework for creating a .net core application (But I think this is obvious!). And we need the “Common lib” for our .net projects which will be described below.

Common lib

We need the common lib for this part as well as for our workers (next parts!) which should handle the sent notifications. The common lib source can be downloaded/cloned from the feature/part2 branch and put it next to the infrastructure folder. I will not list all the code here, but I will give a short overview about the content and describe the folders below.

Data

In this folder is the notification entity located which are saving in a azure storage table. For this there is also a repository, which take the job for communicate with storage table. (Please read the code for getting more information how this work in detail and or visit https://docs.microsoft.com/en-us/azure/cosmos-db/tutorial-develop-table-dotnet – and do not be confused that we are using the cosmos db api. This api is working with azure storage table.)

Extensions

One extension is to register to application insights and the other one for using the key vaults where we get later our secrets from. The key vault credentials will be pushed via environment variables (but this is part of the next part – 🙂).

Protobuf/Messages

The protobuf folder contains the notification message which we will be used to send/receive to/from our eventhub. I choose to send the message via this binary format. The messages folder contains the C# version of the message (which we will use in our code). If you take a look at the “Notifier.Common.csproj” fill you will find an item group which take this generation job (for this the gprc tool will be used).

Settings

Here are all settings defined as objects, which we will use in our needed .net projects.

WebApi

Again we should here create first a repository for the web api where we can push our code for creating the pipelines etc. in the next part. So let’s create an azure devops repository named “WebApi” in our “Infrastructure” project and clone this next to the “Infrastructure” and “Common” folders. The complete sourcecode can be also downloaded/cloned here.

Base Setup

Now create a .net core console project and name it (incl. the solution) “Notifier.WebApi”. We start by editing the “csproj” file. So replace/edit your contents with the following.

One interesting thing in the first property group is the “UserSecretsId”. So yes we use for local development the user secrets feature. So no secrets has to be in the repository. (The user secret will be created when you right click for the first time the “project -> manage user secrets”. Here we will do it directly in the project file and a good thing is to add the application name to the secret. Else it is very hard find it on your computer.)

The next item group is obvious. The “appsettings.json” which we will create very soon!

Then we have some nuget packages we will need for our web api and last but not least including the common lib as a project reference (make sure the path is right in your environment). A better approach would be to add the common lib as a git submodule or nuget package, but for now this is ok here.

It is time to create the “appsettings.json” file in the project root. Please put the content below into that file.

First we configure the log level for our logging. Then we see settings for our azure resources. These are secrets, so we will define them in our user secrets file later. And at last the core settings which is for our later angular frontend to access the the web api without cors issues.

Let’s go to our user secrets file. For this right click the project and select “Manage User Secrets”. If the file does not open (sometimes with older .net core versions I had the issue) you could do it by using the .net core cli or simply open the file located in “C:\Users\{YOUR_USER}\AppData\Roaming\Microsoft\UserSecrets\notifier-webapi-6fd34aeb-1b78-4492-86dd-a5aa00ce38cd” then put the following in there and find your secrets in your azure portal.

We use here the secrets from our acceptance environment. Here can you find your secrets…

  • Application Insights – Instrumentation Key: select notifier-application-insights-acc resource -> Overview and find the key on the top right.
  • Storage Table – Connection String: select notifierstoreacc resource -> Access keys and then copy the primary connection string.
  • Event Hubs – Connection String: select eventhubs-acc -> Event Hubs -> notifications -> Shared access policies -> send and copy the primary connection string.

Implementation

We start by directly implementing the functionality of the api. Later we wire this together when we setup the “Program.cs” and “Startup.cs”. we do this in this order, because else we have to jump between files and/or have to deal with errors because the things we want to wire are not existent at this time…

So let’s start by creating the model for the notification response. Please create a folder “Models” in the root project directory and create a class named “NotificationModel.cs” and put the following in to it.

The model only contains a message and a timestamp. So let’s go further with the services. For this create a folder named “Services” and create the following in files in there:

Above the service interface (“INotificationService”) with two simple methods in there. And now the implementation (“NotificationService”):

The constructor takes arguments for the logger, eventhub settings (object come from the common lib) and the repository which is also located in the common lib. With this we have all to start here. The “CreateAndSendAsync” does exactly what it is called. First it creates the entity and save it into the table and second it sends to the event hub. The functionality is split into two private methods which makes it more cleaner and better to read. Please check the private methods and the common lib functionality for further information how the event hubs and storage table is used here. (This is a very simple implementation herem without retries etc.)

Now we are ready to create the controller which defines our endpoints. So we start again with creating a folder “Controllers” and create a class named “NotificationController.as” in there and put the following code into it.

We inject the notification service here to let the service do our work. The controller creates the endpoint and format the response for our two methods. So we created here the following endpoints:

  • GET /api/notifications – returns all notifications
  • POST /api/notifications – create, save and send the notification

This is all we need for the logic! Now we need to wire this together…

Wiring the Parts

We start with the “Program.cs” where the entry point of the application is. Please open that file and replace the content with the following.

We start creating the host when the main function is called. At first we configuring the app by calling “ConfigureAppConfiguration”. We add the appsettings.json, environment variables and commandline arguments to our configuration. The we add (or try to add) the user secrets (which is the case in local development). Then we add (or try to add) the key vault (which is the case when we pass the credentials for it via environment variables – but we will discuss this in the next chapter). All the secrets will be replaced by user secrets or key vault. Next we configure our logging. First we register our console logger and second to log in application insights (which results in traces there). And in the last step we configure our web server where we will use “Kestrel” and call our web server startup class “Startup.cs” which contains the following.

Here we will do some startup stuff, but the main focus should be how configure our services. In line 45 we start configuring our app settings, so we can inject them with the IOptions<T> interface in our services, etc. Then we add the repository, and the notification service to our di context. You will find some other basic configuration here, which I will not describe here in detail.

Test

If we have all done here correct, you could run the application locally and test the endpoints. You could use postman or like me the the REST Client (for visual studio code plugin), which is quiet cool, because I can code my requests here and versioning them, etc. Following the requests:

So and if you have created some messages and get them you could check application insights on azure portal and view for example the logs (notifier-application-insights-acc -> Logs -> traces). Or check the application map, which should should show the connection between the components. At this time we can see that our web api sends to event hub and calls to to the azure table.

Conclusion

We have created a .net core web api here which cares about secrets and uses diverse azure resources like event hubs, application insights, key vaults and azure tables. We can now create, persist and read/receive notifications.

Preview

In the next PART 2.1, we will bring the web api to our acceptance stage running in docker and the kubernetes cluster. And this we want to integrate and automate in the azure pipelines.