stytex Blog

topics all around development, DevOps and more

Build CD Pipeline With Gitlab + GitLab CI for Spring Cloud Microservices

| Comments

Today I am writing about a continuous delivery pipeline, built on top of GitLab and GitLab CI. As a very simple but production tested deployment solution, I will push to a Dokku cloud.

In short, we will:

  • setup GitLab with CI
  • secure connections with TLS certificates using LetsEncrypt and Certbot
  • setup GitLab Container Registry
  • configure a pipeline for staging and production using a Dokku host
  • run the pipeline

The first part will cover the configuration part of GitLab, the second will demonstrate one possible deployment configuration to zero-downtime-deploy a microservice application, built with JHipster. But before we dig into the details, I will discuss what this setup is supposed to achieve.

(CD pipeline with GitLab CI)

This setup follows both, the immutable server pattern and a policy (I don’t know if there is a definition), where deployment to production must be allowed using CI by enforcing passed tests in all stages. In other words: we want to the one side, having the entire deployment process be as easy as git push origin master, but preventing failing deployments using CI and a good test coverage.

In practice, after following this tutorial, you should have an staging and production environment, which can be deployed using either git push origin master or git push origin staging. Before the funny part can start, we need some preparation. There are of course developers, who don’t have to care about such DevOps things, as they usually get some working CI/CD infrastructure, before they even start working. I had to figure out these things, having nothing more than a workstation, a good idea convincing my boss (microservices for business intelligence) and the permission to order bare metal. As a side effect of this tutorial, you will also be able to build and host an entire development + staging / production infrastructure in less then 100€ / month to run spring cloud microservices.

Setup

We now walk through four component setups we need for this:

  • git host
  • CI platform
  • container registry
  • staging / production cloud

This is nothing special for GitLab itself, but a general setup if you want to switch to modern CD pipelines. The reason why I decided to go with GitLab, is that the first three components of that listed are covered by GitLab out-of-the-box in their Community Edition. So let’s go:

Requirements

I have tried CentOS 6.X and 7, ubuntu 14 and 16 as well as good old debian 8, and found myself most happy with debian. Everything worked just after installation, no suspicious docker-in-docker bugs… For the minimum setup you should go with two dedicated servers running debian 8. Having more servers reduces the damage taken in system failure.

The first server will run GitLab and its CI runners. So first, we install GitLab. Despite there is a docker way to go, I prefer to install it via package manager to be able to the update. I followed this guide for installation using apt. You will get updates directly from GitLab via apt, when they got released.

We need at least version 8.8

Setting up GitLab CI

GitLab CI is really working similar to Travis CI. It publishes build tasks to “pending” state and waits, until one of the free workers starts with it. A runner itself is a docker container, which is able to start with a defined image to perform build tasks, such as testing.

In GitLab CI you either can register runners for individual projects, or shared runners for the whole GitLab instance. To make things easy, we define shared runners. This can be done in

Admin Area -> Runners

Here we find our registration token. To finish the CI setup, we just start runners. But this is one of the tricky things with GitLab CI. While these runners are docker containers, they must run docker commands on their own behalf, too. There are some different solutions, such as docker-in-docker or privileged mode, which all didn’t really work for me. So I ended up sharing the hosts docker socket to the runners to solve the runner side problems. To make my images building docker containers, I built some custom containers based on java:8 and node:4 by adding the docker client to them. Now step by step:

(on the GitLab machine)

1
2
3
4
5
6
7
8
9
10
11
12
docker run -d --name "gitlab-runner-1" --restart always \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /srv/gitlab-runner/config:/etc/gitlab-runner \
  gitlab/gitlab-runner:latest

docker exec -it "gitlab-runner-1" gitlab-runner register -n \
  --url https://<your-gitlab-host>/ci \
  --registration-token "<YOUR-TOKEN>" \
  --executor docker \
  --description "ci-runner-1" \
  --docker-image "docker:latest" \
  --docker-volumes /var/run/docker.sock:/var/run/docker.sock

Or you can use my Gist and

1
2
3
$ ./create-ci-runner.sh <runner-number> <registration-token> <gitlab-host>
$ # example
$ ./create-ci-runner.sh 1 xXXXYYYz https://my.git.lab.net

For spring applications, in particular JHipster apps, we can use xetys/java-8-docker or xetys/node-4-docker. So we be able to both, run our tests using maven / gradle / npm, and create docker images, which will be pushed to our container registry. And this is where we go next:

Setting up container registry

There are several public container registries for docker, such as Docker Hub. But for internal work, you sure want to have a private registry running. As of version 8.8, GitLab comes with a integrated registry, which has access policies according to the access control settings inside GitLab, but also support tokens for temporal sign ups inside CI runners.

Well, there are good news and bad news. The bad news is: since several versions docker it is very rude, if you work with insecure HTTP connections. More on that, you connection is treated as insecure, if you even use self-signed certificates. So the best way to go would be, using trusted certificates. Some years ago you would stop reading this article, as it would now cost you money to proceed with this article.

But happily, there are the good news: you can use LetsEncrypt and Certbot. This is a free TLS provider. To answer frequent questions directly: yes, it is really free, yes it works, you will have to be ok with your cert info will contain only your domain name, no further details.

A solution, which worked always for me, independent from the linux distribution, was:

1
2
3
$ wget https://dl.eff.org/certbot-auto
$ chmod a+x certbot-auto
$ mv certbot-auto /usr/local/sbin

Now in practice: GitLab comes with its own nginx configuration, and registry is enabled by default. We may activate the registry without TLS enforced, but will experience problems during CI because of dockers security policies. So first we will have to generate the certs, then setup GitLabs gitlab.rb properly. For the generation process, we will need to tell LetsEncrypt, that we are indeed are the owner of the GitLab servers domain. For this we shortly stop the GitLab nginx, generate the certs, and start it back again:

1
2
3
$ gitlab-ctl stop nginx
$ certbot-auto certonly --standalone -d <your-gitlab-host>
$ gitlab-ctl start nginx

note: after successfully cert creation, you may save this command into a cronjob script like this

1
2
3
/usr/bin/gitlab-ctl stop nginx
/usr/local/sbin/certbot-auto certonly --standalone -d <your-gitlab-host> -n
/usr/bin/gitlab-ctl start nginx

the -n option will renew the existing certs only if they are about to expire

There will be generated certificates in /etc/letsencrypt/live/. The last thing to do is, configure GitLab to use TLS for HTTP and registry, by providing a GitLab gitlab.rb as this:

/etc/gitlab/gitlab.rb
1
2
3
4
5
6
7
8
9
10
11
# ...
external_url 'https://<your-gitlab-host>'
registry_external_url 'https://<your-gitlab-host>:4567'

nginx['ssl_certificate'] = "/etc/letsencrypt/live/<your-gitlab-host>/fullchain.pem"
nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/<your-gitlab-host>/privkey.pem"


registry_nginx['ssl_certificate'] = "/etc/letsencrypt/live/<your-gitlab-host>/fullchain.pem"
registry_nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/<your-gitlab-host>/privkey.pem"
# ...

To make the changes take effect, run

1
$ gitlab-ctl reconfigure

And now on your workstation:

1
$ docker login <your-gitlab-host>:4567

with your GitLabs login credentials. If everything configured properly, you should now be able to push and pull to your registry.

Setup production environment

Well, there are a lot of solutions to actually deploy your docker images, cloud providers such as AWS, GCP, Azure, Profit Bricks, etc. If your application will have to scale, and you have operators to overwatch massive deployment, there sure will be some way to deploy docker images. The details differ from provider to provider. As a higher level of abstraction, you may deploy to kubernetes, which provide a unified API, abstracting the underlying layer off from the process of deployment. Although this is a very nice solution, it is still a hard job to get a production ready installation running. To be more precise: At my work I am currently operating things by myself on bare metal, so I just didn’t had the time to setup a kubernetes cluster I could really use for production. Maybe I will report in this blog when I find a straight way to go…So for now, I just need some quick and simple solution to get started with docker deployment, but more advanced then docker-compose. So Dokku was a wonderful way to go for me, as it is 100 lines of bash, run on top of some linux with docker on it.

note: Since I got familiar with these technology, docker released integrated swarm support since 1.12, which is maybe a more elegant solution than using dokku. Nevertheless, I didn’t found any zero-downtime-deploy support out-of-the-box by quick reading, which keeps me still running my cloud on dokku.

So in short, dokku is:

  • a small Heroku clone, accepting buildpack deployment from git push dokku master
  • supporting damn simple but customizable zero-downtime-deployments, enabled per default as 10 seconds health check + switch + 60s TTL for old containers.
  • providing a simple plugin system for postgres, elasticsearch and more

and you can install this from a two lines on a clean debian server (the second machine)

So, all we need for this is:

1
2
3
# for debian systems, installs Dokku via apt-get
$ wget https://raw.githubusercontent.com/dokku/dokku/v0.7.2/bootstrap.sh
$ sudo DOKKU_TAG=v0.7.2 bash bootstrap.sh

Then you have to visit your once in a browser, to setup some things you like.

You should setup at least one user to dokku for pushing / pulling, and one for the GitLab server. This can be done using dokkus sshcommand

In short:

1
2
echo "ssh-rsa <KEY1> user@workstation" | sshcommand acl-add dokku user
echo "ssh-rsa <KEY2> deploy@<your-gitlab-host>" | sshcommand acl-add dokku gitlab

This enables for both clients to ssh into the server, and directed to the dokku command, as well as to the internal git directory. I don’t focus on the git deployment capabilities, as this will make this article longer than it already is, there are good docs for this. We will use the dokkus database plugin postgres, and the docker-direct Plugin to perform some docker commands directly, i.E. using ssh <your-production-host> 'docker-direct pull <your-gitlab-host>:4567/project'

As mentioned, I don’t promote this as a best practice, but dokku currently solves a lot of my needs, because everything needed to interact with the host for deployment, is ssh, which is standard for all UNIX systems. This makes things easier in particular in CI.

Testing connectivity

As a dirty workaround for image pulling, I created a “dokku” GitLab user, to perform

1
$ docker login <your-gitlab-host>

with user “dokku” on the dokku host.

To test this, first create some empty GitLab project, add it to your user (in the following examples: “me”), and run the following ssh commands on your workstation:

1
2
3
4
5
$ docker pull nginx
$ docker tag nginx:latest <your-gitlab-host>:4567/me/empty-project # the name and registry path to your empty project
$ docker login <your-gitlab-host> # to be able to push images, this must be done once, and you will be authenticated over time
$ docker push <your-gitlab-host>:4567/me/empty-project # push it to the registry
$ ssh dokku@<your-production-host> 'docker-direct pull <your-gitlab-host>:4567/me/empty-project' # test that dokku is able to pull from your registry

If the last commands is pulling the image properly, you are ready to deploy from CI.

Building the pipeline

At this point, we are running at least two servers, one containing GitLab, several GitLab CI runners on it and the GitLab CI registry, and a dokku host on the second (or some other but similar solution like Deis, kubernetes, etc.). For simplicity we just deploy production and staging environments to one dokku host. I prefer to use 4 hosts, one for GitLab, one for runners, and one for each stage.

At this point we are able to perform dokku related commands just by using ssh, but I recommend to use a dokku client, because this are very simple, no dependency heavy toolbelt!

First, we need a JHipster application. First, so we generate a microservice application named ‘demo-app’ using yo jhipster, which we choose with these options:

  • microservice application
  • gradle as build tool (all this setup work for maven, too)
  • PostgreSQL as database
  • no additional things like elasticsearch, Hazelcast etc.
  • generate some simple entity using yo jhipster:entity

As we know, we need a JHipster registry to run microservice applications. For this, we use once per environment the following setup:

Inside a blank directory, create a Dockerfile:

1
2
3
4
5
FROM jhipster/jhipster-registry:v2.5.0
ENV SECURITY_USER_PASSWORD your-custom-password
ENV SPRING_PROFILES_ACTIVE prod,native

EXPOSE 8761

then, add and push it to dokku:

1
2
3
4
5
$ git init
$ git add Dockerfile
$ git commit -m "deploy registry"
$ git remote add dokku dokku@<your-production-host>
$ git push dokku master

This automatically creates a new dokku app named “registry”, which you can reach by hitting http://registry.:8761.

Once done, we now proceed with the deployment of “demo-app” to the dokku host registering with that registry.

To have the central configuration mapped to the config server:

1
2
$ mkdir /central-config
$ dokku docker-options:add registry "--v /central-config:/central-config"

step by step deployment

With a running JHipster registry, we now can start deploying JHipster microservices as many we need. For one single application, we walk through these steps:

1. create dokku app

1
$ dokku apps:create demo-app # or ssh dokku@<your-production-host> 'apps:create demo-app'

2. create postgres database using dokku postgres plugin:

1
$ dokku postgres:create DemoApp

(we can review the connection details again using dokku postgres:info DemoApp)

3. link database to your app

1
$ dokku postgres:link DemoApp demo-app

4. setting environment variables to overwrite defaults

1
$ dokku config:set demo-app SPRING_PROFILES_ACTIVE=prod SPRING_CLOUD_CONFIG_URI=http://admin:your-custom-password@registry:8761/config

5. link registry to app.

Despite we have the name “registry” clearly associated with that name to the outside, the real container name will be “registry.web.1”, as we can review by docker ps. So we must link that to the apps container, so it can simply access “registry”:

1
$ dokku docker-options:add demo-app deploy "--link=registry.web.1:registry"

6. setup spring cloud config

Here we can manage our configuration for all microservice application using spring cloud config. For that, inside directory “/central-config” we place a file named “DemoApp-prod.yml”:

/central-config/DemoApp-prod.yml
1
2
3
4
5
spring:
    datasource:
        url: jdbc:postgresql://dokku-postgres-DemoApp:5432/DemoApp
        username: postgres
        password: take from dokku postgres:info

7. Prepare deployment directly from workstation:

  • create a git repository for the project, like /me/demo-app
  • in projects settings, enable “container registry”, so it gets accessable from /me/demo-app:latest
  • run ./gradlew build -Pprod bootRepackage buildDocker to create a image. Be sure, it has the correct name in “docker.gradle” changed from “demo-app” to “:4567/me/demo-app:latest”
  • push the image using docker push <your-gitlab-host>/me/demo-app:latest

Then, on your workstation (or dokku host)

1
2
3
$ dokku docker-direct pull <your-gitlab-host>/me/demo-app # load image
$ dokku docker tag <your-gitlab-host>/me/demo-app:latest dokku/demo-app:latest # tag it for dokku deployments
$ dokku tags:deploy demo-app latest # deploy!

At this point you should get sure, the apps starts properly, connects to service discovery and cloud config properly.

8. Setup the final CI/CD pipeline

To summarize, what we have done in step 7, we just can make using this bash script:

dokku_deploy.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#!/bin/sh

set -ev

echo "deploying build with ID: $CI_BUILD_REF:$CI_BUILD_ID"


./gradlew -x test build -Pprod buildDocker # no tests, since this is done in other ci stage!

docker tag "$2:build" "$2:$3"
docker push "$2:$3"

ssh "$4" "docker-direct pull $2:$3"
ssh "$4" "docker-direct tag $2:$3 dokku/$1:$3"
ssh "$4" "tags:deploy $1 $3"

and place it to the root of our demo-app.

The general usage is:

dokku_deploy.sh
1
./dokku_deploy.sh <APP-KEBAB-NAME> <CONTAINER-REGISTRY-PATH> <TAGNAME> <SSH-CONFIG-NAME>

And for our demo-app it is

dokku_deploy.sh
1
./dokku_deploy.sh demo-app <your-gitlab-host>:4578/me/demo-app latest <your-production-host>

We add the private key of your deploy user for gitlab inside

Project > Settings > Variables

with key “SSH_PRIVATE_KEY”. We want to achieve that ssh <your-production-host> is automatically authenticated via RSA for user dokku.

So our first gitlab-ci.yml will look like this

.gitlab-ci.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
image: xetys/java-8-docker

before_script:
  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
  - eval $(ssh-agent -s)
  - ssh-add <(echo "$SSH_PRIVATE_KEY")
  - mkdir -p ~/.ssh
  - 'echo -e "Host <your-production-host>\n\tHost <your-production-host>\n\tStrictHostKeyChecking no\n\tUser dokku\n\n" > ~/.ssh/config'
  - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN <your-gitlab-host>:4567

stages:
  - test
  - deploy

buildtest-job:
  stage: test
  script:
    - ./gradlew build test

deploy-job:
  stage: deploy
  script:
    - chmod +x dokku_deploy.sh && ./dokku_deploy.sh demo-app <your-gitlab-host>:4578/me/demo-app latest <your-production-host>
  only:
    - master
  when: on_success

in detail:

  • image: xetys/java-8-docker provides usual java capabilities and docker commands to push images
  • before_script: here we setup the communication to our production host (add more lines like these for more different stage hosts)
  • stages: this are parallel job categories. Deploy jobs and test jobs will run in parallel, but deploy starts only when all test jobs finish
  • only: the deploy task will only start for changes on the master branch
  • the jobs: a job, as described here
  • when: on_success is a expression, which only allows to deploy to master, if all of the test jobs exited with code 0 (make sure to set ignoreFailure to false in build.gradle)
.gitlab-ci.yml
1
2
3
4
5
6
7
  test {
      include '**/*UnitTest*'
      include '**/*IntTest*'

      ignoreFailures false
      reports.html.enabled = false
  }

And finally, there it is. As soon some pushes or merges happens to any branch, the “buildtest-job” will run. If this was the master branch, the deploy-job runs after test jobs, which deploys the application to production hosts.

To be clear about the deployment part, it is really simple and maybe not adoptable to the most greater spring cloud solutions, but is done in a very similar fashion using systems like kubernetes or fabric8. But the principals today, are quite the same: build artifacts in form of docker images when test succeed, push to container registry, deploy from registry to your cloud. In this case, the dokku setup is easy to follow and simple to reproduce in other systems. I am used to follow the pattern, to use master branch for production stage and staging branch for staging environment. Another pattern is to deploy tags to production and master to staging.

As an deployment alternative, you can work with hooks after pushing to registry, so your production hosts pulls the image itself, triggered by that hook. There are a lot of different ways to setup this pipeline. For me it was not easy, to get quickly started with deploying some production cloud, while learning Spring, microservices, and project management, to give my team members some tool they can easy get started with. So even if this way might look unconventional, my team get away from workstation after a git push, walking the way to our product users while the application gets deployed. This is, how continuous delivery looks to me!

Last but not least, using German server hosting provider like Hetzner or server4u, you will pay about 100€/month for a two machine setup, each with 64GB RAM and Intel Xeon CPU inside, what is much cheaper then the same costs with cloud providers.

Conclusion

With this article I presented how to setup a working cloud, similar to several production environments I run with spring cloud microservices built with JHipster with a good performance a flexible system. I hope this is not looking to naive or stupid, since I am actually a software developer, not a system administrator :)

P.S.: After buying servers and domains, I need about 50 minutes to setup this setup including testing everything works.

Have a great week!

JHipster 3.7 Tutorial Pt 3: Secure Service Communication Using OAuth2

| Comments

Intro

It has been a while since my last article. For those, who might be crazy enough to follow my blog should remember the statement of my last article on how to get started with JHipster microservices, which became a core feature since version 3.0.

My statement was:

I am considering to implement a UAA using JHipster and spring-cloud-security and spring-cloud-oauth2, as I did in my last tutorial.

To review, how I got to this statement, I was writing about how Ribbon and Hystrix can be used in order to let one microservice being able to request resources from another microservice, without going over the gateway, and being aware of service failure, load balancing and so on. But I left this without any considerations on security, as with the JWT solution, there wasn’t any suitable workaround, which would do that job in a proper way.

So, let me quickly what happened the last 6 months:

When my last JHipster article made some rounds over the world wide web, I generated off some demo microservice setup and moved the user domain from the gateway to one microservice I called “jhipster-uaa”. After this, I performed the steps from my spring OAuth2 security article, so I had an AuthorizationServerConfigurerAdapter built in that UAA and turned the WebSecurityConfigurerAdapter from the microservices into a ResourceServerConfigurerAdapter. This worked perfectly, so I made some screen:

and Julien Dubois, lead developer of JHipster, asked me to document this. I did a little more than that: contributing the entire setup including further updates for declarative service communication and testing tools directly to JHipster, as my aim was to make this overcomplex stuff adaptable easily for all developers.

In other words, I want to:

I want to have secure microservices, which can scale

be open and easy to use for all JHipster users, as well as spring developers, who just can generate JHipster code to look how it works. So after 6 months of different works on the generators, JHipster released my official documentation on using JHipster UAA, so I can proudly sum up my thoughts from late march this year, because now it is really simple to use.

Setup

To use the new features, the service generation differs from the way I described it in march. Let’s walk to the major differences:

  1. It’s now required to generate one more service, called “uaa”. The very first question of yo jhipster will provide a new type of application: JHipster UAA server
  2. For all other services, there will be a question: “which kind of authentication” to use. Here the “uaa” option is required.
  3. As for now, it is better to use Hazelcast as hibernate cache (I will explain this a bit more below)

You can find the final project in my GitHub repository

To start of quickly, first use the build script in root:

1
$ ./build-all.sh

and then just start the whole setup using:

1
2
$ cd docker/
$ docker-compose up -d

Using these settings, the gateway will not contain the user and account endpoints anymore, since they are now in the UAA. Furthermore, spring cloud security is used in order to provide security. At this point, there isn’t a big difference to the JWT authentication type, to do all the other stuff, which was possible until today.

secure service-to-service communication

In part 2 I showed how to use Ribbon and Hystrix manually with RestTemplate, to let the “foo-service” request data from “bar-service”. To make this secure, now the UAA can be used. I will first describe, how we could that in theory, following the pattern as we did it last time.

First, we would need a new RestTemplate (with Ribbon and Hystrix) to communicate with the UAA. In addition, we need to setup basic authentication with username and password “internal”, to call the following url: http://uaa/oauth/token using POST and one field “grant_type” with value “client_credentials”. As you might recognize, we are about authenticating the microservice for the OAuth client named “internal”. The response will be an access token. Now, we get back to our REST client for “bar-service”, and add an authorization header with that access token as “Bearer”. Despite this is a lot of work, we now achieved a secure communication, as the foo service is now authenticated as a service principal, which has an valid access token. The “bar-service” now is able to setup access controll configuration based on roles and scope.

This sounds awesome, doesn’t it? I thought of this is not straight forward enough, since security is basic, and should not be a leg braking task for development. This is where the Feign project comes into play. When thinking about feign, I always have an association with the moment, when I firstly realized, how “spring-data-jpa” is working, when only an interface has to be defined as repository, without doing any real implementation.

Something similar does feign to you. We just define some interface, using exactly the same @RequestMapping annotations as used for the endpoints in the service we are going to consume, and voila: some magic does the implementation for us…including Ribbon, Hystrix and (with some additional support from JHipster UAA) access token requesting for uaa.

Assuming we got an JHipster UAA generated application setup as it is out of the box, there are now two steps, to get it working

making a representation of “Bar” in “Foo”

Ok, this is really obvious, but we need that less code, that I keep some extra lines in my article on that :)

com/example/client/Bar.java
1
2
3
4
5
6
7
8
9
10
11
class Bar {
  private String value;

  public Bar() {}

  public Bar(String value) {
    this.value = value;
  }

  //getters, setters..
}

You just can copy&paste that from Foo, by removing the JPA and validation annotations.

and finally:

declaring a authorized feign client

com/example/client/BarClient.java
1
2
3
4
5
@AuthorizedFeignClient(name = "bar")
interface BarClient {
  @RequestMapping(value = "/api/bars", method = HttpMethod.GET)
  List<Bar> getBars();
}

and we are done. We now can just @Inject that client to get things working. In order to make some demo endpoint on “foo-service”, we may define a REST controller like this:

1
2
3
4
5
6
7
8
9
10
11
12
@RestController
class SomeController {
  @Inject
  private BarClient barClient;


  //I have to mention I am a big fan of these cool new annotation from Spring Boot 1.4.0, which is now also part of JHipster!!
  @GetMapping("/api/client/bar")
  public List<Bar> getBars() {
    return barClient.getBars();
  }
}

Of cause, this is just one example. This can be done with some services and so on.

Now, what is that @AuthorizedFeignClient annotation? This is some JHipster magic in order to call the UAA server to get a token (if needed, it’s stored in memory until it expires). If it is still not clear: it is possible to use @FeignClient without any predefined configuration as it is, without any JHipster magic, for every other purpose. This is a nice feature now available to all JHipster microservice applications, if you are already familiar with feign.

One more note on @AuthorizedFeignClient, I used the “name” property of feign client declaration, in order to make it working. For now, the other notations, such as @AuthorizedFeignClient("bar") or @AuthorizedFeignClient(value = "bar") are not working, due to a bug inside spring.

using Hazelcast to the rescue

In my recent articles I preferred to generate my projects with a minimal setup, in particular without 2nd level cache. For the moment, there is some issue where a microservice tries to resolve the “uaa” endpoint in configuration phase, to fetch the public key to sign incoming JWTs. Without 2nd level cache this happens to fast, so there aren’t any registered services to load balance, because they weren’t retrieved. Using Hazelcast is a dirty fix for that.

Advanced topics

Despite this solution is quite new to the official JHipster release, it is already battle tested in several applications, where I used my own JHipster branch to generate the application. This led to two advanced topics about testing, which made the UAA feature set much more usable in production.

Component testing

As we can assume, feign clients are generally working, it makes no sense to enable then during test (what would mean we are testing that feign itself works). Instead of this, we use some simple mocking technique, to fake the client, to actually test the components using them.

Lets consider some service like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
@Service
class FooBarService {

  @Inject
  private BarClient barClient;

  @Inject
  private FooRepository fooRepository;

  public void syncFoosWithBars() {
        barClient.getBars().forEach(bar ->
            fooRepository.save(fooRepository.findOneByValue(bar.getValue())
                .orElseGet(() -> new Foo(bar.getValue()))));
    }
}

note: the FooRepository should have a findOneByValue method returning Optional<Foo> and Foo should have an extra constructor Foo(String value)

This service looks up for existing “bars”, and syncs “foos” with the same value.

To test that this service is working properly, we need to write the following test:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
@RunWith(SpringRunner.class)
@SpringBootTest(classes = {FooApp.class, SecurityBeanOverrideConfiguration.class})
public class FooBarServiceUnitExTest {

    @Inject
    private FooRepository fooRepository;

    @Inject
    private FooBarService fooBarService;

    @MockBean
    private BarClient barClient;

    @Test
    public void testSync() {
        given(barClient.getBars()).willReturn(
            Arrays.asList(new Bar("one"), new Bar("two"), new Bar("three"))
        );
        fooBarService.syncFoosWithBars();
        List<Foo> synced = fooRepository.findAll();

        assertFalse(
            synced.stream()
                .map(Foo::getValue)
                .collect(Collectors.toList())
                .retainAll(Arrays.asList("one", "two", "three"))
        );
    }
}

Obviously, this test would fail the most time in a real environment without the mocking, as it logically depends on the contents of bar service, if not exactly these value exists. We use @MockBean to overwrite the actual injection of the bar client, and mock the return behavior using given(...).willReturn(...) from Mockito.

Security testing using @WithMockOAuth2Authentication

Another desired testing technique is, to test how the application behaves when access control configuration is applied. In order to make not to much work on this scenario, we look into the UaaConfiguration and see, that per default internal clients get the scope “web-app” granted. To make a RBAC style restriction, we need to apply the following rule in MictoserviceSecurityConfiguration:

.antMatchers("/api/foos").access("#oauth2.hasScope('web-app')")

which enforces all request to “/api/foos” exclusively granted for internal clients.

To test this configuration holds, we write the following test class.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
@RunWith(SpringRunner.class)
@SpringBootTest(classes = {FooApp.class, SecurityBeanOverrideConfiguration.class})
public class SecurityIntTest {

    @Inject
    private WebApplicationContext context;

    @Inject
    private OAuth2TokenMockUtil oAuth2TokenMockUtil;

    private MockMvc restMockMvc;

    @PostConstruct
    public void setup() {
        this.restMockMvc = MockMvcBuilders
            .webAppContextSetup(context)
            .apply(springSecurity())
            .build();

    }

    @Test
    public void testUnauthorizedAccess() throws Exception {
        restMockMvc.perform(get("/api/foos")
            .with(oAuth2TokenMockUtil.oauth2Authentication("test")))
            .andExpect(status().isForbidden());
    }

    @Test
    public void testAuthorizedAccess() throws Exception {
        restMockMvc.perform(get("/api/foos")
            .with(oAuth2TokenMockUtil.oauth2Authentication("test", Sets.newSet("web-app"))))
            .andExpect(status().isOk());
    }
}

Conclusion

I really hope “my considering of writing some UAA for JHipster” lead to some mighty tools, which demystify the usage of OAuth 2 in context of microservice security solution, which easily can be used for JHipster (or just spring based) applications.

This contribution also lead finally to a positive voting for my membership at the JHipster core developer team, which is a great honor to me.

As usual I am open for questions and feedback in the comments section as well as on StackOverflow using the “jhipster” or “spring-cloud-security” tag or on GitHub.

For the further articles the following topics are in my pipeline:

  • GitLab CI + dokku setup (complete deployment pipeline)
  • spring cloud consul as alternative to Eureka

Have a great upcoming weekend!

Security in Applications: 2nd Order Privileging

| Comments

A word on security

While developing modern web or even cloud applications, using tools like Spring, JHipster or Rails or whatever, security comes in many different facets. This article is not bound on a specific framework. This is a general concept.

Today I am going to talk about how to bring access management in a easy business friendly, but mighty concept. It’s about, how to organize access grant to your business resources. You maybe think, this is “authorization”. While authorization is a technological implementation of bringing authorizing information when it’s needed, “privileging” is the way of what authorities exists and how they are related to the system users.

Definitions

It’s much easier to talk about these thinks, when there are exact terms, so lets define some!

user and authorities

Users of the system are allowed to do things, defined by authorities or roles. An authority is nothing more than a word, which may have some convention as “RESOURCE_READ” or “RESOURCE_WRITE”. So developers can use these words, to secure or restrict access to resources, forcing users to have certain authorities.

authentication, authorization

The process, which exchanges user login credentials (username and password) into a set of user details and authorities, is called authentication.

The implementation of giving access, by analyzing authorities, is authorization.

privileging

While we got users and authorities, and a cool system to implement any kind of authentication and authorization, there is still a system missing, which defines “who can do what, depending on authorities”.

first order privileging

This is what I call the most plain way to accomplish this. Behind this scary name is a very usual process:

In application logic, business data or processes are defined as resources. A resource can be a noun like “article” or “product”, or a proccess like “invoice”, “billing”. We can perform actions on these resources. But for simplicity, lets everything to do with resources, just call resource.

many users has many authorities

There is a flat n to m relation between users and authorities.

a resource may be secured by zero to many authorities

A resource may be secured by a set of several authorities. But how many of them the user should have, to get access granted?

disjunctive restriction policy

user is having any of the authorities

This is the most common use case of restriction. Suppose you secure the resource “employee” with “EMPLOYEE_READ, STAFF_READ”. A user will be able to get employee data, by having one of these authorities.

In Spring Security, this would be

1
antMatchers("/employee").hasAuthorities("EMPLOYEE_READ, STAFF_READ")

conjunctive restriction policy

The same as disjunctive, but now the user have to hold all of the securing authorities.

custom restriction policy

Suppose resource R has to be secured by the authorities “A”, “B” and “C”.

disjunctive is: user own A or B or C conjunctive is: user own A and B and C custom would then be something more specific like: user own (A or B) and C

restriction types

We are living in a world, where CRUD APIs are ruling the day. More ore less, but we actually perform CRUD on some resources all the time. Or more general, we read and write them. So there are exactly 2 types of restriction: read and write.

It clearly makes sense, to force a naming convention with suffixes “X_READ” and “X_WRITE”, to distinguish these.

implementation

There are resources, users and authorities.

many users have many authorities

many resources are secured by many authorities by a restriction policy

That’s all about first order privileging. It’s time to focus first on

Advantages

It’s a little bit of theory, but actually a very simple system, which can be easily communicated between product owner and development team. Nearly any kind of access restriction can be modeled with this pattern.

The most commonly used web frameworks offer tools, which are perfect to apply this kind of privileging, since an disjunctive restriction can be implemented as one line of code. One line of code making a whole resource secured :D

Disadvantages

With first order privileging, is still one major question open:

Who decides, which user gets which authorities?

So with first order privileging, the authorities are also a resource of your business domain. In a very simple use case, there are two authorities “AUTHORITY_READ” and “AUTHORITY_WRITE”. But at the moment, when there is a need of individual users being granted to write to only certain authorities, things get ugly.

Of course you can hold several resources, aggregating them together into your authorities. You could have different entities or a per-row access restriction.

But isn’t there a more elegant way?

second order privileging

An alternate term would also be: hierarchical access restriction. So like members of companies or organizations are structured in a hierarchy, the same often happens by restricting access to the business domain depending on its users.

Suppose an CRM application, managing commission contracts and customer relation to these. So here the data model itself may have to be secured in tree style.

Boss can open and change every contract, having his partners, which only can open their own contracts, or contracts done by partners working for them. The good old pyramid. Of course, boss doesn’t want the partner of his partner of his partner to lookup his commissions. More on that, he employs some trainees, managing the built in CMS service. His chief content manager should be able to decide without boss, which pages of the CMS may be managed by his trainees.

And so on…you know what I mean, I hope!

Let’s dive into a little bit of theoretical computer science for this:

the “grant” restriction type and parent grant authority relation

Now two more things are applied to the authority model. First is straight forward:

a “grant” authority is responsible for being able to grant subsequent authorities

Or just: a user having “RESOURCE_GRANT” can give other users “RESOURCE_READ” and “RESOURCE_WRITE”. But not because there is a “RESOURCE_” in the authorities title. It’s because of the second thing: “RESOURCE_READ” and “RESOURCE_WRITE” have “RESOURCE_GRANT” as their parent grant authority

Here is an example:

As you see, the second order privileging can be visualized as an authority tree.

In this setup, a user having “CONTENT_GRANT” can permit users (including himself) all authorities in the tree, but a “blog manager” having “BLOG_GRANT” only for blogs. But a user with “CONTENT_WRITE” can only write all the content, and cannot decide who else is permitted to write a blog or a page.

Back to securing resources, nothing have changed. The new grant type is not affecting the read/write access to the resource, so this would be a spring style setup:

1
2
3
4
.antMatchers(HttpMethod.GET, "/blogs").hasAuthorities("CONTENT_READ", "BLOG_READ")
.antMatchers(HttpMethod.POST, "/blogs").hasAuthorities("CONTENT_WRITE", "BLOG_WRITE")
.antMatchers(HttpMethod.GET, "/pages").hasAuthorities("CONTENT_READ", "PAGES_READ")
.antMatchers(HttpMethod.POST, "/pages").hasAuthorities("CONTENT_WRITE", "PAGES_WRITE")

forcing implicit authorities

What if there is a grant authority called “SYSTEM_GRANT”, which is on top of the authority tree, and a user, having just only this authority. When he attempts to change a page, he should be allowed to do so, because if not, he just could give himself the proper authority and repeat his attempt.

So in this concept, it makes sense to say:

a user is permitted to do every thing, what he can grant to himself

For me there are at least two approaches to accomplish this: check the tree during access request, or during access granting. Doing it while the user is already awaiting a result from your application, might be a hard task.

The other way is, to give the user all implicit authorities, when they are changed.

The advantage of access request analyzing is, a user always get access grant to new sub authorities, when they are created, but may be complex and time wasting process.

Giving implicit authorities is the fastest solution during access request, because at this moment it’s the same as in first order privileging. But when the authorities change, the rights have to be recalculated.

authority tree traversal

In general, a tree traversal is a mapping from a tree into a ordered set or a list. Although there some different kinds of traversal, the most of them fit the needs of authority implication. So we just pickup pre-order tree traversal. You just go from left to right, starting from a node.

Giving a user a grant authority, is the same as applying a subtree of your authority tree to him. Since we want a list, the tree traversal of this subtree is the set of implicit authorities.

Need some examples?

The pre-order tree traversal of the content authorities (see fig.) would be

1
2
"CONTENT_GRANT", "CONTENT_READ", "PAGES_GRANT", "PAGES_READ", "PAGES_WRITE",
"BLOG_GRANT", "BLOG_READ", "BLOG_WRITE", "CONTENT_WRITE"

This are pretty much all authorities, what makes sense, if you grant the root authority of the tree.

Traversing starting from “BLOG_READ” would only give

1
"BLOG_GRANT", "BLOG_READ", "BLOG_WRITE"

top-bottom grant, top-bottom revoke, bottom-top revoke

Let authorityTraversal be a mapping (a function) from node in your tree into a list of authorities.

top-bottom grant is actually what I was talking about the last sub chapter: granting a authority having authority children, generates a list of authorities using

1
2
implcitAuthorites = authorityTraversal(authority)
newUserAuthorities = UNION(userAuthorities, implcitAuthorites)

union means combining two list without duplicates

This is mandatory to second order privileging.

top-bottom-revoke is

When you revoke a authority, you should also revoke it’s children

or:

1
2
implcitAuthorites = authorityTraversal(authority)
newUserAuthorities = userAuthorities - implcitAuthorites

bottom-top-revoke is

When you revoke an authority, go from parent to parent in the users authorities list, and revoke the traversal of it

or: when a user has BLOG_GRANT and CONTENT_GRANT, you should revoke also CONTENT_GRANT, when revoking BLOG_GRANT. or:

1
2
3
4
5
6
topAuthority = authority
while(topAuthority.parent != null and topAuthority.parent in userAuthorities)
  topAuthority = topAuthority.parent

implcitAuthorites = authorityTraversal(topAuthority)
newUserAuthorities = userAuthorities - implcitAuthorites

The two revoke styles could be optional, for comfortable performing of access changes when having a massive authority database.

Advantages

This concept now can model a very specific access restriction model, which allows fine-grained privileging. During access request, second order privileging behaves like first order privileging, when the implicit authorities were computed while granting.

Once the privileging of an application is designed, it also can be easily explained to the product owner, and gives the developers the possibility of quick response to new requests on business logic security.

Disadvantages

More complex :-)

why that freaky?

I am a big fan of mighty concepts, which can model simple cases at the beginning and are open to become very complex, without breaking them.

One important thing is, the very most popular frameworks are offering all the tools needed for that out of the box. So it’s independent from the concrete implementation and doesn’t need to develop complex libraries, just using common tools. It takes the power of trees, which can be easily prepared during on-write time and also fast on-read analyzing, using modern document based or even better: graph based DMBS like Neo4j.

This little concept might be more a conclusion of current practices in web application development or security design, then a new innovating concept. It’s packaging it into a bit of theory and some kind of a recipe for your next security design.

closing words

I promised some actions in my last article for positive feedback, which was awesome! I was actually very busy with contributing the promised UAA solution directly to JHipster. I will write about this later!

If some fans of JHipster or the developers themselves would like to see a prototype inside the generators or as module, I might consider to implement this also :)

My aim was to share my complex thoughts on this in a article which is easy to understand, and I hope I did accomplish this. If not, I am open to critics and suggestions. Be free to give feedback.

Have a great weekend!

JHipster 3.0 Tutorial Pt 1+2: Intro + Basic Service Communication

| Comments

Introduction

It has been a while since my last post, so since JHipster 3.0 was releases during past week, it’s a perfect time to talk about microservices, and how JHipster can help.

The subtopics today are:

  1. scaffolding a microservice architecture with JHipster 3.0
  2. communication between services with decentralized load balancing (Ribbon) and optional circuit switching (Hystrix)
  3. (maybe in next article) applying the full power of OAuth2 client credential grant to apply fine-grained securing (with possible use cases for this)

Part 1: Scaffolding a microservice application using JHipster

For those who didn’t know: JHipster is a great project for creating awesome applications with Spring, Spring Cloud OSS and more tools for backend, as well as AngularJS, Bower and Gulp on the frontend. Basicly it is a set of yeoman generators, providing creation and modification of a standard setup width all these tools wired up in a working project. It would take to long to list all things inside the box of JHipster, so I follow directly to making our hands dirty :)

Prepare your system with all the tools mentioned in the Installation Guide from JHipster. In short you will need a JDK, npm, docker and gradle working.

And of course: a bit of patency! It took me nearly half of a day to let all the npm installs and gradle builds and IntelliJ 2016.1 monster slow indexing :)

Definition of Done: the basic mircoservice setup

We will generate an application containing a JHipster gateway (which contains Netflix Zuul and the AngularJS frontend), 2 service applications “foo” and “bar”, which will have one entity “Foo” or “Bar” with a field named “value”.

1
2
3
$ mkdir foobar
$ cd foobar
$ mkdir foo-service bar-service gateway docker

These commands will create a basic directory structure. “docker” will contain a docker-compse.yml to start all the services, the other directories for the services themself.

So first we start with foo-service:

1
2
$ cd foo-service
$ yo jhipster

The generator will ask you for a lot of things, but this is how we answer: We want a microservice application on port 8081 with any SQL based Database, no Cache (don’t make things harder then needed, we just start off :D ), we want gradle building!

Remember these settings, we will generate the bar-service the same way. This will take some time, but then we get an application with a lot of code, but actually no entities. Therefore we do:

1
2
$ yo jhipster:entity Foo
$ ./gradlew build -Pprod buildDocker

with exactly one field “value” of type “String”, no validations and no relation ships.

The gradle command will fetch all the fancy dependencies for spring and finally generates a docker image, which will be saved to our local image registry (I will write some words about that in the docker part).

We are done here, so we to exactly the same with bar-service on port 8082:

1
2
3
4
5
6
$ cd ../bar-service
$ yo jhipster
[...]
$ yo jhipster:entity Bar
[...]
$ ./gradlew build -Pprod buildDocker

Arriving here we got 2 docker images ready for serving their resources, without having any GUI to show them. This is where the gateway comes into play. We generate it in our gateway directory by answering microservices gateway in the generator, routing it to port 8080 and also going with a SQL based Database, with gradle selected as build tool.

Well, we now can directly import the previously generated entities here, by doing:

1
2
3
4
$ yo jhipster:entity Foo
[...]
$ yo jhipster:entity Bar
$ ./gradlew build -Pprod -x test buildDocker

The generator will ask if we want to generate them from our microservices and of course were to find them (../foo-service and ../bar-service).

I have skipped the tests, because I expierenced some issues in this phase using the frontend test frameworks…

Now we got all our applications ready. But we need more of the microservice ecosystem to start it right off. At a very minimum we need a JHipster Registry", a service discovery and configuration server. We could check it out from their GitHub and run the registry and all three services each in a terminal. Since the ports are unique, this will give us a possible (and sometimes very comfortable) option, to test your application.

Using Docker Compose to start the cloud

But there is the more elegant way using docker. Docker provides lightweight virtual machines called “containers”. A container may consists of an entire OS like ubuntu or CentOS, but fully running on its hosts kernel. So you can isolate complete environments inside a single docker image and starting your application on them.

This is not just very usefull during development, you can also deploy these images in production.

A very powerful tool is Docker Compose, which configures several docker containers at once and start them together, wiring things like network and shared volumes, which can be configured in a single YAML.

Long text, quick solution:

1
2
$ cd docker
$ yo jhipster:docker-compose

Here you submit “../” as directory, select the three services and finally:

1
$ docker-compose up -d

Depending on how many you or your company spent on your current computer and your network…this takes some time :D

But after a while, you will get a Eureka running on http://localhost:8761, your application running on http://localhost:8080 and a very cool Kibana dashboard on http://localhost:5601 (if you choosed to use it in the docker-compose sub-generator, I spent about an hour of clicking inside it the first time :) )

Howto: Configuring IntelliJ IDEA

After a long fight with 2016.1 and final downgrade to IDEA 15.0.2, I found out the most pleasant way is to have a central gradle configuration. Go to our root folder and add these files:

build.gradle
1
2
3
4
5
repositories {
    mavenCentral()
}

apply plugin: 'java'

and

settings.gradle
1
2
3
include 'foo-service'
include 'bar-service'
include 'gateway'

Now you can point IDEA to the central gradle file and use it for its indexing.

End of the lazy part

We didn’t code a lot till here, only generating. Since you can use JHipsters domain language JDL for modelling your entities or even UML, this is really huge. You just scaffolded a working cloud API with basic security, API documentation and completelly tested in this generated state.

Even if you don’t consider to use it in a real application development, JHipster is a good tool to see how things work.

Part 2: inter-service communication

A lot of questions come up when playing arround with all these tools. One quite obvious is:

How do services can communicate with each other?

This question was asked also on the JHipster 3.0 night in Singapore, and the answer was like

You should design your services in a way, so they won’t have to communicate. But you always can access them using the gateway

This actually, is a solution. But it doesn’t fit the microservice idea really, because this will turn the gateway into a centralized proxy and load balancer, which becomes a single point of failure.

One other point is, that each services has the tools, to get the service hosts by their own, using Eureka Discovery Client. The gateway to the same thing, using its built in routing. Suppose you have got 4 Services A,B,C,D and a gateway, so the worst call chain were A asks B,C and D would be like

A > GW > B, GW > C, GW > D

were it would be like

A > B, A > B, A > D

when would use discovery clients inside the services.

With this, every service has its own loadbalancer!

Definition of Done: Ribbon loadbalancing

Suppose, we want service “foo” to communicate with “bar”. For this we define a “BarClient”, which acts as a service for later using (similar to the internal repositories and services). This client will use Netflix Ribbon for load balancing client and retrieve the Bar API with RestTemplates.

Or let just an integration test explain, what should work:

src/test/java/de.stytex.foo/BarClientTest.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(classes = FooApp.class)
@WebAppConfiguration
@IntegrationTest
public class BarClientTest {
    Logger log = LoggerFactory.getLogger(BarClientTest.class);

    @Inject
    BarClient barClient;

    @Test
    public void testContextLoads() throws Exception {
        //should be loaded

    }

    @Test
    public void testEntityLifeCycle() throws Exception {
        Collection<Bar> bars = barClient.findAll();
        int barCount = bars.size();

        Bar myBar = new Bar();
        myBar.setValue("my awesome bar!");

        //test creating
        Bar result = barClient.create(myBar);

        assert result.getId() > 0;
        log.info("created bar entity with id {}", result.getId());


        //test entity get
        myBar = barClient.getOne(result.getId());

        assertEquals(myBar.getId(), result.getId());

        //test collection get
        bars = barClient.findAll();
        assertEquals(barCount + 1, bars.size());

        //test entity update
        myBar.setValue("my changed value");
        result = barClient.update(myBar);

        assertEquals(myBar.getValue(), result.getValue());

        //test delete
        barClient.delete(result.getId());
    }
}

This is what the bar client should be capable of after this part. We just do normal CRUD things here. You may notice, we use the Bar class in our foo service, which never was generated. We have to add this of course and only as a POJO, not as an entity bean.

src/main/java/de.stytex.foobar.domain/Bar.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
public class Bar {
    private long id;
    private String value;

    public String getValue() {
        return value;
    }

    public void setValue(String value) {
        this.value = value;
    }

    public long getId() {
        return id;
    }

    public void setId(long id) {
        this.id = id;
    }
}

To have a general toolset of a load balancing client, lets define:

src/main/java/de.stytex.client/AbstractMicroserviceClient.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
public abstract class AbstractMicroserviceClient<E> {
    private String serviceName;

    @Inject
    protected ObjectMapper mapper;


    /**
     * force the descendants to call super("SERVICE_NAME")
     *
     * @param serviceName the service name known by service discovery client
     */
    public AbstractMicroserviceClient(String serviceName) {
        this.serviceName = serviceName.toUpperCase();
    }

    abstract public Collection<E> findAll();

    abstract public E getOne(long id);

    abstract public E create(E object);

    abstract public E update(E object);

    abstract public void delete(long id);

    protected RestOperations restTemplate;

    private LoadBalancerClient loadBalancerClient;

    /**
     * let lately inject the client to retrieve host and port of the target service
     *
     * @param loadBalancerClient autowire parameter
     */
    @Autowired(required = false)
    public void setLoadBalancerClient(LoadBalancerClient loadBalancerClient) {
        this.loadBalancerClient = loadBalancerClient;
    }

    /**
     * Constructs a url for rest template
     *
     * @param path resource path on the service
     * @return a url String for use in RestTemplate
     */
    protected String getUrl(String path) {
        String url;
        ServiceInstance instance = loadBalancerClient.choose(serviceName);
        String prefix = instance.isSecure() ? "https://" : "http://";

        url = prefix + instance.getHost() + ":" + instance.getPort() + "/api/" + path;


        return url;
    }

    /**
     * Helper method, because getUrl("resource", 1) is cooler than getUrl("resource/" + 1)
     *
     * @param path the resource entities path
     * @param id a numeric resource identifier
     * @return a url String for use in RestTemplate
     */
    protected String getUrl(String path, long id) {
        return getUrl(path + "/" + id);
    }

    @Inject
    public void setRestTemplate(RestOperations restTemplate) {
        this.restTemplate = restTemplate;
    }

    /**
     * generates a JSON string for entity of type E
     *
     * @param entity the entity to be converted
     * @return a JSON representation of the entity
     * @throws JsonProcessingException
     */
    protected HttpEntity<String> getJsonEntity(E entity) throws JsonProcessingException {
        HttpHeaders headers = new HttpHeaders();
        headers.setContentType(MediaType.APPLICATION_JSON);
        String entityJson = mapper.writeValueAsString(entity);

        return new HttpEntity<>(entityJson, headers);
    }
}

Ok, this is a bit more source now. This is an abstract class, so we force an concrete implementation. It is a generic typed class, so you can use this implementation for different resources. We use springs IoC container to give us all we need. The LoadBalancerClient interface will be injected with ribbon, which uses its Eureka configuration to ask for services. As a result it picks one available instance and prepares the url. All this is done with the getUrl helper method. This method can be used to access the resource straight by passing it’s path like

1
getUrl("bars", 123);

So here comes the implementation for Bar Client

src/main/java/de.stytex.foobar.client/BarClient.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
public class BarClient extends AbstractMicroserviceClient<Bar> {
    public BarClient() {
        super("BAR");
    }

    /**
     * A reduced version of a Bar, with ignoring id on Jacksons serialization
     */
    @JsonIgnoreProperties({"id"})
    static class NewBar extends Bar {
        public NewBar(Bar base) {
            this.setValue(base.getValue());
        }
    }


    @Override
    public Collection<Bar> findAll() {
        return Arrays.asList(restTemplate.getForEntity(getUrl("bars"), Bar[].class).getBody());
    }

    @Override
    public Bar getOne(long id) {
        return restTemplate.getForObject(getUrl("bars", id),Bar.class);
    }

    @Override
    public Bar create(Bar object) throws JsonProcessingException {
        HttpEntity<String> entity = getJsonEntity(new NewBar(object));
        ResponseEntity<Bar> responseEntity = restTemplate.postForEntity(getUrl("bars"), entity, Bar.class);

        return responseEntity.getBody();
    }

    @Override
    public Bar update(Bar object) throws IOException {
        HttpEntity<String> entity = getJsonEntity(object);
        ResponseEntity<String> responseEntity = restTemplate.exchange(getUrl("bars"), HttpMethod.PUT, entity, String.class);

        return mapper.readValue(responseEntity.getBody(), Bar.class);
    }

    @Override
    public void delete(long id) {
        restTemplate.delete(getUrl("bars", id));
    }
}

Can we already test this?

No we can’t, because JHipster is configured to run with eureka disabled in tests. We need to comment this fields out in

src/test/resources/config/application.yml
1
2
3
#eureka:
#    client:
#        enabled: false

and there is no default bean for RestOperations, so we either just create a new RestTemplate inside AbstractMicroservicesClient or add a configuration like this:

src/main/de.stytex.foobar.config/RestTemplateConfiguration.java
1
2
3
4
5
6
7
@Configuration
public class RestTemplateConfiguration {
    @Bean
    public RestOperations restTemplate() {
        return new RestTemplate();
    }
}

Urgh, can we test it? Lets test!

If everything was wired up correctly, you still get a

org.springframework.web.client.HttpClientErrorException: 403 Forbidden

So, what is wrong? We are requesting the service api, which is secured by JWT tokens. So we must provide a token when consuming the bar resource. I will come back to this topic in my later part, since it is not that easy. For testing purposes we just can open up bars resource by modifying its SecurityConfiguration like this:

src/main/java/de.stytex.foobar.config/MicroservicesSecurityConfiguration.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
...
@Override
    public void configure(WebSecurity web) throws Exception {
        web.ignoring()
            .antMatchers(HttpMethod.OPTIONS, "/**")
            .antMatchers("/api/**")     //FOR TESTING!
            .antMatchers("/app/**/*.{js,html}")
            .antMatchers("/bower_components/**")
            .antMatchers("/i18n/**")
            .antMatchers("/content/**")
            .antMatchers("/swagger-ui/index.html")
            .antMatchers("/test/**")
            .antMatchers("/h2-console/**");
    }

    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http
            .csrf()
            .disable()
            .headers()
            .frameOptions()
            .disable()
        .and()
            .sessionManagement()
            .sessionCreationPolicy(SessionCreationPolicy.STATELESS)
        .and()
            .authorizeRequests()

            .antMatchers("/api/logs/**").hasAuthority(AuthoritiesConstants.ADMIN)
            .antMatchers("/api/**").permitAll() //FOR TESTING!
            //.antMatchers("/api/**").authenticated()
            .antMatchers("/metrics/**").permitAll()
...

And now the integration test should pass.

The good news are…

Using this pattern, each service inside the project can communicate directly with each other using a simple RestTemplate.

Is that all?

No! You also can directly add Hystrix circuit switchting easily here. Suppose your foo service experience a lot of timeouts when asking for bar service. In a real world scenario this could happen, because some bug on bar service shows up on massive traffic, which rapidly reaches the service in production. It causes the bar service to crash. This would usually provoke errors on the foo service and maybe also the gateway, so one service fail lead to the fail of the whole request chain. Now comes the second part: since foo service is down, no request reach bar service anymore, so it returns from crash. But this will let a whole wall of holded “bar requests”, which now reach bar at one time and the circle starts again.

Hystrix stores your results in a cache and use it, when the origin service is down. When the origin service returns from downtime, hystrix still will use the local cache for a while to put up traffic slowly on bar over time.

This is a great feature of spring cloud…but how to use it?

Just add @EnableCircuitBreaker on FooApp like this:

src/main/java/de.stytex.foobar/FooApp.java
1
2
3
4
5
6
7
@ComponentScan
@EnableAutoConfiguration(exclude = { MetricFilterAutoConfiguration.class, MetricRepositoryAutoConfiguration.class })
@EnableConfigurationProperties({ JHipsterProperties.class, LiquibaseProperties.class })
@EnableCircuitBreaker
@EnableEurekaClient
class FooApp {
...

and modify the bar client like this:

src/main/java/de.stytex.foobar.client/BarClient.java
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
@Component
public class BarClient extends AbstractMicroserviceClient<Bar> {

    Collection<Bar> barCache;

    public Collection<Bar> getBarCache() {
        return barCache;
    }

    public BarClient() {
        super("BAR");
    }

    /**
     * A reduced version of a Bar, with ignoring id on Jacksons serialization
     */
    @JsonIgnoreProperties({"id"})
    static class NewBar extends Bar {
        public NewBar(Bar base) {
            this.setValue(base.getValue());
        }
    }


    @Override

    @HystrixCommand(
        fallbackMethod = "getBarCache",
        commandProperties = {
            @HystrixProperty(name = "circuitBreaker.requestVolumeThreshold", value = "2")
        }
    )
    public Collection<Bar> findAll() {
        barCache = Arrays.asList(restTemplate.getForEntity(getUrl("bars"), Bar[].class).getBody());
        return barCache;
    }

    @Override
    @HystrixCommand(
        fallbackMethod = "getOneCache",
        commandProperties = {
            @HystrixProperty(name = "circuitBreaker.requestVolumeThreshold", value = "2")
        }
    )
    public Bar getOne(long id) {
        return restTemplate.getForObject(getUrl("bars", id),Bar.class);
    }

    public Bar getOneCache(long id) {
        return barCache.stream().filter(bar -> bar.getId() == id).findFirst().get();
    }
...

This is just a bit of code doing a huge work.

Following this pattern makes your microservice project as much decentralized as possible. The responsibility of the gateway is reduced, since it is not the central load balancer and circuit breaker, but only one in a row.

The bad news…

We had to turn off the security on bar services API to make this kind of communication possible. But this is of course a no go to keep our sensible resources open to the whole world, isn’t it?

So how to solve the security problem? Why it is actually a problem?

JHipster uses JWT tokens for authorization (not authentication!). Which means, every request should have a bearer access token inside a Authorization header. This JWT can be decrypted on the services side and we can extract the Principal, without querying the gateway, since the gateway is responsible for authentication.

The test just cannot pass, because the very initial request is fired inside the test. You could access some foo service endpoint via the angular frontend, which would add the access token inside the HTTP headers. Spring Cloud Netflix Zuul is able to get this token. So you could store it in a session scoped bean and add on your rest template on output. Then the tests still would fail, but in real world the requests would be authorized.

An other approach would be to make your RestTemplate been wrapped into a bean with this interface:

1
2
3
4
5
6
interface JHipsterAuthenticatedRest {
  public RestTemplate getTemplate();
  public boolean isAuthenticated();
  public void authenticate(String user, String password);
  public HttpHeaders getHeaders();
}

You would add the “system” users credentials or any other “machine” user, to let the service authenticate first, to get a own token. The getTemplate() method could ask isAuthenticated() first, then return a RestTemplate and prepared HttpHeaders for authorized calls, or authenticate before.

Not sweet, but a working solution..

Can we do that better?

Yes, we can do it with:

Part 3: OAuth2

Using JWT tokens is a great idea, but the guys reinvented the wheel a bit by implementing a specific authorization mechanism. I can absolutelly understand, why they did it. Some months ago I had that decision, too.

The answer of the question:

What is a proper way to manage authentication and authorization inside microservices?

When I first faced OAuth2, it looked to complicated and complex for a simple thing I wanted. But security inside services can get quite complex as well, so using OAuth2 perfectly fits the needs.

Which needs? What does the built in authentication cannot what OAuth2 can?

In my second approach to apply security on the services I mentioned usage of a machine user. The current implementation does not support distinction between “user access” and “machine access”.

There are a lot use cases, where to use it. For example: trust machines, don’t trust users. So you allow a machine to do everything, and apply just privileges on user sessions. If you don’t like this, you can apply the authorities from user session to the machine call, getting the same roles as the user has. You can use “scopes” to even distinguish between users privilges, called “authorities”, and machine privileges, called “scopes”

Performing service calls could be done using OAuth2 client credential grant type, so we never need a user authentication, to get secure priveleged by the resource servers.

Let me say, this is not a trivial step anymore, and this article is growing a bit huge right now. I am considering to implement a UAA using JHipster and spring-cloud-security and spring-cloud-oauth2, as I did in my last tutorial.

update: This “consideration” has an happy end: read on on my article about secure service communication

Conclusion

Yes, it was sayed a lot of times: JHipster is a great job! There is a lot of work you can save just by generating. This is very interesting to me, and I hope this article gives some a good introduction into some practices.

Please be free to comment, criticize or ask questions in the comments. If some one likes this work, my motivation of implementing an authorization service will grow exponential :)

Have a great week

Resources

Securing Spring Cloud Microservices With OAuth2

| Comments

update on 2016-10-19: more on JWT, fixed typos, clarified some aspects

From Zero to OAuth2 in Spring cloud

Today I am presenting hours of research about a (apparently) simple question: “How can I maintain security in my microservices architecture?”. The task is to enable a simple but mighty possibility to secure spring cloud services down to method invocation level, having a central point of where users and authorities can be assigned.

To achieve this as efficient as possible, OAuth2 is the solution.

In this article we are going to implement an authorization server, holding user authorities and client information, and a resource service with protected resources, using Spring OAuth2 and JSON Web Tokens (JWT). I will demonstrate, how the resource server can host a RESTful resource, having different security levels, which is defined in example authorities “FOO_READ” and “FOO_WRITE”.

The implementation can be downloaded and tested on my GitHub Repository.

Since I am really new to Spring and Spring Cloud including all its concepts, this was a quite hard way of research. This might sound weird, but at the beginning I couldn’t get, why they are all talking about Facebook/GitHub authentication in context of how to secure internal data. It was leading to an obvious question:

Why OAuth2?

Despite microservices are a rather new topic in modern software development, OAuth2 is a well known authorization technology. It is widely used, to give web applications developers access to users data at Google/Facebook/GitHub directly from the foreign services in a secure way. But before I explain more details, I will first refocus, what we initially want to achieve: cloud security.

So, what could we do generally, to gain/forbid access to resources inside the cloud to users? Let’s take some dumb stupid ways.

We could secure the edge server, assuming all the access to our data will go through it. This is nearly the same as you would do with classic Spring MVC Security. But there is no way of method security and - what is the most important - your data is insecure from inside attacks.

Other way: We share the user credential database for all services and authenticate the user on each service before access. Sounds somehow really stupid, but it’s actually a working approach with all the spring security features available.

Better way: The user authenticates on a authorization service, which maps the user session to a token. Any further API call to the resource services must provide this token. The services are able to recognize the provided token and ask the authorization service, which authorities this token grants, and who is the owner of this token.

This sounds like a good solution, doesn’t it? But what’s about secure token transmission? How to distinguish between access from a user and access from another service (and this is also something we could need!)

So this leads us to: OAuth2. Accessing sensible data from Facebook/Google is pretty much the same as accessing protected data from the own backend. Since they are working for some years on this solution, we can apply this battleground approved solution for our needs.

How OAuth2 works

Implementation of OAuth2 in Spring is quite easy, when you understand the concept of OAuth2. Let us describe the scenario “AwesomeApp wants Peters profile data from facebook for Peters profile in AwesomeApp”

OAuth2 defines 4 roles in this process:

  • Resource Owner - this is Peter
  • Resource Server - this is Facebook
  • Authorization Server - this is also Facebook, because it knows Peter and its Session and data
  • Client - the AwesomeApp

When Peter tries to sign up with Facebook Login, Awesome App redirects him to FBs authorization server. This knows about Peter, whether he is logged in, and even if Peter already was here before and already approved it’s data. When Peter visits this page for the first time, he has to allow AwesomeApp to access his email and profile data. These two sources are defined as scopes, which AwesomeApp defines in Facebooks AwesomeApp Facebook-app. The developer of AwesomeApp provided to ask exactly for these two permissions.

Peter gives his permission, and is redirected to AwesomeApp with a access token. AwesomeApp then uses the access token to retrieve Peters email and profile data directly from Facebook, with no need to authenticate with Peters credentials. More than that, each time Peter signs in to AwesomeApp, he visit the authorization page again, which already knows his decision and directly responds with a token.

So far…but how to apply this on our brutal real world? In springs OAuth2 implementation as well as in all examples, they are talking about clients and scope.

Are OAuths “clients and scopes” the same as our classical “user and authorities”?

Do I have to map authorities to scopes or users to clients?

Why do I need clients?

You are maybe trying to map the roles from the first scenario to enterprise case. This is tricky!

Second scenario: Any kind of application provides user login. This login results in a exchange of user credentials to access token. All further API calls provide this token in its HTTP header. The services inside are asking the authorization server to check the token and grant access. In the other direction, the services may request data from another service, granted by authorization service using a different client with different permissions than the user, which initiated the request.

As you see, the four OAuth2 roles depend of the direction in which data is requested. For asking protected business data from resource server, the authorization server is what it is, the resource servers also, the application is the client and the service holding the permissions (often the same as authorization server), is the owner. When asking the users data, the authorization service becomes a resource server, and resource server the client.

Scopes and Roles, Clients and Users

With OAuth2 you can define, which application (web, mobile, desktop, additional website) can access which resources. So there is one dimension, where we have to decide which user can access which data, but also which application or service, can access which resource. In other words: scopes are access controlling which endpoints are visible to clients, and authorities filter the data to the user based on his permissions.

In a web shop, the frontend may act as an client, having access to products, orders and customers, but the backend also about logistics, contracts and more, independent of the users authorities. In the other way, a user may have potential access to a service but no access to all its data, because he is using a web application, where other users are permitted to access while he is not. This service-to-service access, is our other dimension. If you are familiar with math, I can say: the client-scope-relation is linear independent to the user-authority-relation in OAuth2.

Why JWT?

OAuth2 is not about, how the token is looking like and where it is stored. So one approach is, to generate random strings and save token related data to these string in a store. Over a token endpoint, other services may ask something “is this token valid, and which permissions does it grant?”. This is the “user info URL” approach, where the authorization server is turning into a resource server for the user info endpoint.

As we talking about microservices, we need a way to replicate such a token store, to make the authorization server scalable. Despite this is a tricky task, there is one more issue with user info URL. For each request against a resource microservice containing a token in the header, the service will perform another request to authorization server to check the token. So without caching tricks, we got twice more request we really need, only for security. So both, scaling token store and token request are affecting the scalability of our architecture heavily. This is where JWT (pronounced as “jot”) come into play.

In short, the answer of that user info URL request, containing info about the OAuth client, optionally the user with its authorities and the granted scope, is serialized into JSON first, encoded with base64 and finally signed using a token. The result is a so called JSON Webtoken, which we use as an access token directly. When these JWTs are signed using RSA, the authorization server first signs it with the RSA private key, assuming every resource server will have a public key. When this token is passed via HTTP header, the resource servers just have to take this JWT, verify it was really signed by the proper private key (meaning this token is coming from authorization server), and instead of asking user info, deserializing the JSON content into a OAuth2Authentication, establishing a SecurityContext based on this.

Using JWT provides a simple way of transmitting hard to fake tokens, containing the permissions and user data in the access token string. Since all the data is already inside, there neither is a need to maintain a token store, nor the resource servers must ask authorization for token checks.

So, using JWT makes OAuth2 available to microservices, without affecting the architectures scalability.

implementing the authorization service

For the later token exchange we first generate a JWT token keystore

1
keytool -genkeypair -alias jwt -keyalg RSA -dname "CN=jwt, L=Berlin, S=Berlin, C=DE" -keypass mySecretKey -keystore jwt.jks -storepass mySecretKey

Then execute

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$ keytool -list -rfc --keystore jwt.jks | openssl x509 -inform pem -pubkey
Keystore-Kennwort eingeben:  mySecretKey
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwR84LFHwnK5GXErnwkmD
mPOJl4CSTtYXCqmCtlbF+5qVOosu0YsM2DsrC9O2gun6wVFKkWYiMoBSjsNMSI3Z
w5JYgh+ldHvA+MIex2QXfOZx920M1fPUiuUPgmnTFS+Z3lmK3/T6jJnmciUPY1pe
h4MXL6YzeI0q4W9xNBBeKT6FDGpduc0FC3OlXHfLbVOThKmAUpAWFDwf9/uUA//l
3PLchmV6VwTcUaaHp5W8Af/GU4lPGZbTAqOxzB9ukisPFuO1DikacPhrOQgdxtqk
LciRTa884uQnkFwSguOEUYf3ni8GNRJauIuW0rVXhMOs78pKvCKmo53M0tqeC6ul
+QIDAQAB
-----END PUBLIC KEY-----
-----BEGIN CERTIFICATE-----
MIIDGTCCAgGgAwIBAgIEOkszIDANBgkqhkiG9w0BAQsFADA9MQswCQYDVQQGEwJE
RTEPMA0GA1UECBMGQmVybGluMQ8wDQYDVQQHEwZCZXJsaW4xDDAKBgNVBAMTA2p3
dDAeFw0xNjAyMDExNzQwMTlaFw0xNjA1MDExNzQwMTlaMD0xCzAJBgNVBAYTAkRF
MQ8wDQYDVQQIEwZCZXJsaW4xDzANBgNVBAcTBkJlcmxpbjEMMAoGA1UEAxMDand0
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwR84LFHwnK5GXErnwkmD
mPOJl4CSTtYXCqmCtlbF+5qVOosu0YsM2DsrC9O2gun6wVFKkWYiMoBSjsNMSI3Z
w5JYgh+ldHvA+MIex2QXfOZx920M1fPUiuUPgmnTFS+Z3lmK3/T6jJnmciUPY1pe
h4MXL6YzeI0q4W9xNBBeKT6FDGpduc0FC3OlXHfLbVOThKmAUpAWFDwf9/uUA//l
3PLchmV6VwTcUaaHp5W8Af/GU4lPGZbTAqOxzB9ukisPFuO1DikacPhrOQgdxtqk
LciRTa884uQnkFwSguOEUYf3ni8GNRJauIuW0rVXhMOs78pKvCKmo53M0tqeC6ul
+QIDAQABoyEwHzAdBgNVHQ4EFgQUxebRVICNPg65T2RgrOe2J5qDMXMwDQYJKoZI
hvcNAQELBQADggEBAAHBF3JQ2ZsEjuYYwU5cp9BzBJoTQyChF37AA76EorrcSeqo
Rui1dUIfXbImIOZ5PBNk34IFWROTwpw80zBCZQ7NQ81ITzuhsxjbX7Wxj6iCq3u9
TDN+IxiaZMvJ2PDfeRqr93HOwMTMttxyW4KVa3geQ+yMMSZrxagEpqMA1Fviqa6T
5u8DNqfXQ8Hg+yG2bMNQs6GleAFkRprkHjR6yY7ehmIVMZ7iBkkXh8IO8fKy2WNK
uWa+DO2lXJj1W7HLXeaeT0twAqwyoNj2/pxMuv/JrTlNkhcUTmP+UBAJZih0KSGD
9TSKs5HlBGsIUpILuauNzZk1VS2RCyVtD1zf7vM=
-----END CERTIFICATE-----

You will be prompted to enter your secret key. Copy the public key (including the dashed lines) into a file named “public.cert”.

Copy this file to src/main/resources.

the plan

In this example we will define a resource “foo”, which can be read with authority FOO_READ and can be written with FOO_WRITE.

On the authentication service we have the user “reader”, who can read a foo, and a “writer”. To make the resource server accessible by a web application, we define a “web_app” client.

First we define our gradle file. Use start.spring.io to generate a gradle boot application with “Web” dependency.

this gradle file will be used for resource server also!

Now we adjust the dependencies to this

1
2
3
4
5
6
dependencies {
  //...
  compile('org.springframework.security.oauth:spring-security-oauth2')
  compile('org.springframework.security:spring-security-jwt')
  //..
}

To make the configuration non-confusing, I will use separate configurations instead of mixing all in inside the Application class. In both examples they are as they roll out from spring initialzr.

OAuth2Configuration

We begin with the following configuration

1
2
3
4
5
@Configuration
@EnableAuthorizationServer
public class OAuth2Configuration extends AuthorizationServerConfigurerAdapter {
    //..
}

To define a default spring configuration, and enable the current application as an OAuth2 authorization server.

We inherit the class from AuthorizationServerConfigurerAdapter, to configure the details.

src/main/java/package/config/OAuth2Configuration.java
1
2
3
4
5
6
7
8
9
    @Override
    public void configure(ClientDetailsServiceConfigurer clients) throws Exception {
        clients.inMemory()
                .withClient("web_app")
                .scopes("FOO")
                .autoApprove(true)
                .authorities("FOO_READ", "FOO_WRITE")
                .authorizedGrantTypes("implicit","refresh_token", "password", "authorization_code");
    }

We assume FOO as resource access identity (so we can check this with #oauth2.hasScope(‘FOO’) to apply client access permission), auto approve for the scope for code authorization and pass the authorities for resource server.

Now we configure the OAuth2 endpoints to adjust the authentication manager (which will represent the web-security users), and JWT token store configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
@Override
public void configure(AuthorizationServerEndpointsConfigurer endpoints) throws Exception {
    endpoints.tokenStore(tokenStore()).tokenEnhancer(jwtTokenEnhancer()).authenticationManager(authenticationManager);
}

@Autowired
@Qualifier("authenticationManagerBean")
private AuthenticationManager authenticationManager;

@Bean
public TokenStore tokenStore() {
    return new JwtTokenStore(jwtTokenEnhancer());
}

@Bean
protected JwtAccessTokenConverter jwtTokenEnhancer() {
    KeyStoreKeyFactory keyStoreKeyFactory = new KeyStoreKeyFactory(new ClassPathResource("jwt.jks"), "mySecretKey".toCharArray());
    JwtAccessTokenConverter converter = new JwtAccessTokenConverter();
    converter.setKeyPair(keyStoreKeyFactory.getKeyPair("jwt"));
    return converter;
}

We define another configuration for the web security.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
@Configuration
class WebSecurityConfig extends WebSecurityConfigurerAdapter {

    @Override
    @Bean
    public AuthenticationManager    authenticationManagerBean() throws Exception {
        return super.authenticationManagerBean();
    }

    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http
            .csrf().disable()
                .exceptionHandling()
                .authenticationEntryPoint((request, response, authException) -> response.sendError(HttpServletResponse.SC_UNAUTHORIZED))
            .and()
                .authorizeRequests()
                .antMatchers("/**").authenticated()
            .and()
                .httpBasic();
    }

    @Override
    protected void configure(AuthenticationManagerBuilder auth) throws Exception {
        auth.inMemoryAuthentication()
                .withUser("reader")
                .password("reader")
                .authorities("FOO_READ")
                .and()
                .withUser("writer")
                .password("writer")
                .authorities("FOO_READ", "FOO_WRITE");
    }
}

Note that there is no “@EnableWebSecurity”, because it’s automatically applied through @EnableAuthorizationServer. We also declare the authenticationManagerBean as a bean, which will be injected in the OAuth configuration above. HttpSecurity and user definition is quite straight forward, and can be implemented in various ways like UserDetailsService.

We configure the application to run on port 9999 for now.

indepth: we are using OAuth2 as authentication protocol, too. This approach is similar to OpenID connect, which is also a standard authentication protocol over OAuth2, more relying to public identity providers, such as Google, GitHub etc.

get access tokens

I will show how to get access token directly via username/password. There are other authorization types such as authorization code, implicit, client credentials.

To get a token, just

1
2
$ curl -XPOST "web_app:@localhost:9999/oauth/token" -d "grant_type=password&username=reader&password=reader"
{"access_token":"eyJhbGciOiJSUzI1NiJ9.eyJleHAiOjE0NTQ0MDA3MzQsInVzZXJfbmFtZSI6InJlYWRlciIsImF1dGhvcml0aWVzIjpbIkZPT19SRUFEIl0sImp0aSI6IjU1MWI4MTY4LTMwZmItNDZlNS1iMzJlLTc4ODRjNjJlNzZlYiIsImNsaWVudF9pZCI6IndlYl9hcHAiLCJzY29wZSI6WyJGT08iXX0.cKcgkLcbBECSWlz5fllb5V0EkfvrIq6RxjId34mNvhifS5bseQD5c8SlsQ_MvLf6unmosIHT_WL9TP56UUPX5TFrQpT09c2RPvnyhKD5PLlrf9o2RAAL5xS1yQqAWoSoNlx73m8cs8xOjIEix3mthNzEDlLYgsBbQci0ZWBCQHwnRE3OW4oykm4YH5X59X-8Juq1enztbdcjcyt4aFQOG7KVstW5M0MN3y3MMD4O9QgsatzBWDL2lPoazhKuYkR9LcoBZrKF_WzQgwolMhK_ousOxLEHNbKoWxOWJPJnayi6NW8o_2SlkTs7ykDh_GEGOSswpMGhkw98DI5dwFcTQg","token_type":"bearer","refresh_token":"eyJhbGciOiJSUzI1NiJ9.eyJ1c2VyX25hbWUiOiJyZWFkZXIiLCJzY29wZSI6WyJGT08iXSwiYXRpIjoiNTUxYjgxNjgtMzBmYi00NmU1LWIzMmUtNzg4NGM2MmU3NmViIiwiZXhwIjoxNDU2OTQ5NTM0LCJhdXRob3JpdGllcyI6WyJGT09fUkVBRCJdLCJqdGkiOiI0MTBlZWNjMS01NTRiLTQ0OGQtOGUyOC1iMGE3NTg5N2JlNzMiLCJjbGllbnRfaWQiOiJ3ZWJfYXBwIn0.Rw5ASYQjsJtPfWMMNIQ1TQA53VAqMSoDze8RHzbdRgXkn_BS-Qc84rTNg5deICL_Qdz6D3OtRL2pXgAkOn6ImCDJGaKcroZscZ1Mpy7lmBbsBf1pOolqOsXbCItOPh7h8CpB41ZipTeq-v_-5LQ7wNqwMTOzW_zL8On7bc0ZLF66PY-HK8BlFYUaiJRdJqP1PjfCh8hmOUMYnX8slQcdVMP4V1m6ZzdVFuhywKi3LD6tzrU-q1s2FEUVIpOCKJ6pKv9ts6tSK_lcjLjFO0rRzjTSdtywKE5Gc1rvC4BJALN_ZOn_uiskzo8IIztDUefZJV5OCAZ41igDUXbJHb1NSA","expires_in":43199,"scope":"FOO","jti":"551b8168-30fb-46e5-b32e-7884c62e76eb"}

implementing the resource server

We generate another Spring Boot application with Web starter and take the same gradle dependencies we used for the authorization server before. Copy the public.cert file into src/main/resources.

First, we have to implement the resource itself:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
@RestController
@RequestMapping("/foo")
public class WebController {

    @GetMapping
    public String readFoo() {
        return "read foo " + UUID.randomUUID().toString();
    }

    @PostMapping
    public String writeFoo() {
        return "write foo " + UUID.randomUUID().toString();
    }
}

This is a simple controller. I use UUID random strings to be sure every response will be unique (for cache things).

Since we use JWT tokens, we have to configure the token store and token converter for JWT.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
@Configuration
public class JwtConfiguration {
    @Autowired
    JwtAccessTokenConverter jwtAccessTokenConverter;


    @Bean
    @Qualifier("tokenStore")
    public TokenStore tokenStore() {
        return new JwtTokenStore(jwtAccessTokenConverter);
    }

    @Bean
    protected JwtAccessTokenConverter jwtTokenEnhancer() {
        JwtAccessTokenConverter converter =  new JwtAccessTokenConverter();
        Resource resource = new ClassPathResource("public.cert");
        String publicKey = null;
        try {
            publicKey = new String(FileCopyUtils.copyToByteArray(resource.getInputStream()));
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
        converter.setVerifierKey(publicKey);
        return converter;
    }
}

Now lets configure the resource server security:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
@Configuration
@EnableResourceServer
public class ResourceServerConfiguration extends ResourceServerConfigurerAdapter{

    @Override
    public void configure(HttpSecurity http) throws Exception {
        http
                .csrf().disable()
                .authorizeRequests()
                .antMatchers("/**").authenticated()
                .antMatchers(HttpMethod.GET, "/foo").hasAuthority("FOO_READ");
                //.antMatchers(HttpMethod.POST, "/foo").hasAuthority("FOO_WRITE");
                //you can implement it like this, but I show method invocation security on write
    }


    @Override
    public void configure(ResourceServerSecurityConfigurer resources) throws Exception {
        resources.resourceId("foo").tokenStore(tokenStore);
    }

    @Autowired
    TokenStore tokenStore;

    @Autowired
    JwtAccessTokenConverter tokenConverter;
}

Note I commented the access rule for write requests on foo. I will show how to secure this via method invocation level security.

adding method security

To enable method security we just create a configuration like this:

1
2
3
4
@Configuration
@EnableGlobalMethodSecurity(prePostEnabled = true)
public class GlobalMethodSecurityConfiguration {
}

and change the writeFoo method in our rest controller:

1
2
3
4
5
@PreAuthorize("hasAuthority('FOO_WRITE')")
@PostMapping
public String writeFoo() {
    return "write foo " + UUID.randomUUID().toString();
}

You may ask why I didn’t use “secureEnabled = true” and the @Secured annotation. Sadly, this doesn’t work at the moment.

We run this application on port 9090.

testing

Now copy the access token from our last curl command and create a local variable TOKEN:

1
$ TOKEN=eyJhbGciOiJSUzI1NiJ9.eyJleHAiOjE0NTQ0MDA3MzQsInVzZXJfbmFtZSI6InJlYWRlciIsImF1dGhvcml0aWVzIjpbIkZPT19SRUFEIl0sImp0aSI6IjU1MWI4MTY4LTMwZmItNDZlNS1iMzJlLTc4ODRjNjJlNzZlYiIsImNsaWVudF9pZCI6IndlYl9hcHAiLCJzY29wZSI6WyJGT08iXX0.cKcgkLcbBECSWlz5fllb5V0EkfvrIq6RxjId34mNvhifS5bseQD5c8SlsQ_MvLf6unmosIHT_WL9TP56UUPX5TFrQpT09c2RPvnyhKD5PLlrf9o2RAAL5xS1yQqAWoSoNlx73m8cs8xOjIEix3mthNzEDlLYgsBbQci0ZWBCQHwnRE3OW4oykm4YH5X59X-8Juq1enztbdcjcyt4aFQOG7KVstW5M0MN3y3MMD4O9QgsatzBWDL2lPoazhKuYkR9LcoBZrKF_WzQgwolMhK_ousOxLEHNbKoWxOWJPJnayi6NW8o_2SlkTs7ykDh_GEGOSswpMGhkw98DI5dwFcTQg

and peform a call to read the foo resource

1
$ curl -H "Authorization: Bearer $TOKEN" "localhost:9090/foo"

and you should get a positive result, while

1
$ curl -XPOST -H "Authorization: Bearer $TOKEN" "localhost:9090/foo"

should result in access denied. To get access to write on foo, we must get a token as “writer”.

Conclusion

This should be a very brief introduction into how you can use OAuth2 in spring cloud microservices. If I explained something wrong, please be free to correct me. I am open to any kind of critics :)

The most important is, this sample works so you can try it out and change it in a way you need. May someone find this article useful :)

More resources

  • JHipster UAA docs for using a working setup using this basics to secure spring cloud microservices
  • JHiosoter UAA demo setup: a set of several JHipster microservices, demonstrating both user-to-service (Angular client) and service-to-service authorization (Feign Clients)

Have a great week.

Log Analysis With ELK Stack in Spring Cloud

| Comments

Introduction

Today I am writing about log aggregation and analysis inside a Spring Cloud. When working with a lot of spring cloud services, you get a lot of logs out of the box. When analyzing one big log file coming from a monolith is something you can handle easely, doing the same for maybe 100 unique services is not.

To solve this in a elegant way, we must include an installation of a node including a database just for log, some kind of port for accepting incoming logs and a clear backend with capabilities of performing complex search over the entire log base.

Elasticsearch + Logstash + Kibana

Elasticsearch is a modern document based database, built on top of Apache Lucene, which is powerfull in searches on millions of records and is cluster scalable out of the box via REST.

Logstash is a tool wiring log streams our sources and saving them into elastichsearch. The very basic task logstash can be used, is to define a shared volume for all docker container and placing the logs there. Logstash allows to apply different filter on your input, to define how your logs are parsed. This is useful, when collecting logs from different sources, but I will only talk about letting all the services sending their logs to logstash directly in JSON format, to keep the configuration simple.

Kibana actually is a backend offering several tools for log analysis.

How to make it work

To see a working example with docker-compose, download the sources from my GitHub. You are also free to

1
$ git clone https://github.com/xetys/microservices-example.git

and contribute.

What we are going to do

Logstash will run with UDP Port 5000 open waiting for log in JSON format. So we have to tell all the services to send their logs. We will use Logback with an UDP appender to accomplish this. We also have to have the ELK-instance avaible to all services.

So we start with configuring the ELK-Stack with a logstash config:

1
2
3
4
5
6
7
8
9
10
11
input {
    udp {
       port => 5000
       codec => json
    }
}

output {
  elasticsearch { protocol => "http" }
  stdout { codec => rubydebug }
}

This is quite easy, because with codec json logstash automatically knows how to deal with the input. So we won’t define any filters. We tell also the output to elastic search, which logstash automatically finds since it’s on the same machine.

This file has to be applied in this Dockerfile

1
2
3
4
FROM willdurand/elk


COPY logstash-spring-cloud.conf /etc/logstash/logstash-spring-cloud.conf

which is also quite easy. Logstash will look for config files in /etc/logstash and waiting for logs incoming on port 5000

This container also starts a Kibana instance on port 80, but we want to have it on its usual port 8200 exposed to the host. So the new docker-compose.yml will look like this now:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
elk:
  build: ./elk
  ports:
    - "9200:9200"
    - "8200:80"
    - "5000:5000/udp"
eureka:
  build: ./eureka
  ports:
    - "8761:8761"
  links:
    - elk
simple1:
  build: ./simple1
  links:
   - eureka
   - elk
railsdemo:
  build: ./RailsEurekaClient
  links:
   - eureka
   - elk
  ports:
   - "3000:3000"
simple2:
  build: ./simple2
  links:
   - eureka
   - elk
zuul:
  build: ./zuul
  links:
   - eureka
   - elk
  ports:
    - "8080:8080"

Note that every instance now links elk. Exposing port 9200 is optinally, if you want to access elasticsearch also.

Appending logs to logstash

So if we start the cloud now, we will have a fully running ELK-Stack without collecting any logs.

To make this happen, we just have to add

1
compile('net.logstash.logback:logstash-logback-encoder:3.5')

to our gradle depencies in each service, and also a logback configuration:

src/main/resources/logback.xml
1
2
3
4
5
6
7
8
9
10
11
12
<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="false">
    <include resource="org/springframework/boot/logging/logback/base.xml" />
    <appender name="UDP2LOGSTASH" class="net.logstash.logback.appender.LogstashSocketAppender">
        <host>elk</host>
        <port>5000</port>
    </appender>

    <root level="INFO">
        <appender-ref ref="UDP2LOGSTASH"/>
    </root>
</configuration>

Now bootRepackage all the services and

1
2
$ docker-compose build
$ docker-compose up -d # start all containers daemonized

And you are done!

Conclusion

You know can acces Kibana on http://localhost:8200/ and explode its features. If there are more different services to send to logstash, you may use a shared location to fetch logfiles, passing them through filters.

Have a great weekend!

How to Integrate Any Webapp Into Spring Cloud Using Sidecar Applications

| Comments

Introduction

You may were looking for:

  • How can non-JVM application be plugged into a spring cloud microservice infrastructure?
  • How can I integrate my Rails / NodeJS / Express / PHP application into a Spring Cloud?
  • …and also give the (maybe legacy) applications all the feature a Spring Boot Service inside the cloud have
  • …without big changes on the “outside” apps dependencies or even code?

So this what sidecar applications are for: integrating web applications outside the cloud infrastructure accessible in both directions.

What are Sidecar Applications?

Inside the Spring Cloud each Spring Boot application gains its power through dependencies and annotation magic performed on the classes. So things, which usualy consist of a lot of implementations behind, are adjustable through one annotation. But inside a microservice cloud there is also a need of enabling usage of other applications written in different frameworks, languages or even running on other plattforms. One central argument for using microservice architectures is, that there are no limitations (ideally) for which tools you use to build web applications.

But inside a distributed system, this is not trivial. One possible way to achive this to build or find libraries written in the native language/framework of the outside application to communicate with the cloud, implementing features as Eureka service registration and discovery, Hystric curcuit breaking and so on. This actually, would be a lot of work…

Remember inversion of control philosophy: “Can’t do this just someone else for me?”

It’s time for one more service, exactly for registrating your outside application with your service discovery (I will use Eureka in the examples). There is a lot more you can do with sidecar applications to wire the cloud features. I will focus on the initial setup for now.

Spring Cloud Netflix Sidecar

This feature inside Spring Clouds toolset is inspired by Netflix Prana. A Spring Sidecar application runs on the same host where your outside application and frequently checks the health of your application. The sidecar application registers itself on service discovery and is forwarding calls from cloud to the outside application by a defined sidecar application name. From the view of the outside application, you can access the sidecar over its port to get the registered service instances inside the cloud.

Playing arround with a working example

To start, just

1
$ git clone https://github.com/xetys/microservices-sidecar-example

This repo includes a complete spring cloud microservice infrastructure with Eureka Service Discovery, Zuul Edgeserver, two simple REST services for exposing random numbers as resource, and an Ruby on Rails application with no function, to be integrated with the cloud.

Your machine needs to fit the follow requirements to run the cloud:

  • JDK 8
  • Oracle VirtualBox
  • docker
  • docker compose
  • docker machine

Preparation

1
./gradlew bootRepackage

What? This builds all sub modules and creates a bundle of jars to run on the docker containers.

Note: If the above command results in an error with finding gradles main wrapper class, just apply the following command

1
$ /path/to/gradle/bin/gradle wrapper

Why? The project root consists of a docker-compose.yml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
eureka:
  build: ./eureka
  ports:
    - "8761:8761"
simple1:
  build: ./simple1
  links:
   - eureka
railsdemo:
  build: ./RailsApplication
  links:
   - eureka
  ports:
   - "3000:3000"
   - "9090:9090"
simple2:
  build: ./simple2
  links:
   - eureka
zuul:
  build: ./zuul
  links:
   - eureka
  ports:
    - "8080:8080"

This configuration expects to have all the subdirectories listed there containing a Dockerfile, which expects presence of a jar file (excepting the rails application) like gradles “bootRepackage” task generates it. All the spring containers use ewolff/docker-java repository, containing a alpine with java 8 on it, and rails runs inside my xetys/rails-java container, which is inherited from the official rails docker container with an applied JRE 8.

Next:

1
$ ./docker_create.sh

What? This script first creates a new virtual machine with boot2docker on it, then executing a

1
$ docker-compose build

automatically, to build all docker containers.

I have automated this part dirty, to make this example easely run. In fact I am really new to this topic at this time, this is not the best way to achieve this. Gradle has some docker plugins which may things I do more elegant and efficient. But you can take a look inside this script to understand, how the cloud ist setted up for development.

Next:

1
$ docker-compose up -d

This finally starts the application! The result should be:

OMG! Nothing works?! No reaction? Error pages?!? HELP!!!

Be patient! Dependend on how good your machine is, it takes some time to boot up the entire cloud. On my Dell XPS 13 it takes up to 5 minutes…so don’t CTRL+C in panic when error occurs at the very beginning.

How it works

First, we should create a sidecar application. This is done by applying ‘org.springframework.cloud:spring-cloud-netflix-sidecar’ dependency to our build gradle and annotate your Application class like this:

SidecarApplication.java
1
2
3
4
5
6
7
@SpringBootApplication
@EnableSidecar
public class SidecarApplication {
    public static void main(String[] args) {
        SpringApplication.run(SidecarApplication.class, args);
    }
}

Then we need to configure it properly:

application.yml
1
2
3
4
5
6
7
8
9
10
11
server:
  port: 9090

spring:
  application:
    name: ${side-app-name}

sidecar:
  port: ${port:3000}
  health-uri: http://localhost:${sidecar.port}/${health-uri:health.json}
  home-page-uri: http://localhost:${sidecar.port}/

What? This configuration implies, you must provide at least a side-app-name when starting the sidecar. The avaible startup parameters are now:

  • –side-app-name , the name which will appear for service discovery
  • –port , the port of the outside application
  • –health-uri , a URI accessible from the sidecar intending, the outside app is still up

You can add more configurable fields in this file if you want. But for our rails application, this is enough to start a sidecar with

1
$ java -jar sidecar-1.0.jar --side-app-name=my-rails-app

To contact the clouds service discovery via eureka, we have to provide this configuration also:

bootstrap.yml
1
2
3
4
5
6
7
8
9
10
11
12
spring:
  cloud:
    config:
      enabled: false

eureka:
  client:
    serviceUrl:
      defaultZone: ${eureka-url:http://localhost:8761/eureka/}
  instance:
    lease-renewal-interval-in-seconds: ${eureka-interval:5}
    prefer-ip-address: true

Why? As you can see, there is something like a health check configured. This tells you, your application has to contain a route (/health.json per default or as configured in application.yml) where the response should look like:

1
2
3
{
   "status": "UP"
}

This is quite straight forward by defining a new controller action in rails application controller:

application_controller.rb
1
2
3
4
5
  def health
    respond_to do |format|
      format.json { render json: {status: 'UP'}}
    end
  end

and wire it

routes.rb
1
get 'health' => 'application#health'

Wire up!

The rails application is started up on a docker container with rails and JRE running on it. Directly after the startup of our rails application, we also start a sidecar next to it. The jar file was previosly copied in docker_create.sh.

Well, there we are: Our “sidecarred” rails application appears as “CLOUD-RAILS” in Eureka and is avaible on http://localhost:8080/cloud-rails/ with Zuuls default routing, as soon everything has synchronized well.

Have a good day, hope this article helps someone!

Blog Launch and Topic Anouncement

| Comments

Blog Launch

Hello Folks,

so today I - finally - launched this blog. First I was going to code it on my own, just for practice, but maintaining a blog system isn’t something I want to spend my time with. Instead I was looking for a solid solution, which fits my needs, and is not auto-hackable as WordPress.

So, first: sorry for my poor english, I am not a native. And then: I am going to write about general topics about development in many ways, because I feel I can someone tell things, I had to discover on my own.

Upcomming topics

Writing about nothing, is not my aim, so here are the core topics for the next time:

  • How to evolve from a script developper to enterprise level with Java
  • How to start through with microservices using Spring Cloud and NetflixOSS
  • Something else I want to tell the world :)

More information

More information about who I am, the recent posts I am going to publish later in pages or blog posts.

How to Make Decisions?

| Comments

I was stucked on the question: “What do I going todo now? SW:ToR or some music creation?”.

I just couldn’t decide. But I assumed, I had wish to play some SW:ToR PvPs to about 70%, and I would like to drop a 7/3 coin instead of a classic 1/1 coin.

A bit of quick JavaScript helped me through:

1
decide = function (x) { return Math.random() * 100 < x; }

This should return true with a probality of x %….

I was quite sure, the random number must be less then x, but did some tests to prove it:

1
2
n=100;l = []; for(i=0;i<n;i++) l[i] = decide(70); s=0; for(i=0;i<n;i++) s+=l[i]; s / n
=> 0.64
1
2
3
4
n=1000;l = []; for(i=0;i<n;i++) l[i] = decide(70); s=0; for(i=0;i<n;i++) s+=l[i]; s / n
=> 0.716
n=1000;l = []; for(i=0;i<n;i++) l[i] = decide(70); s=0; for(i=0;i<n;i++) s+=l[i]; s / n
=> 0.685

and the big ones :D

1
2
3
4
5
6
n=10000;l = []; for(i=0;i<n;i++) l[i] = decide(70); s=0; for(i=0;i<n;i++) s+=l[i]; s / n
=> 0.7039
n=10000;l = []; for(i=0;i<n;i++) l[i] = decide(70); s=0; for(i=0;i<n;i++) s+=l[i]; s / n
=> 0.6986
n=10000;l = []; for(i=0;i<n;i++) l[i] = decide(70); s=0; for(i=0;i<n;i++) s+=l[i]; s / n
=> 0.6971

And for fairness, my very first result of decide(70) is qualified for my final decision. Happily, it was

1
2
3
4
decide = function (x) { return Math.random() * 100 < x; }
decide(x)
decide(70)
true

:D

P.S.:

You can sometimes take a look at my twitch channel

GIT vs Subversion

| Comments

For a quite long time my opinion in this question was: Subversion. Maybe this has todo with “oldschool” :)

But no, that wasn’t only that. It is the simplicity of subversion. All started with the task “Choose your weapon!” when I had to setup a versioning system for future projects. I had used subversion before, so I was similar with it’s commands, and even newbie friendly server administration. I spent some time researching, how a setup of a git hoster. Compared to the same task with subversion, my decision was clear so far.

But let’s go through the facts!

Subversion

Subversion is a centralized versioning system, what makes thinks really easy at the beginning. Of course, everything about trunk, brachning and tagging, was foundet in this system. But you are free just to use Subversion as a pure versioning system. This means, you are not managing a standard layout as trunk/, branches/ and tags/. But when you do, you have all the benefits of branching and merging.

But the greatest benefit of centralization is, it’s simple. You have got one central node, or host, which keeps your repository. Every commit is going through it. The first time you notice this, is when you setup a Subersion host. There at least 2 steps: installing subversion, and doing a svnserve. Okay, you can also make some init scripts and so on. But this is candy :)

More on benefits: not only a repository can act as an repository, the subdirectories also can. For example, you can create a “project”, which has no standard layout, but its subdirectories have. So you are fully free to design your own structure, by keeping the subversion theory of trunk-branch-tag.

Now to the disadvantages, you will have to think about backups. Subversion gives no solutions for that. More that this, each checkout of your repo is not a complete replica of the repo. All the logs and commits are saved on host.

And back again, it has a free design. I managed it to use subversion a long time, without having any idea what branches are. With great power comes great responsibility.

GIT

One things I hated right at the beginning is that git is beeing hyped that strong all the time. Git is king, git is best, fuck subversion, it’s for noobs :) The most thing I dislike, that no one can tell me WHY git is better then subversion, but it is better? Maybe it has something to do with git is made by Linus Torwalds, so it just must be better by definition.

Now seriously, when you work on subversion with strict standard layouts, doing branches and tags, git brings this right out of the box. In git, you just can’t really go beside branches. When you init a repo, you are already in “master” branch, which is the trunk from subversion. The good is, branching and tagging is a core git feature, which is not realized with directories. For example: when you just create a branches and a tags directory in subversion, there is no difference between them. So when you just tag, you still can commit into this directory or even merge it back to trunk. You can avoid it by your own, or justify more options. In git, this is standard.

But let’s talk about gits distributive behaviour. This is mighty, but only if you know the benefits of it. This make your life a bit harder, because every git repo is by it’s own in nature, independent of something else. But this also means, every clone is a full replica, including commits and history. Looking for a way to make a backup of your repo quickly? Just clone it somewhere. You will be able reproduce the repo on host, if it will lost. When you are looking for remote repositories like in subversion, you will need a git host. But when you commit, you commit inside your own repository. To make a subversion style commit, you “push”. This sounds confusing, but make things more structured. I prefer to make commits more often, and using the messages as notes, what I had done recently. Using subversion, I know, every commit goes to the host, which is open for all. In git, I say, when all my commit are going to the host. “A” host? No, git is distributed, so you can also have several hosts :) You see, you are free to do more, if you know how you can profit from it.

One more thing: I really really hate subversion set props for managing ignore files. I was crying of happiness, when I saw how gitignore works…

When things get critically

Why do we need all this? We need collaboration tools for modern software development. That’s why developpers are sometimes looking for the right answer: git or subversion?

So it’s time to talk about, what you really need.

You will need:

  • a versioning system
  • a host for your VCS, what is simply accessible from all common plattforms
  • a deployment system, which is not to complicated
  • Starting from a bare VCS, you will have a long and hard way to go. In my past I had managed such systems, but I was the only one, who was able to use it in critical situations as - – reverting releases, when they failed.

I have changed my mind from subversion to git, when I found Meat!

Meat comes with a mighty collaboration system, as you know it from github. It actually is working similar to github, but adds release scenarios. Finally I got in touch with git, after studying how to use subversion correctly. And a lot of thinks, I had to manage manually in subversion, was working out of the box in git. Like branching, read only tags, complex hosting nodes. And even in the point of “how should I backup my files”, I am very chilled with git.

Both systems are mighty, but now I can say: git brings all that things you need from a software developer oriented versioning system.

Together with Meat!, git is my new best friend :)