E2e acceptance testing with Cucumber, Cypress, Angular, Gitlab-CI, Docker compose, and Asciidoctor

August 02, 2020

This article shows a complete setup of Cucumber, Cypress, Angular, Gitlab-CI, Docker compose, and Asciidoctor/Cukedoctor that can be used to automate the testing of acceptance criteria in an end-to-end manner.

This article focusses on the the technical side and does not discuss best practices or anti-patterns of acceptance testing nor the usage of Cucumber or Cypress.

In a nutshell, we want

to document our business requirements and maintain acceptance criteria
to ensure that the business requirements are fulfilled by our software solution
document the execution results of the tests of the business requirements
mock as least as possible

The solution strategy looks like this:

Write the business requirements in the form of Gherkin Cucumber specs. https://cucumber.io/
Enrich the specification in natural language in asciidoc. https://asciidoc.org/
Write the step definitions in Cypress. https://www.cypress.io/
Set-up a disposable staging system in Docker compose. https://docs.docker.com/compose/
Run the specs and the staging system in Gitlab-CI. https://docs.gitlab.com/ee/ci/
Process the execution results with Cukedoctor and publish a living documentation. https://github.com/rmpestano/cukedoctor

We are fully aware that Cypress considers it to be bad practice to introduce state into the test execution and avoid mocking of communication etc. But we don't use Cypress for isolated UI tests, rather we use Cypress for end-to-end testing. Therefore, we want to use the application in the same way as the end user would do it. We want to make sure that the state is handled correctly, not only during component and unit testing, but also in the final application. We want to make sure that the full communication path via back-end systems down to the database works. Hence, we are looking to the top of the test pyramid.

But of course the application should also apply other tests of different granularity. (Martin Fowler describes this pyramid in detail here: https://martinfowler.com/articles/practical-test-pyramid.html)

The disadvantages of end-to-end testing are maintenance effort, low execution speed and the problem of false positives. Nevertheless, from our perspective the combination of Cucumber and Cypress works very well and minimizes these shortcomings.

Setup

Our system consists of different parts.

e2e.git contains a Cypress application, the Cucumber specifications, and the e2e pipeline specification that is executed by Gitlab-CI
front-end.git contains our Angular application
back-end.git contains our back-end application (we assume a Spring Boot app - but this doesn't really matter)
a Docker registry with all of our images that we need to run the pipelines
the Gitlab-CI system as our main execution environment, that is capable of running docker-compose and other docker-in-docker stuff
a regular web server that serves the generated asciidoc based documentation

We will assume that back-end and front-end already have working CI-pipelines and the built artifacts are deployed as Docker images in the registry.

CI setup

Angular setup

There is no special setup required in order to run an Angular app with Cypress. Nevertheless, some points could be considered.

API URL

Depending on the setup of the Angular App, there might be a point where the URL of the back-end API is configured. If the production build shall be used during e2e test, this URL must be changed most likely. This could be done as follows in the environment.prod.ts file:

function isTestMode(): boolean {
  return !!(window as any).Cypress
}

export const environment = {
  production: true,
  backendUrl: isTestMode()
    ? 'http://docker:8080'
    : 'https://back-end.bytethefrog.de',
}

Please note the URL http://docker:8080. The host name docker is the important part, because this is the name that is available when the staging system is run within docker-compose within a Gitlab-CI pipeline.

Navigation

If we would like to call methods during test execution directly in Angular, a possible solution would be to store callback functions in the window that can be called by Cypress.

export class AppComponent {
  constructor(private router: Router, private ngZone: NgZone) {
    if ((window as any).Cypress) {
      (window as any).cypressNavigateByUrl = (url: string) =>
        this.ngZone.run(() => this.router.navigateByUrl(url))
    }
  }
}

Cypress-Cucumber-Preprocessor

Cucumber tests consist of two parts, the Gherkin specification and the step definitions. A Gherkin spec is a high level description of the test execution, while the step definition implements the actual programming logic.

The Cypress-Cucumber-Preprocessor glues these two parts together. The setup is not very complicated and you can find a working example with typescript and webpack here: https://github.com/TheBrainFamily/cypress-cucumber-webpack-typescript-example

Base URL vs. visit()

The Cypress base URL is an important configuration setting. If it is chosen unfavorably Cypress will reload the Angular application with every visit() statement.

CYPRESS_BASE_URL="http://localhost:4200/#" will work with Angular when useHash: true is configured in RouterModule. In Cypress the following could be used then:

cy.visit('/')
// ...
cy.visit('/logout')
// ...
cy.visit('/login')
// ...

Furthermore, it is a good idea to keep CYPRESS_BASE_URL outside your tests, so you will be able to run the same tests against different systems (e.g. local development, staging, production, ...).

Configuration of JSON reporter

For processing of the execution result we need to provide reports in JSON format to Cukedoctor. This can be configured in the package.json with the following snippet. The reports (<featureName>.cucumber.json) then will be put into the cucumber-json directory.

"cucumberJson": {
  "generate": true,
  "outputFolder": "cucumber-json",
  "filePrefix": "",
  "fileSuffix": ".cucumber"
}

A final package.json could look like this:

{
  "name": "e2e",
  "version": "1.0.0",
  "description": "Cucumber specifications and end-to-end tests",
  "scripts": {
    "run": "cypress run",
    "test": "cypress run --spec \"**/*.feature\"",
    "test:all": "cypress run --spec \"**/*.features\"",
    "test:local": "CYPRESS_WORKING_DIR=`pwd` CYPRESS_BASE_URL=\"http://localhost:4200/#\" cypress run --spec \"**/*.feature\"",
    "run:local": "CYPRESS_WORKING_DIR=`pwd` CYPRESS_BASE_URL=\"http://localhost:4200/#\" cypress open"
  },
  "dependencies": {
    "@cypress/webpack-preprocessor": "^4.0.2",
    "@types/cypress-cucumber-preprocessor": "^1.14.1",
    "@types/node": "^10.12.11",
    "cypress": "^4.9.0",
    "cypress-cucumber-preprocessor": "^2.5.0",
    "cypress-wait-until": "^1.7.1",
    "ts-loader": "^5.3.1",
    "typescript": "^3.4.5",
    "webpack": "^4.28.2"
  },
  "cypress-cucumber-preprocessor": {
    "nonGlobalStepDefinitions": true,
    "cucumberJson": {
      "generate": true,
      "outputFolder": "cucumber-json",
      "filePrefix": "",
      "fileSuffix": ".cucumber"
    }
  },
  "devDependencies": {
    "prettier": "^2.0.5"
  }
}

Extend Cypress with custom commands written in Typescript

Since we are writing our Cypress tests in Typescript, we also need type definitions if we write custom commands. This can be achieved easily by providing a cypress/support/index.d.ts in the following form:

declare namespace Cypress {
  interface Chainable<Subject> {
    navigate(url: string): Chainable<any>
  }
}

And cypress/support/commands.js could contain:

Cypress.Commands.add('navigate', url => {
  cy.window().then(win => {
    win.cypressNavigateByUrl(url)
  })
})

Living documentation with Cukedoctor

Cukedoctor processes the execution result of a Cucumber run. In the first step an Asciidoctor document is created. This document is then converted into a html page or a pdf.

Cukedoctor can be run in a Docker container as well with this command:

docker run --rm -v "$PWD/cucumber-json/:/output" rmpestano/cukedoctor -f html -o /output/generated_doc/documentation

Using asciidoc

Embedding videos

Furthermore, Cypress records videos of the test execution that can be embedded in the documentation as well.

word-guess.feature

Feature: Guess the word

  video::word-guess.feature.mp4[]

  # The first example has two steps
  Scenario: Maker starts a game
    When the Maker starts a game
    Then the Maker waits for a Breaker to join

  # The second example has three steps
  Scenario: Breaker joins a game
    Given the Maker has started a game with the word "silky"
    When the Breaker joins the Maker's game
    Then the Breaker must guess a word with 5 characters

Gitlab, docker-in-docker and Docker compose

The general idea is to run a full production-like system as docker containers. This allows to perform end-to-end tests with real back-end, database and other related infrastructure that might be required to implement the specified business logic. For sure, if the targeted system is large it could be impossible to create a production-like environment within docker. In these cases it would be necessary to mock missing parts and to test the different sections of the whole system separately or to use a different staging than Docker compose.

Cypress docker-in-docker (dind) image

In order to spin up the Docker compose containers and run Cypress in the same container, we need an image that is capable of doing this. Cypress provides a Docker image for CI purposes but this does not contain a suitable Docker and Docker compose installation. Therefore, we extend the Cypress image. You need to build the image and store it in some place that can be accessed by Gitlab.

FROM cypress/included:4.11.0

ENTRYPOINT []

## Remove previous docker installation
RUN apt-get -y remove docker docker.io runc
RUN apt -y autoremove

RUN apt-get -y install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
RUN apt-key fingerprint 0EBFCD88
RUN add-apt-repository \
       "deb [arch=amd64] https://download.docker.com/linux/debian \
       $(lsb_release -cs) \
       stable"
RUN apt-get update
RUN apt-get -y install docker-ce docker-ce-cli containerd.io

# add credentials store
RUN apt-get -y install pass

## Install docker compose
RUN apt-get -y install docker-compose

Gitlab-CI

Finally, all this is used in the .gitlab-ci.yml. The most important parts are documented below.

variables:
  NPM_CONFIG_CACHE: '$CI_PROJECT_DIR/.npm'
  CYPRESS_CACHE_FOLDER: '$CI_PROJECT_DIR/.cache/Cypress'
  NODE_OPTIONS: '--max_old_space_size=4096'
  YARN_CACHE_FOLDER: '$CI_PROJECT_DIR/.yarn'
  CYPRESS_BASE_URL: 'http://docker:80/de/#'

services:
  - docker:19.03.12-dind

e2e:
  variables:
    DOCKER_TLS_CERTDIR: ''
    DOCKER_HOST: tcp://docker:2375
  stage: test
  image: registry.bytethefrog.de/cypress-docker:latest
  before_script:
    - docker login https://registry.bytethefrog.de --username ${DOCKER_REGISTRY_USER} --password ${DOCKER_REGISTRY_PW}
    - docker-compose up -d
  script:
    - yarn install --frozen-lockfile
    - yarn test || true
    - docker run --rm -v "$PWD/cucumber-json/:/output" rmpestano/cukedoctor -f html -o /output/generated_doc/documentation
    - mv cucumber-json/generated_doc/documentation.html cucumber-json/generated_doc/index.html
  after_script:
    - docker-compose logs
    - docker-compose down
  artifacts:
    expire_in: 1 week
    when: always
    paths:
      - cypress/videos
      - cypress/screenshots
      - cucumber-json/generated_doc
  cache:
    key: npm-cache
    paths:
      - .yarn
      - .npm
      - .cache

Furthermore, after Cypress has executed the tests and Cukedoctor generated the documentation, you should set up another Gitlab-CI job that publishes the collected artifact.

As mentioned before, all docker-compose containers can be reached by the hostname docker and the exposed port that is configured in the docker-compose.yml.

Docker-compose

The docker-compose setup is straightforwards. You need an ordinary docker-compose.yml without any special features.

version: '3'

services:
  back-end:
    image: registry.bytethefrog.de/back-end:latest
    environment:
      TZ: 'Europe/Berlin'
      INSTANCE: '1'
      STAGE: development
      SPRING_APPLICATION_JSON: '{
        "spring.datasource.url": "jdbc:postgresql://db:5432/test",
        "spring.datasource.username": "test",
        "spring.datasource.password": "test123",
        }'
    ports:
      - 8080:8080
    depends_on:
      - db

  front-end:
    image: registry.bytethefrog.de/front-end:latest
    ports:
      - 80:80

  db:
    image: postgres:alpine
    environment:
      POSTGRES_DB: test
      POSTGRES_USER: test
      POSTGRES_PASSWORD: test123

Trigger e2e-pipeline

Gitlab-CI provides a simple method in order to trigger pipelines of other projects. The documentation can be found here: https://docs.gitlab.com/ee/ci/multi_project_pipelines.html

All projects that want to start e2e tests (in this article it would be front-end.git and back-end.git) need a job like this in their .gitlab-ci.yml:

e2e:
  stage: e2e
  only:
    refs:
      - master
  trigger:
    project: btf/e2e
    branch: master

It would be possible to create more complex scenarios as well. But for the sake of demonstration, the example above should work well.