One of the most important properties of good tests is that they should be isolated. As Kent Beck puts it “tests should return the same results regardless of the order in which they are run”. That is to say no test should affect the results of any other test. This can get challenging when tests use a shared resource like a database. It should be easy to imagine how one test could break another by modifying a shared database as they both run.

So how do we isolate tests that use a database?

A simple solution is to run the tests serially. With serial tests all we need for isolation is some setup code to reset the database at the start of every test. With Jest we can run tests serially using the --runInBand option. The downside of this approach is that the tests will run much more slowly than they otherwise might. All your extra cpu cores will sadly sit idle!

A more complex solution is to isolate tests using database transactions. In Rails, for example, every test is run inside a database transaction. At the end of each test the transaction is rolled back so changes made by one test are not observable from any other test. This solution usually requires support from your database and testing libraries. Unfortunately I’m currently using Jest and TypeORM which does not yet support this.

Luckily there is another way to isolate these tests while still achieving a high degree of parallelism: run multiple databases. The idea is to split our tests into N groups and assign each group its own database. Tests within each group are run serially and each group is run in parallel to each other. This effectively implements the original solution N times and in parallel. It is also remarkably easy to implement using Docker and Jest.

Step 1. Choose a value for N

In terms of grouping and running tests Jest behaves exactly as we need it to. Jest spawns one or more “workers” to execute tests. Each worker runs in parallel to each other but the tests within each worker are performed serially. By default the number of workers used is equal to the number of cpu cores available minus one (for the main thread). This means that on my new 10-core MacBook Pro Jest spawns 9 workers.

The value of N is the number of cpu cores on your machine minus one. This is the number of workers Jest will use and we will need to create a database for each one.

Step 2. Set up N databases

We can quickly run multiple databases on our machine using Docker. Describing Docker is beyond the scope of this blog post so if you are not already familiar with it I highly recommend investing the time to learn it. Below is a docker-compose.yml file with a configuration for running 9 PostgreSQL databases. Importantly each instance is listening on a different local port – sequentially from 5433 – 5441. You will see later on that our solution depends on these ports being sequential. We start the databases by running docker-compose up --detach.

version: '3.1'
services:

  testdb-1:
    image: postgres:10.14
    restart: unless-stopped
    ports:
      - '5433:5432'
    environment:
      POSTGRES_PASSWORD: mypassword
      POSTGRES_DB: mydb

  testdb-2:
    image: postgres:10.14
    restart: unless-stopped
    ports:
      - '5434:5432'
    environment:
      POSTGRES_PASSWORD: mypassword
      POSTGRES_DB: mydb

  testdb-3:
    image: postgres:10.14
    restart: unless-stopped
    ports:
      - '5435:5432'
    environment:
      POSTGRES_PASSWORD: mypassword
      POSTGRES_DB: mydb

  testdb-4:
    image: postgres:10.14
    restart: unless-stopped
    ports:
      - '5436:5432'
    environment:
      POSTGRES_PASSWORD: mypassword
      POSTGRES_DB: mydb

  testdb-5:
    image: postgres:10.14
    restart: unless-stopped
    ports:
      - '5437:5432'
    environment:
      POSTGRES_PASSWORD: mypassword
      POSTGRES_DB: mydb

  testdb-6:
    image: postgres:10.14
    restart: unless-stopped
    ports:
      - '5438:5432'
    environment:
      POSTGRES_PASSWORD: mypassword
      POSTGRES_DB: mydb

  testdb-7:
    image: postgres:10.14
    restart: unless-stopped
    ports:
      - '5439:5432'
    environment:
      POSTGRES_PASSWORD: mypassword
      POSTGRES_DB: mydb

  testdb-8:
    image: postgres:10.14
    restart: unless-stopped
    ports:
      - '5440:5432'
    environment:
      POSTGRES_PASSWORD: mypassword
      POSTGRES_DB: mydb

  testdb-9:
    image: postgres:10.14
    restart: unless-stopped
    ports:
      - '5441:5432'
    environment:
      POSTGRES_PASSWORD: mypassword
      POSTGRES_DB: mydb

Step 3. Configure each test to use the correct database

The last step is to configure each test to connect to the right database. We need all tests from the same worker to connect to the same database. Jest assigns each worker a unique integer id starting with 1 and a test can find the id of its worker in the JEST_WORKER_ID environment variable. This allows us to dynamically select the database using JEST_WORKER_ID as a database port number offset. I use code like the below to configure the database connection details in each test.

config = {
    type: 'postgres',
    host: 'localhost',
    port: 5432 + parseInt(process.env.JEST_WORKER_ID),
    username: 'postgres',
    password: 'mypassword',
    dropSchema: true, // drop schema and then
    synchronize: true, // create schema based on current code
}

As you can see tests from the first worker will use the database on port 5433. Tests from the second worker will use the database on port 5434 and so on.

TypeORM users will note that I use dropSchema: true and synchronize: true. These are TypeORM specific flags that do the work of fully resetting the database at the start of each test. Users of other database libraries will need to implement their own solution for resetting the database between tests.

Conclusion

In this post I show how you can run tests in parallel with Jest using multiple databases. Tests within a given worker are isolated because they run serially and include code to reset the database at the start of each test. Tests across workers are isolated because they use completely different databases. This is a relatively simple way to achieve high parallelism for tests that use a database. I was able to implement this solution in about 15 minutes and on my 10-core MacBook Pro it cut my build times in half.