A year with Testcontainers

Fri, Apr 10, 2020

Tags:

Testcontainers is a Java library for integrating Docker containers into tests. It provides tools to spin up build images, spin up containers and interact with them. Extensions have been provided to wrap commonly used tools, for example, launching a Postgres Database and getting the JDBC connection is super simple.

@Test
public void testSimple() throws SQLException {
	try (PostgreSQLContainer postgres = new PostgreSQLContainer<>()) {
		postgres.start();
		....
	}
}

Our team has been using Testcontainers for just over a year now and I though I’d share some experiences, pros and cons. This is not a “how to” blog post though.

Usage

The first use for database integration testing. Our product can support multiple server types at the back-end, creating a bit of an integration test nightmare. Tests against the different database types would have to be done manually, usually at the end of the development cycle and required testing infrastructure (VMs running databases). It was all slow, delaying feedback until late in the cycle, and was error prone.

This sounded like a job for containers!

After all, this was the promise. Infrastructure that could be spun up and spun down at the drop of a hat. No VMs or humans involved.

Testcontainers seemed like the perfect place to start. It allowed our test code to launch the required databases (in containers), and for automated tests to be built.

It worked!

In no time at all we had a framework set up. Now automated tests ran our code against all the database types we supported. The manual, error prone, slow process was gone and feedback (test failures) was much, much, much more timely.

For once, hype was lived up to.

The party went on.

Tests were built for other “hard to test” things including:

Shell scripts
Installers
System control scripts (Systemd)

However, some limitations started to emerge.

Resource leaks

Although Testcontainers tries to safeguard against resource leaks (containers, temporary images etc…), it wasn’t perfect. If the test process was aborted, this cleanup would be skipped sometimes.

This resulted in disks becoming full with redundant images, and in one cause leaked containers were using so much memory that one of our Jenkins build nodes stopped working.

Complex deployments

Some integration tests require multiple components to be launched in their own containers and then to interact together. The safe way to this is with a temporary Docker network to which all the containers join, and pass this to all the containers involved.

try (
        Network network = Network.newNetwork();

        GenericContainer foo = new GenericContainer()
                .withNetwork(network)
                .withNetworkAliases("foo")
                .withCommand("/bin/sh", "-c", "while true ; do printf 'HTTP/1.1 200 OK\\n\\nyay' | nc -l -p 8080; done");

        GenericContainer bar = new GenericContainer()
                .withNetwork(network)
                .withCommand("top")
) {
    foo.start();
    bar.start();

    String response = bar.execInContainer("wget", "-O", "-", "http://foo:8080").getStdout();
    assertEquals("received response", "yay", response);
}

This starts to get quite ugly with large deployments.

Testcontainers does provide a Docker Compose library, allowing for a whole stack of servers to be spun up and down together. However, this requires having docker-compose yaml files which start the right services for the integration test that you wish to perform which feels less flexible than just programming in code.

Truthfully, this type of use-case is stretching Testcontainers and goes against its original design goals. There are better tools available to do large scale, black-box testing.

Reliant on Docker

Testcontainers is very coupled with Docker, using its REST API to interact with containers.

This is great… if you can use Docker.

Currently the Docker daemon has some well known security problems which are problematic to some organisations. Also, all the containers run on a single machine, which can easily become littered with old data (images, volumes, containers, networks etc…).

For these reasons, my current employer has decided to move away from Docker, which means we have to move away from Testcontainers.

Conclusion

Testcontainers delivers in providing a simple way to use containers in tests. It has allowed us to automate a bunch of laborious manual testing, saving time and reducing errors.

Its great for interacting with a single container, and is OK for slightly larger deployments if you can manage the extra code complexity.

However, its dependency on Docker could prove to be a problem longer term. Although Docker popularised containers and made them easy to use, it has its limitations and there are viable alternatives now.

It would be interesting to see if Testcontainers pivots to support other container runtimes (i.e. podman), or starts to supports running containers in Kubernetes.