Easier DynamoDB local testing

I'm using DynamoDB local for unit testing. It's not bad, but has some drawbacks. Specifically:

You have to somehow start the server before your tests run
The server isn't started and stopped before each test so tests become inter-dependent unless you add code to delete all tables, etc. after each test
All developers need to have it installed

What I want to do is something like put the DynamoDB local jar, and the other jars upon which it depends, in my test/resources directory (I'm writing in Java). Then before each test I'd start it up, running with -inMemory, and after the test I'd stop it. That way anyone pulling down the git repo gets a copy of everything they need to run the tests and each test is independent of the others.

I have found a way to make this work, but it's ugly, so I'm looking for alternatives. The solution I have is to put a .zip file of the DynamoDB local stuff in test/resources, then in the @Before method, I'd extract it to some temporary directory and start a new java process to execute it. That works, but it's ugly and has some drawbacks:

Everyone needs the java executable on their $PATH
I have to unpack a zip to the local disk. Using local disk is often dicey for testing, especially with continuous builds and such.
I have to spawn a process and wait for it to start for each unit test, and then kill that process after each test. Besides being slow, the potential for left-over processes seems ugly.

It seems like there should be an easier way. DynamoDB Local is, after all, just Java code. Can't I somehow ask the JVM to fork itself and look inside the resources to build a classpath? Or, even better, can't I just call the main method of DynamoDB Local from some other thread so this all happens in a single process? Any ideas?

PS: I am aware of Alternator, but it appears to have other drawbacks so I'm inclined to stick with Amazon's supported solution if I can make it work.

Solution 1:

In order to use DynamoDBLocal you need to follow these steps.

Get Direct DynamoDBLocal Dependency
Get Native SQLite4Java dependencies
Set sqlite4java.library.path to show native libraries

1. Get Direct DynamoDBLocal Dependency

This one is the easy one. You need this repository as explained here.

<!--Dependency:-->
<dependencies>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>DynamoDBLocal</artifactId>
        <version>1.11.0.1</version>
        <scope></scope>
    </dependency>
</dependencies>
<!--Custom repository:-->
<repositories>
    <repository>
        <id>dynamodb-local</id>
        <name>DynamoDB Local Release Repository</name>
        <url>https://s3-us-west-2.amazonaws.com/dynamodb-local/release</url>
    </repository>
</repositories>

2. Get Native SQLite4Java dependencies

If you do not add these dependencies, your tests will fail with 500 internal error.

First, add these dependencies:

<dependency>
    <groupId>com.almworks.sqlite4java</groupId>
    <artifactId>sqlite4java</artifactId>
    <version>1.0.392</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.almworks.sqlite4java</groupId>
    <artifactId>sqlite4java-win32-x86</artifactId>
    <version>1.0.392</version>
    <type>dll</type>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.almworks.sqlite4java</groupId>
    <artifactId>sqlite4java-win32-x64</artifactId>
    <version>1.0.392</version>
    <type>dll</type>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.almworks.sqlite4java</groupId>
    <artifactId>libsqlite4java-osx</artifactId>
    <version>1.0.392</version>
    <type>dylib</type>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.almworks.sqlite4java</groupId>
    <artifactId>libsqlite4java-linux-i386</artifactId>
    <version>1.0.392</version>
    <type>so</type>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>com.almworks.sqlite4java</groupId>
    <artifactId>libsqlite4java-linux-amd64</artifactId>
    <version>1.0.392</version>
    <type>so</type>
    <scope>test</scope>
</dependency>

Then, add this plugin to get native dependencies to specific folder:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <version>2.10</version>
            <executions>
                <execution>
                    <id>copy</id>
                    <phase>test-compile</phase>
                    <goals>
                        <goal>copy-dependencies</goal>
                    </goals>
                    <configuration>
                        <includeScope>test</includeScope>
                        <includeTypes>so,dll,dylib</includeTypes>
                        <outputDirectory>${project.basedir}/native-libs</outputDirectory>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

3. Set sqlite4java.library.path to show native libraries

As last step, you need to set sqlite4java.library.path system property to native-libs directory. It is OK to do that just before creating your local server.

System.setProperty("sqlite4java.library.path", "native-libs");

After these steps you can use DynamoDBLocal as you want. Here is a Junit rule that creates local server for that.

import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient;
import com.amazonaws.services.dynamodbv2.local.main.ServerRunner;
import com.amazonaws.services.dynamodbv2.local.server.DynamoDBProxyServer;
import org.junit.rules.ExternalResource;

import java.io.IOException;
import java.net.ServerSocket;

/**
 * Creates a local DynamoDB instance for testing.
 */
public class LocalDynamoDBCreationRule extends ExternalResource {

    private DynamoDBProxyServer server;
    private AmazonDynamoDB amazonDynamoDB;

    public LocalDynamoDBCreationRule() {
        // This one should be copied during test-compile time. If project's basedir does not contains a folder
        // named 'native-libs' please try '$ mvn clean install' from command line first
        System.setProperty("sqlite4java.library.path", "native-libs");
    }

    @Override
    protected void before() throws Throwable {

        try {
            final String port = getAvailablePort();
            this.server = ServerRunner.createServerFromCommandLineArgs(new String[]{"-inMemory", "-port", port});
            server.start();
            amazonDynamoDB = new AmazonDynamoDBClient(new BasicAWSCredentials("access", "secret"));
            amazonDynamoDB.setEndpoint("http://localhost:" + port);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }

    @Override
    protected void after() {

        if (server == null) {
            return;
        }

        try {
            server.stop();
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }

    public AmazonDynamoDB getAmazonDynamoDB() {
        return amazonDynamoDB;
    }

    private String getAvailablePort() {
        try (final ServerSocket serverSocket = new ServerSocket(0)) {
            return String.valueOf(serverSocket.getLocalPort());
        } catch (IOException e) {
            throw new RuntimeException("Available port was not found", e);
        }
    }
}

You can use this rule like this

@RunWith(JUnit4.class)
public class UserDAOImplTest {

    @ClassRule
    public static final LocalDynamoDBCreationRule dynamoDB = new LocalDynamoDBCreationRule();
}

Solution 2:

In August 2018 Amazon announced new Docker image with Amazon DynamoDB Local onboard. It does not require downloading and running any JARs as well as adding using third-party OS-specific binaries (I'm talking about sqlite4java).

It is as simple as starting a Docker container before the tests:

docker run -p 8000:8000 amazon/dynamodb-local

You can do that manually for local development, as described above, or use it in your CI pipeline. Many CI services provide an ability to start additional containers during the pipeline that can provide dependencies for your tests. Here is an example for Gitlab CI/CD:

test:
  stage: test
  image: openjdk:8-alpine
  services:
    - name: amazon/dynamodb-local
      alias: dynamodb-local
  script:
    - DYNAMODB_LOCAL_URL=http://dynamodb-local:8000 ./gradlew clean test

Or Bitbucket Pipelines:

definitions:
  services:
    dynamodb-local:
      image: amazon/dynamodb-local
…
step:
  name: test
  image:
    name: openjdk:8-alpine
  services:
    - dynamodb-local
  script:
    - DYNAMODB_LOCAL_URL=http://localhost:8000 ./gradlew clean test

And so on. The idea is to move all the configuration you can see in other answers out of your build tool and provide the dependency externally. Think of it as of dependency injection / IoC but for the whole service, not just a single bean.

After you've started the container you can create a client pointing to it:

private AmazonDynamoDB createAmazonDynamoDB(final DynamoDBLocal configuration) {
    return AmazonDynamoDBClientBuilder
        .standard()
        .withEndpointConfiguration(
            new AwsClientBuilder.EndpointConfiguration(
                "http://localhost:8000",
                Regions.US_EAST_1.getName()
            )
        )
        .withCredentials(
            new AWSStaticCredentialsProvider(
                // DynamoDB Local works with any non-null credentials
                new BasicAWSCredentials("", "")
            )
        )
        .build();
}

Now to the original questions:

You have to somehow start the server before your tests run

You can just start it manually, or prepare a developsers' script for it. IDEs usually provide a way to run arbitrary commands before executing a task, so you can make IDE to start the container for you. I think that running something locally should not be a top priority in this case, but instead you should focus on configuring CI and let the developers start the container as it's comfortable to them.

The server isn't started and stopped before each test so tests become inter-dependent unless you add code to delete all tables, etc. after each test

That's trueee, but… You should not start and stop such heavyweight things and recreate tables before / after each test. DB tests are almost always inter-dependent and that's ok for them. Just use unique values for each test case (e.g. set item's hash key to ticket id / specific test case id you're working on). As for the seed data, I'd recommend moving it from the build tool and test code as well. Either make your own image with all the data you need or use AWS CLI to create tables and insert data. Follow the single responsibility principle and dependency injection principles: your test code must not do anything but tests. All the environment (tables and data in this case should be provided for them). Creating a table in a test is wrong, because in a real life that table already exist (unless you're testing a method that actually creates a table, of course).

All developers need to have it installed

Docker should be a must for every developer in 2018, so that's not a problem.

And if you're using JUnit 5, it can be a good idea to use a DynamoDB Local extension that will inject the client in your tests (yes, I'm doing a self-promotion):

Add a dependency on me.madhead.aws-junit5:dynamodb-v1

pom.xml:

<dependency>
    <groupId>me.madhead.aws-junit5</groupId>
    <artifactId>dynamo-v1</artifactId>
    <version>6.0.1</version>
    <scope>test</scope>
</dependency>

build.gradle

dependencies {
    testImplementation("me.madhead.aws-junit5:dynamo-v1:6.0.1")
}

Use the extension in your tests:

@ExtendWith(DynamoDBLocalExtension.class)
class MultipleInjectionsTest {
    @DynamoDBLocal(
        url = "http://dynamodb-local-1:8000"
    )
    private AmazonDynamoDB first;

    @DynamoDBLocal(
        urlEnvironmentVariable = "DYNAMODB_LOCAL_URL"
    )
    private AmazonDynamoDB second;

    @Test
    void test() {
        first.listTables();
        second.listTables();
    }
}

Easier DynamoDB local testing

Solution 1:

Solution 2:

Related

Recent Posts