JUnit test with dynamic number of tests

Take a look at Parameterized Tests in JUnit 4.

Actually I did this a few days ago. I'll try to explain ...

First build your test class normally, as you where just testing with one input file. Decorate your class with:

@RunWith(Parameterized.class)

Build one constructor that takes the input that will change in every test call (in this case it may be the file itself)

Then, build a static method that will return a Collection of arrays. Each array in the collection will contain the input arguments for your class constructor e.g. the file. Decorate this method with:

@Parameters

Here's a sample class.

@RunWith(Parameterized.class)
public class ParameterizedTest {

    private File file;

    public ParameterizedTest(File file) {
        this.file = file;
    }

    @Test
    public void test1() throws Exception {  }

    @Test
    public void test2() throws Exception {  }

    @Parameters
    public static Collection<Object[]> data() {
        // load the files as you want
        Object[] fileArg1 = new Object[] { new File("path1") };
        Object[] fileArg2 = new Object[] { new File("path2") };

        Collection<Object[]> data = new ArrayList<Object[]>();
        data.add(fileArg1);
        data.add(fileArg2);
        return data;
    }
}

Also check this example

JUnit 3

public class XTest extends TestCase {

    public File file;

    public XTest(File file) {
        super(file.toString());
        this.file = file;
    }

    public void testX() {
        fail("Failed: " + file);
    }

}

public class XTestSuite extends TestSuite {

    public static Test suite() {
        TestSuite suite = new TestSuite("XTestSuite");
        File[] files = new File(".").listFiles();
        for (File file : files) {
            suite.addTest(new XTest(file));
        }
        return suite;
    }

}

JUnit 4

import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

@RunWith(Parameterized.class)
public class TestY {

    @Parameters
    public static Collection<Object[]> getFiles() {
        Collection<Object[]> params = new ArrayList<Object[]>();
        for (File f : new File(".").listFiles()) {
            Object[] arr = new Object[] { f };
            params.add(arr);
        }
        return params;
    }

    private File file;

    public TestY(File file) {
        this.file = file;
    }

    @Test
    public void testY() {
        fail(file.toString());
    }

}

Junit 5 Parameterized Tests

JUnit 5 parameterized tests support this by allowing the use of a method as data source:

@ParameterizedTest
@MethodSource("fileProvider")
void testFile(File f) {
    // Your test comes here
}

static Stream<File> fileProvider() {
    return Arrays.asList(new File(".").list()).stream();
}

JUnit 5 DynamicTests

JUnit 5 also supports this through the notion of a DynamicTest, which is to be generated in a @TestFactory, by means of the static method dynamicTest.

import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.TestFactory;
import static org.junit.jupiter.api.DynamicTest.dynamicTest;

import java.util.stream.Stream;

@TestFactory
public Stream<DynamicTest> testFiles() {
    return Arrays.asList(new File(".").list())
            .stream()
            .map((file) -> dynamicTest(
                    "Test for file: " + file,
                    () -> { /* Your test comes here */ }));
}

The tests run in your IDE (IntelliJ here) will be displayed like this:

Should be possible in JUnit 3 by inheriting from TestSuite and overriding the tests() method to list the files and for each return an instance of a subclass of TestCase that takes the filename as constructor parameter and has a test method that tests the file given in the constructor.

In JUnit 4 it might be even easier.

You could consider using JUnitParams library, so you would have a few more (cleaner) options:

@org.junit.runner.RunWith(junitparams.JUnitParamsRunner.class)
public class ParameterizedTest {

    @org.junit.Test
    @junitparams.Parameters(method = "data")
    public void test1(File file) throws Exception {  }

    @org.junit.Test
    @junitparams.Parameters(method = "data")
    public void test2(File file) throws Exception {  }

    public static File[] data() {
        return new File[] { new File("path1"), new File("path2") };
    }
}

@org.junit.runner.RunWith(junitparams.JUnitParamsRunner.class)
public class ParameterizedTest {

    @org.junit.Test
    @junitparams.Parameters(value = { "path1", "path2" })
    public void test1(String path) throws Exception {
        File file = new File(path);
    }

    @org.junit.Test
    @junitparams.Parameters(value = { "path1", "path2" })
    public void test2(String path) throws Exception {
        File file = new File(path);
    }
}

You can see more samples of usage here.

In addition about JUnitParams, why writting parameterized tests with it is easier and more readable:

JUnitParams project adds a new runner to JUnit and provides much easier and readable parametrised tests for JUnit >=4.6.

Main differences to standard JUnit Parametrised runner:

• more explicit - params are in test method params, not class fields
• less code - you don't need a constructor to set up parameters
• you can mix parametrised with non-parametrised methods in one class
• params can be passed as a CSV string or from a parameters provider class
• parameters provider class can have as many parameters providing methods as you want, so that you can group different cases
• you can have a test method that provides parameters (no external classes or statics anymore)
• you can see actual parameter values in your IDE (in JUnit's Parametrised it's only consecutive numbers of parameters)