parameterized test with cartesian product of arguments in pytest

Just wondering, is there any (more) elegant way of parameterizing with the cartesian product? This is what I figured out so far:

numbers    = [1,2,3,4,5]
vowels     = ['a','e','i','o','u']
consonants = ['x','y','z']

cartesian = [elem for elem in itertools.product(*[numbers,vowels,consonants])]

@pytest.fixture(params=cartesian)
def someparams(request):
  return request.param

def test_something(someparams):
  pass

At least I'd like to encapsulate numbers, vowels, consonants and cartesian in the fixture function.

Solution 1:

You can apply multiple parametrize arguments, in which case they will generate a product of all parameters:

import pytest

numbers = [1,2,3,4,5]
vowels = ['a','e','i','o','u']
consonants = ['x','y','z']


@pytest.mark.parametrize('number', numbers)
@pytest.mark.parametrize('vowel', vowels)
@pytest.mark.parametrize('consonant', consonants)
def test(number, vowel, consonant):
    pass

Solution 2:

I can think of two ways to do this. One uses parametrized fixtures, and one parametrizes the test function. It's up to you which one you find more elegant.

Here is the test function parametrized:

import itertools
import pytest

numbers = [1,2,3,4,5]
vowels = ['a','e','i','o','u']
consonants = ['x','y','z']


@pytest.mark.parametrize('number,vowel,consonant',
    itertools.product(numbers, vowels, consonants)
)
def test(number, vowel, consonant):
    pass

Of note, the second argument to the parametrize decorator can be an iterable, not just a list.

Here is how you do it by parametrizing each fixture:

import pytest

numbers = [1,2,3,4,5]
vowels = ['a','e','i','o','u']
consonants = ['x','y','z']


@pytest.fixture(params=numbers)
def number(request):
    return request.param

@pytest.fixture(params=vowels)
def vowel(request):
    return request.param

@pytest.fixture(params=consonants)
def consonant(request):
    return request.param


def test(number, vowel, consonant):
    pass

Your intuition was correct. By parametrizing each of multiple fixtures, pytest takes care of creating all the permutations that arise.

The test output is identical. Here is a sample (I ran py.test with the -vv option):

test_bar.py:22: test[1-a-x] PASSED
test_bar.py:22: test[1-a-y] PASSED
test_bar.py:22: test[1-a-z] PASSED
test_bar.py:22: test[1-e-x] PASSED
test_bar.py:22: test[1-e-y] PASSED
test_bar.py:22: test[1-e-z] PASSED
test_bar.py:22: test[1-i-x] PASSED

Solution 3:

I think besides an elegant solution, you should also consider both the amount of time each option will take and the amount of code you'll have to maintain.

Possible solutions

1. Using parametrize once with itertools (provided by Frank T)
2. Using 3 fixtures (provided by Frank T)
3. Using parametrize 3 times (provided by Bruno Oliveira)
4. Using 1 fixture and itertools (provided in the question)

Solution 1

@pytest.mark.parametrize('number, vowel, consonant',
                         itertools.product(numbers, vowels, consonants))
def test(number, vowel, consonant):
    pass

Solution 2

@pytest.fixture(params=numbers)
def number(request): return request.param

@pytest.fixture(params=vowels)
def vowel(request): return request.param

@pytest.fixture(params=consonants)
def consonant(request): return request.param


def test(number, vowel, consonant):
    pass

Solution 3

@pytest.mark.parametrize('number', numbers)
@pytest.mark.parametrize('vowel', vowels)
@pytest.mark.parametrize('consonant', consonants)
def test(number, vowel, consonant):
    pass

Solution 4

@pytest.fixture(params=cartesian)
def someparams(request):
  return request.param

def test_something(someparams):
  pass

When it comes to elegance, I consider that Solution 3 is the best option because it has less code maintain, and it does not require to import itertools. After that Solution 1 is the best choice because you don't need to write fixtures as Solution 4, and Solution 2. Solution 4 is probably better than Solution 2 because it requires less code to maintain.

When it comes to performance I run each solution using numbers = list(range(100)), and I got the following results:

|  Solution  |  Time    | 
| Solution 1 |  3.91s   |
| Solution 2 |  3.59s   |
| Solution 3 |  3.54s   |
| Solution 4 |  3.09s   |