github gitlab twitter
Customizing your pytest test suite (part 1)
Mar 25, 2019
22 minutes read

PyBerlin is a new meetup community in Berlin, Germany that strives to be open and inclusive, with talks from a broad range of topics for anyone who is interested in Python.

The PyBerlin organizers invited me to give the opening talk at the first event, which took place last month in February, and asked me to speak about automated testing with pytest. I was super excited and I also felt very honored to be the first speaker for the community. I wanted to prepare something special! πŸ‘¨πŸ»β€πŸ’»

New concepts stick much better with me when applied to a real problem rather than when explained with a bunch of trivial examples. So instead of using a slide deck packed with short code snippets to explain the core concepts of writing tests in Python with pytest, I decided to do some live coding on earth, a small example project, and work on a patch for an open GitHub issue.

earth is a library for organizing and running events: You can invite adventurers from around the world, see how they pack their bags and travel by airplane to the event location, and finally greet them and make introductions. πŸ•

The GitHub issue reads:

Earth 🌍 is such an important project, but I’m concerned that without adequate test coverage we may be leaving its users vulnerable. Let’s identify the features that are not yet tested and ensure we can maintain excellent coverage in the future.

This blog post is the written form of my PyBerlin talk “Customizing your pytest test suite” If you find it useful, please share it with your colleagues and Python friends! πŸ˜πŸ“

Another version of this talk was live streamed on YouTube and the recording is now available on our Mozilla YouTube channel, if you would rather watch my talk than read this rather lengthy blog post. πŸ“Ί

picture from PyBerlin

Picture taken on Feb 21, 2019 by Tammo Behrends at PyBerlin πŸ“·

About this blog post

I’ve structured this blog post as a tutorial. πŸ“

We will learn how to…

  • get started with writing automated tests with pytest
  • use pytest fixtures and markers to organize our tests
  • use a number of different CLI options for test selection
  • use powerful features of several awesome pytest plugins
  • and write our own plugin to customize our test suite

As we go over these topics, I encourage you to try out the code yourself and follow the links to the documentation if you want to learn more about specific features.

Good tutorials are difficult to write. I’ll try my best to give you enough information to complete the steps without going into too much detail. If you spot any typos, find bugs in the code examples, or have any questions, please open a new issue on GitHub. πŸ’»

I will commit code changes to the earth repository on a separate branch named increase-test-coverage and create commits for the steps in this tutorial. Should you get stuck at any point, please check out the commits on that branch and continue from there.

Let’s get started! πŸ˜„

Our task

We will work on identifying blind spots in the earth project and increasing its code coverage by developing a series of automated tests. 🌏

We need a local copy of the earth GitHub repository as well as a new Python 3.7 virtual environment with attrs installed. 🐍

Test requirements

Please also install the following Python packages into your activated virtual environment as we will be using these plugins in this tutorial:

pytest
pytest-cov
pytest-emoji
pytest-html
pytest-md
pytest-repeat
pytest-variables

Explore the project

There are a number of tests in the tests directory of the earth project. Have a look around and open a few files in your text editor to make yourself familiar with what the tests are doing. πŸ•΅οΈβ€β™€οΈ

tests/
β”œβ”€β”€ __init__.py
β”œβ”€β”€ adventurers
β”‚Β Β  β”œβ”€β”€ __init__.py
β”‚Β Β  β”œβ”€β”€ test_adventurers_01.py
β”‚Β Β  β”œβ”€β”€ test_adventurers_02.py
β”‚Β Β  └── test_adventurers_03.py
β”œβ”€β”€ earth
β”‚Β Β  β”œβ”€β”€ __init__.py
β”‚Β Β  β”œβ”€β”€ test_earth_01.py
β”‚Β Β  └── test_earth_02.py
β”œβ”€β”€ events
β”‚Β Β  β”œβ”€β”€ __init__.py
β”‚Β Β  β”œβ”€β”€ test_events_01.py
β”‚Β Β  β”œβ”€β”€ test_events_02.py
β”‚Β Β  β”œβ”€β”€ test_events_03.py
β”‚Β Β  └── test_events_04.py
β”œβ”€β”€ old
β”‚Β Β  β”œβ”€β”€ __init__.py
β”‚Β Β  └── stuff
β”‚Β Β      β”œβ”€β”€ __init__.py
β”‚Β Β      β”œβ”€β”€ test_stuff_01.py
β”‚Β Β      β”œβ”€β”€ test_stuff_02.py
β”‚Β Β      └── test_stuff_03.py
β”œβ”€β”€ travel
β”‚Β Β  β”œβ”€β”€ __init__.py
β”‚Β Β  β”œβ”€β”€ test_travel_01.py
β”‚Β Β  β”œβ”€β”€ test_travel_02.py
β”‚Β Β  └── test_travel_03.py
└── year
    β”œβ”€β”€ __init__.py
    β”œβ”€β”€ test_year_01.py
    └── test_year_02.py

7 directories, 25 files

You will find that only tests/year/test_year_02.py imports the earth library, which indicates that the majority of the existing tests do not use the earth library and can’t possibly generate code coverage.

Run the tests

Let’s start with running the existing tests as they are:

pytest --verbose
E     File "earth/tests/old/stuff/test_stuff_03.py", line 6
E       print "hello world"
E                         ^
E   SyntaxError: Missing parentheses in call to 'print'. Did you mean
    print("hello world")?

Hmm… a SyntaxError before pytest could run any of the tests? πŸ˜•

Skip a single test

When pytest tried to collect the tests, it failed to import the following file which contains a test that uses the Python 2 print statement. That’s not supported under Python 3, which caused a SyntaxError! 😐

tests/old/stuff/test_stuff_03.py
def test_numbers():
    assert 1234 == 1234


def test_hello_world():
    print "hello world"


def test_foobar():
    assert True

Let’s comment out the invalid code, add a comment so that we don’t forget to look into this later and skip the test.

tests/old/stuff/test_stuff_03.py
import pytest

def test_numbers():
    assert 1234 == 1234


@pytest.mark.skip(reason="Outdated Python syntax")
def test_hello_world():
    # TODO: Can we remove this test?
    # print "hello world"
    pass


def test_foobar():
    assert True

Run the tests (again)

Run pytest again and watch closely what happens:

pytest --verbose

You will find that we don’t see the error anymore, but the 4 tests in tests/old/stuff/test_stuff_02.py take a fairly long time to complete. 😴

Skip entire test module

Let’s generate a test coverage report using pytest-cov to see if the test module with the slow tests generates test coverage:

pytest --cov earth/
Name                   Stmts   Miss  Cover
------------------------------------------
earth/__init__.py          3      0   100%
earth/adventurers.py      71     41    42%
earth/events.py           24     12    50%
earth/travel.py           31     14    55%
earth/year.py             14      0   100%
------------------------------------------
TOTAL                    143     67    53%

Then skip the entire test module:

tests/old/stuff/test_stuff_02.py
import time
import unittest

import pytest

pytest.skip("This does not generate code coverage", allow_module_level=True)


class TestStringMethods(unittest.TestCase):
    def setUp(self):
        # A long time ago in a galaxy far, far away...
        time.sleep(2)

    def test_upper(self):
        self.assertEqual("foo".upper(), "FOO")

    def test_upper_bar(self):
        self.assertEqual("foo".upper(), "BAR")

    def test_isupper(self):
        self.assertTrue("FOO".isupper())
        self.assertFalse("Foo".isupper())

    def test_split(self):
        s = "hello world"
        self.assertEqual(s.split(), ["hello", "world"])

        # check that s.split fails when the separator is not a string
        with self.assertRaises(TypeError):
            s.split(2)

Let’s generate a new test coverage report. πŸ’»

You will see that the test file was skipped and that the test coverage remained at 53%. This means that the test does not generate any code coverage and is safe to skip.

Run the example

The README file in the earth repo contains an example for how the library can be used and the very same example is also copied to the example1.py script.

example1.py
from earth import adventurers, Event, Months


def main():
    print("Hello adventurers! πŸ•")
    print("-" * 40)

    friends = [
        adventurers.new_frog("Bruno"),
        adventurers.new_lion("Michael"),
        adventurers.new_koala("Brianna"),
        adventurers.new_tiger("Julia"),
    ]

    event = Event("PyCon US", "North America", Months.MAY)

    for adventurer in friends:
        event.invite(adventurer)

    print("-" * 40)

    for attendee in event.attendees:
        attendee.get_ready()
        attendee.travel_to(event)

    print("-" * 40)

    event.start()


if __name__ == "__main__":
    main()

Let’s run the example and see what it does:

python example1.py
Hello adventurers! πŸ•
----------------------------------------
Bruno accepted our invite! πŸ˜ƒ
Michael accepted our invite! πŸ˜ƒ
Brianna accepted our invite! πŸ˜ƒ
Julia accepted our invite! πŸ˜ƒ
----------------------------------------
🐸 Bruno is packing πŸ‘œ
🐸 Bruno is travelling: South America ✈️  North America
🦁 Michael is packing πŸ‘œ
🦁 Michael is travelling: Africa ✈️  North America
🐨 Brianna is packing πŸ‘œ
🐨 Brianna is travelling: Australia ✈️  North America
🐯 Julia is travelling: Asia ✈️  North America
----------------------------------------
Welcome to PyCon US in North America! πŸŽ‰
Let's start with introductions...πŸ’¬
🐸 Hello, my name is Bruno!
🦁 Hello, my name is Michael!
🐨 Hello, my name is Brianna!
🐯 Hello, my name is Julia!

Write a happy path test

The example from the README is super important for the adoption of our earth project. If this code snippet doesn’t work out of the box, chances are users will be frustrated and will look for alternative libraries. We definitely want to have test coverage for that. ⚠️

Let’s create a new test file and write a test based on the example. Note that we won’t perform any assertions in our test, but only test for unhandled exceptions.

I also recommend to add custom pytest markers to tests that we currently work on as it makes selecting them that much easier. We’ll use happy because our new test will be a happy path test and wip to indicate that this test is a work in progress.

tests/test_earth.py
import pytest

from earth import adventurers, Event, Months


@pytest.mark.wip
@pytest.mark.happy
def test_earth():
    print("Hello adventurers! πŸ•")
    print("-" * 40)

    friends = [
        adventurers.new_frog("Bruno"),
        adventurers.new_lion("Michael"),
        adventurers.new_koala("Brianna"),
        adventurers.new_tiger("Julia"),
    ]

    event = Event("PyCon US", "North America", Months.MAY)

    for adventurer in friends:
        event.invite(adventurer)

    print("-" * 40)

    for attendee in event.attendees:
        attendee.get_ready()
        attendee.travel_to(event)

    print("-" * 40)

    event.start()

Run only our test

We can now use the markers to select our test and see if it passes:

pytest -m wip
============================ test session starts =============================
collected 50 items / 49 deselected / 1 skipped

tests/test_earth.py .

============= 1 passed, 1 skipped, 49 deselected in 0.11 seconds =============

Yay it works! πŸ˜„

Start using pytest fixtures

Now that we know that the test passes, let’s refactor our code and separate out test dependencies from the test implementation using pytest fixtures and also remove the prints as they are not required for the test.

tests/test_earth.py
import pytest

from earth import adventurers, Event, Months


@pytest.fixture
def event():
    return Event("PyCon US", "North America", Months.MAY)


@pytest.fixture
def friends():
    return [
        adventurers.new_frog("Bruno"),
        adventurers.new_lion("Michael"),
        adventurers.new_koala("Brianna"),
        adventurers.new_tiger("Julia"),
    ]


@pytest.mark.wip
@pytest.mark.happy
def test_earth(event, friends):
    for adventurer in friends:
        event.invite(adventurer)

    for attendee in event.attendees:
        attendee.get_ready()
        attendee.travel_to(event)

    event.start()

Generate an HTML coverage report

It’s a good idea to check whether adding this test changed the code coverage. This time we will run the tests again and generate an HTML coverage report to find out not only the percentage of lines covered by tests, but also which lines are not executed when we run the tests.

pytest --cov earth/ --cov-report html

This creates a new file at htmlcov/index.html πŸ“„

When you open that coverage report in your web browser, you will see that our test coverage increased quite drastically. It is now at 84% - woohoo! πŸ˜ƒ

Missing coverage

The HTML coverage report also shows code coverage for individual files. We can use this information to work on increasing the code coverage for earth even more. The first file in our report adventurers.py has a coverage of 80%. If you click on the link in the HTML, you will see a number of red lines, which means we don’t have coverage for them.

The following functions are not called in any of our tests:

  • new_panda()
  • new_bear()
  • new_fox()

Let’s do something about that! 🐼🐻🦊

Write another test

We’ll leave the test for the example from the README as is and write a new test for a larger group of adventurers. Rename the fixture friends to small_group and rename the test from test_earth to test_small_group.

Create a new fixture large_group based on small_group but return a complete list of adventurers and copy test_small_group to a new test_large_group test.

tests/test_earth.py
import pytest

from earth import adventurers, Event, Months


@pytest.fixture
def event():
    return Event("PyCon US", "North America", Months.MAY)


@pytest.fixture
def small_group():
    return [
        adventurers.new_frog("Bruno"),
        adventurers.new_lion("Michael"),
        adventurers.new_koala("Brianna"),
        adventurers.new_tiger("Julia"),
    ]


@pytest.fixture
def large_group():
    return [
        adventurers.new_frog("Bruno"),
        adventurers.new_panda("Po"),
        adventurers.new_fox("Dave"),
        adventurers.new_lion("Michael"),
        adventurers.new_koala("Brianna"),
        adventurers.new_tiger("Julia"),
        adventurers.new_fox("Raphael"),
        adventurers.new_fox("Caro"),
        adventurers.new_bear("Chris"),
        # Bears in warm climates don't hibernate 🐻
        adventurers.new_bear("Danny", availability=[*Months]),
        adventurers.new_bear("Audrey", availability=[*Months]),
    ]


@pytest.mark.wip
@pytest.mark.happy
def test_small_group(event, small_group):
    for adventurer in small_group:
        event.invite(adventurer)

    for attendee in event.attendees:
        attendee.get_ready()
        attendee.travel_to(event)

    event.start()


@pytest.mark.wip
@pytest.mark.happy
def test_large_group(event, large_group):
    for adventurer in small_group:
        event.invite(adventurer)

    for attendee in event.attendees:
        attendee.get_ready()
        attendee.travel_to(event)

    event.start()

Run the tests (again)

Let’s run our work in progress tests again and see if the new test passes:

pytest -m wip
============================ test session starts =============================
collected 51 items / 49 deselected / 1 skipped / 1 selected

tests/test_earth.py .F                                                  [100%]

================================== FAILURES ==================================
______________________________ test_large_group ______________________________

    @pytest.mark.wip
    @pytest.mark.happy
    def test_large_group(event, large_group):
>       for adventurer in small_group:
E       TypeError: 'function' object is not iterable

tests/test_earth.py:55: TypeError
======== 1 failed, 1 passed, 1 skipped, 49 deselected in 0.12 seconds ========

Wait, what? Our test fails with a TypeError?! πŸ€”

Specify fixture names

We forgot to rename small_group in the for loop in test_large_group πŸ€¦β€β™‚

However because we define a function with name small_group in the module scope and everything in Python is an object, Python tries to iterate over our fixture function instead of raising a NameError.

I recommend overwriting the auto-generated fixture name to avoid this problem.

tests/test_earth.py
import pytest

from earth import adventurers, Event, Months


@pytest.fixture(name="event")
def fixture_event():
    return Event("PyCon US", "North America", Months.MAY)


@pytest.fixture(name="small_group")
def fixture_small_group():
    return [
        adventurers.new_frog("Bruno"),
        adventurers.new_lion("Michael"),
        adventurers.new_koala("Brianna"),
        adventurers.new_tiger("Julia"),
    ]


@pytest.fixture(name="large_group")
def fixture_large_group():
    return [
        adventurers.new_frog("Bruno"),
        adventurers.new_panda("Po"),
        adventurers.new_fox("Dave"),
        adventurers.new_lion("Michael"),
        adventurers.new_koala("Brianna"),
        adventurers.new_tiger("Julia"),
        adventurers.new_fox("Raphael"),
        adventurers.new_fox("Caro"),
        adventurers.new_bear("Chris"),
        # Bears in warm climates don't hibernate 🐻
        adventurers.new_bear("Danny", availability=[*Months]),
        adventurers.new_bear("Audrey", availability=[*Months]),
    ]


@pytest.mark.wip
@pytest.mark.happy
def test_small_group(event, small_group):
    for adventurer in small_group:
        event.invite(adventurer)

    for attendee in event.attendees:
        attendee.get_ready()
        attendee.travel_to(event)

    event.start()


@pytest.mark.wip
@pytest.mark.happy
def test_large_group(event, large_group):
    for adventurer in small_group:
        event.invite(adventurer)

    for attendee in event.attendees:
        attendee.get_ready()
        attendee.travel_to(event)

    event.start()

Now if we run the tests again, we will see a much more helpful error message. I can’t stress enough how important this is. With Python’s duck-typing capabilities, you might not even realize that your tests accidentally pass only because you forgot to add your fixture to your test’s argument list. ⚠️

pytest -m wip
============================ test session starts =============================
collected 51 items / 49 deselected / 1 skipped / 1 selected

tests/test_earth.py .F                                                  [100%]

================================== FAILURES ==================================
______________________________ test_large_group ______________________________

    @pytest.mark.wip
    @pytest.mark.happy
    def test_large_group(event, large_group):
>       for adventurer in small_group:
E       NameError: name 'small_group' is not defined

tests/test_earth.py:55: NameError
======== 1 failed, 1 passed, 1 skipped, 49 deselected in 0.13 seconds ========

Let’s fix that and update the variable name in test_large_group, before we run the tests again. πŸ’»

Show slow running tests

If you run the tests, you will notice that the suite now takes a long time to complete. pytest comes with a really neat CLI flag to run the tests and show a sorted list of slow tests. Let’s show only the top 2 slowest tests ⏱

pytest -m wip --durations 2
============================ test session starts =============================
collected 51 items / 49 deselected / 1 skipped / 1 selected

tests/test_earth.py .F                                                  [100%]

================================== FAILURES ==================================
______________________________ test_large_group ______________________________

    @pytest.mark.wip
    @pytest.mark.happy
    def test_large_group(event, large_group):
        for adventurer in large_group:
            event.invite(adventurer)

        for attendee in event.attendees:
            attendee.get_ready()
            attendee.travel_to(event)

>       event.start()

tests/test_earth.py:62:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
earth/events.py:34: MissingAttendee
--------------------------- Captured stdout call ----------------------------
Bruno accepted our invite! πŸ˜ƒ
Po accepted our invite! πŸ˜ƒ
Dave accepted our invite! πŸ˜ƒ
Michael accepted our invite! πŸ˜ƒ
Brianna accepted our invite! πŸ˜ƒ
Julia accepted our invite! πŸ˜ƒ
Raphael accepted our invite! πŸ˜ƒ
Caro accepted our invite! πŸ˜ƒ
Chris is not available in May! 😒
Danny accepted our invite! πŸ˜ƒ
Audrey accepted our invite! πŸ˜ƒ
🐸 Bruno is packing πŸ‘œ
🐸 Bruno is travelling: South America ✈️  North America
🐼 Po is eating... 🌱
🐼 Po is eating... 🌱
🐼 Po is eating... 🌱
🐼 Po is eating... 🌱
🐼 Po is packing πŸ‘œ
🐼 Po is travelling: Asia ✈️  North America
🦊 Dave is packing πŸ‘œ
🦊 Dave's flight was cancelled 😞 Problems at Berlin Airport 🚧
🦁 Michael is packing πŸ‘œ
🦁 Michael is travelling: Africa ✈️  North America
🐨 Brianna is packing πŸ‘œ
🐨 Brianna is travelling: Australia ✈️  North America
🐯 Julia is travelling: Asia ✈️  North America
🦊 Raphael is packing πŸ‘œ
🦊 Raphael's flight was cancelled 😞 Problems at Berlin Airport 🚧
🦊 Caro is packing πŸ‘œ
🦊 Caro is travelling: Europe ✈️  North America
🐻 Danny is packing πŸ‘œ
🐻 Audrey is packing πŸ‘œ
Welcome to PyCon US in North America! πŸŽ‰
Let's start with introductions...πŸ’¬
🐸 Hello, my name is Bruno!
🐼 Hello, my name is Po!
========================== slowest 2 test durations ==========================
20.01s call     tests/test_earth.py::test_large_group

(0.00 durations hidden.  Use -vv to show these durations.)
======= 1 failed, 1 passed, 1 skipped, 49 deselected in 20.16 seconds ========

The new test eventually failed after taking more than 20 seconds to complete. ⏱

While the --durations CLI flag might not be super helpful with only two tests and we already knew that the first new test passed in less than one second in a previous run, it’s super useful for larger test suites.

Add marker documentation

Let’s add a new custom pytest marker slow to test_large_group. It’s good practice to write documentation for custom markers and provide a short description for every marker, that explains the characteristics of tests with that marker. πŸ“

Add a pytest configuration file with a markers section:

pytest.ini
[pytest]
markers =
    slow: tests that take a long time to complete.

Now if you run pytest --markers you will see the information about pytest’s markers and also our custom one.

Show output

We know which test is slow, but we don’t really know why that is. By default pytest captures any output sent to stdout and stderr. Let’s run the slow test again with capturing disabled, so that we see prints while the test runs.

pytest -s -m slow
============================ test session starts =============================
collected 51 items / 50 deselected / 1 skipped

tests/test_earth.py Bruno accepted our invite! πŸ˜ƒ
Po accepted our invite! πŸ˜ƒ
Dave accepted our invite! πŸ˜ƒ
Michael accepted our invite! πŸ˜ƒ
Brianna accepted our invite! πŸ˜ƒ
Julia accepted our invite! πŸ˜ƒ
Raphael accepted our invite! πŸ˜ƒ
Caro accepted our invite! πŸ˜ƒ
Chris is not available in May! 😒
Danny accepted our invite! πŸ˜ƒ
Audrey accepted our invite! πŸ˜ƒ
🐸 Bruno is packing πŸ‘œ
🐸 Bruno is travelling: South America ✈️  North America
🐼 Po is eating... 🌱
🐼 Po is eating... 🌱
🐼 Po is eating... 🌱
🐼 Po is eating... 🌱
🐼 Po is packing πŸ‘œ
🐼 Po is travelling: Asia ✈️  North America
🦊 Dave is packing πŸ‘œ
🦊 Dave's flight was cancelled 😞 Problems at Berlin Airport 🚧
🦁 Michael is packing πŸ‘œ
🦁 Michael is travelling: Africa ✈️  North America
🐨 Brianna is packing πŸ‘œ
🐨 Brianna is travelling: Australia ✈️  North America
🐯 Julia is travelling: Asia ✈️  North America
🦊 Raphael is packing πŸ‘œ
🦊 Raphael's flight was cancelled 😞 Problems at Berlin Airport 🚧
🦊 Caro is packing πŸ‘œ
🦊 Caro is travelling: Europe ✈️  North America
🐻 Danny is packing πŸ‘œ
🐻 Audrey is packing πŸ‘œ
Welcome to PyCon US in North America! πŸŽ‰
Let's start with introductions...πŸ’¬
🐸 Hello, my name is Bruno!
🐼 Hello, my name is Po!
F

================================== FAILURES ==================================
______________________________ test_large_group ______________________________

    @pytest.mark.wip
    @pytest.mark.slow
    @pytest.mark.happy
    def test_large_group(event, large_group):
        for adventurer in large_group:
            event.invite(adventurer)

        for attendee in event.attendees:
            attendee.get_ready()
            attendee.travel_to(event)

>       event.start()

tests/test_earth.py:63:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

 def start(self):
     print(f"Welcome to {self.name} in {self.location}! πŸŽ‰")
     print(f"Let's start with introductions...πŸ’¬")

     for attendee in self.attendees:
         if attendee.location != self.location:
>            raise MissingAttendee(f"Oh no! {attendee.name} is not here! 😟")
E            earth.events.MissingAttendee: Oh no! Dave is not here! 😟

earth/events.py:34: MissingAttendee
============ 1 failed, 1 skipped, 50 deselected in 20.17 seconds =============

Our slow test failed, but let’s focus on the test duration for now. If you tried this out for yourself, you could see that 🐼 Po is eating... 🌱 for several seconds multiple times!

Random fact from the nature documentary Earth: One Amazing Day πŸ˜‚

Pandas are very fussy when it comes to their diet. Bamboo is almost the only thing they eat. But because bamboo is not very nutritious, pandas need to eat up to 14 hours a day to get enough energy to feed themselves. 🐼

Back to our tests…

Write another test

We have one test that does not use all of the adventurers.py functions, but is really fast. We have one test that uses all of the adventurers.py functions, but is rather slow. The ideal test would be one that uses as many adventurers.py functions as possible and is really fast. πŸ’¨

Let’s write a new pytest fixture no_pandas_group and another test named test_no_pandas_group:

tests/test_earth.py
import pytest

from earth import adventurers, Event, Months


@pytest.fixture(name="event")
def fixture_event():
    return Event("PyCon US", "North America", Months.MAY)


@pytest.fixture(name="small_group")
def fixture_small_group():
    return [
        adventurers.new_frog("Bruno"),
        adventurers.new_lion("Michael"),
        adventurers.new_koala("Brianna"),
        adventurers.new_tiger("Julia"),
    ]


@pytest.fixture(name="large_group")
def fixture_large_group():
    return [
        adventurers.new_frog("Bruno"),
        adventurers.new_panda("Po"),
        adventurers.new_fox("Dave"),
        adventurers.new_lion("Michael"),
        adventurers.new_koala("Brianna"),
        adventurers.new_tiger("Julia"),
        adventurers.new_fox("Raphael"),
        adventurers.new_fox("Caro"),
        adventurers.new_bear("Chris"),
        # Bears in warm climates don't hibernate 🐻
        adventurers.new_bear("Danny", availability=[*Months]),
        adventurers.new_bear("Audrey", availability=[*Months]),
    ]


@pytest.fixture(name="no_pandas_group")
def fixture_no_pandas_group():
    return [
        adventurers.new_frog("Bruno"),
        adventurers.new_fox("Dave"),
        adventurers.new_lion("Michael"),
        adventurers.new_koala("Brianna"),
        adventurers.new_tiger("Julia"),
        adventurers.new_fox("Raphael"),
        adventurers.new_fox("Caro"),
        adventurers.new_bear("Chris"),
        # Bears in warm climates don't hibernate 🐻
        adventurers.new_bear("Danny", availability=[*Months]),
        adventurers.new_bear("Audrey", availability=[*Months]),
    ]


@pytest.mark.wip
@pytest.mark.happy
def test_small_group(event, small_group):
    for adventurer in small_group:
        event.invite(adventurer)

    for attendee in event.attendees:
        attendee.get_ready()
        attendee.travel_to(event)

    event.start()


@pytest.mark.wip
@pytest.mark.slow
@pytest.mark.happy
def test_large_group(event, large_group):
    for adventurer in large_group:
        event.invite(adventurer)

    for attendee in event.attendees:
        attendee.get_ready()
        attendee.travel_to(event)

    event.start()


@pytest.mark.wip
@pytest.mark.happy
def test_no_pandas_group(event, no_pandas_group):
    for adventurer in no_pandas_group:
        event.invite(adventurer)

    for attendee in event.attendees:
        attendee.get_ready()
        attendee.travel_to(event)

    event.start()

Test selection via marker expression

We now have three different tests and added custom markers to them. Let’s run all wip tests that are not slow. πŸ’»

pytest -m "wip and not slow"
============================ test session starts =============================
collected 52 items / 50 deselected / 1 skipped / 1 selected

tests/test_earth.py ..                                                  [100%]

============= 2 passed, 1 skipped, 50 deselected in 0.19 seconds =============

Hmm… both tests passed. πŸ€”

Repeating tests

Remember in our previous test run the slow test failed due to an unhandled exception. According to the error our fox adventurer “Dave” didn’t make it to PyCon. 🦊

earth.events.MissingAttendee: Oh no! Dave is not here! 😟

It’s rather suspicious that the test_no_pandas_group passed, when the only difference to the test_large_group test is that it doesn’t have any pandas. This might be because tests with foxes are flaky and only fail under certain conditions.

We can check this by running our tests multiple times with pytest-repeat. Let’s run our fast, wip tests 10 times.

pytest -v -m "wip and not slow" --count 10
============================ test session starts =============================
collecting ... collected 520 items / 500 deselected / 1 skipped / 19 selected

tests/test_earth.py::test_small_group[1/10] PASSED                      [  5%]
tests/test_earth.py::test_small_group[2/10] PASSED                      [ 10%]
tests/test_earth.py::test_small_group[3/10] PASSED                      [ 15%]
tests/test_earth.py::test_small_group[4/10] PASSED                      [ 20%]
tests/test_earth.py::test_small_group[5/10] PASSED                      [ 25%]
tests/test_earth.py::test_small_group[6/10] PASSED                      [ 30%]
tests/test_earth.py::test_small_group[7/10] PASSED                      [ 35%]
tests/test_earth.py::test_small_group[8/10] PASSED                      [ 40%]
tests/test_earth.py::test_small_group[9/10] PASSED                      [ 45%]
tests/test_earth.py::test_small_group[10/10] PASSED                     [ 50%]
tests/test_earth.py::test_no_pandas_group[1/10] FAILED                  [ 55%]
tests/test_earth.py::test_no_pandas_group[2/10] FAILED                  [ 60%]
tests/test_earth.py::test_no_pandas_group[3/10] FAILED                  [ 65%]
tests/test_earth.py::test_no_pandas_group[4/10] FAILED                  [ 70%]
tests/test_earth.py::test_no_pandas_group[5/10] FAILED                  [ 75%]
tests/test_earth.py::test_no_pandas_group[6/10] PASSED                  [ 80%]
tests/test_earth.py::test_no_pandas_group[7/10] FAILED                  [ 85%]
tests/test_earth.py::test_no_pandas_group[8/10] FAILED                  [ 90%]
tests/test_earth.py::test_no_pandas_group[9/10] FAILED                  [ 95%]
tests/test_earth.py::test_no_pandas_group[10/10] FAILED                 [100%]

======= 9 failed, 11 passed, 1 skipped, 500 deselected in 0.30 seconds =======

As we can see from the CLI report, the test_small_group test passed in 10 out of 10 runs, whereas test_no_pandas_group passed only on 1 out of 10 runs. We might not know the exact reason for why that is, but we can tell from the stdout in the CLI report that there appear to be problems at TXL airport…sounds familiar? πŸ€·β€β™€

Marking flaky tests

We can use the built-in xfail marker from pytest for our flaky test: Tests marked with xfail are expected to fail due to the specified reason. If they fail, they will be reported as XFAIL, however if they unexpectedly pass they will be reported as XPASS.

tests/test_earth.py
import pytest

from earth import adventurers, Event, Months


@pytest.fixture(name="event")
def fixture_event():
    return Event("PyCon US", "North America", Months.MAY)


@pytest.fixture(name="small_group")
def fixture_small_group():
    return [
        adventurers.new_frog("Bruno"),
        adventurers.new_lion("Michael"),
        adventurers.new_koala("Brianna"),
        adventurers.new_tiger("Julia"),
    ]


@pytest.fixture(name="large_group")
def fixture_large_group():
    return [
        adventurers.new_frog("Bruno"),
        adventurers.new_panda("Po"),
        adventurers.new_fox("Dave"),
        adventurers.new_lion("Michael"),
        adventurers.new_koala("Brianna"),
        adventurers.new_tiger("Julia"),
        adventurers.new_fox("Raphael"),
        adventurers.new_fox("Caro"),
        adventurers.new_bear("Chris"),
        # Bears in warm climates don't hibernate 🐻
        adventurers.new_bear("Danny", availability=[*Months]),
        adventurers.new_bear("Audrey", availability=[*Months]),
    ]


@pytest.fixture(name="no_pandas_group")
def fixture_no_pandas_group():
    return [
        adventurers.new_frog("Bruno"),
        adventurers.new_fox("Dave"),
        adventurers.new_lion("Michael"),
        adventurers.new_koala("Brianna"),
        adventurers.new_tiger("Julia"),
        adventurers.new_fox("Raphael"),
        adventurers.new_fox("Caro"),
        adventurers.new_bear("Chris"),
        # Bears in warm climates don't hibernate 🐻
        adventurers.new_bear("Danny", availability=[*Months]),
        adventurers.new_bear("Audrey", availability=[*Months]),
    ]


@pytest.mark.wip
@pytest.mark.happy
def test_small_group(event, small_group):
    for adventurer in small_group:
        event.invite(adventurer)

    for attendee in event.attendees:
        attendee.get_ready()
        attendee.travel_to(event)

    event.start()


@pytest.mark.wip
@pytest.mark.slow
@pytest.mark.happy
@pytest.mark.xfail(reason="Problems with TXL airport")
def test_large_group(event, large_group):
    for adventurer in large_group:
        event.invite(adventurer)

    for attendee in event.attendees:
        attendee.get_ready()
        attendee.travel_to(event)

    event.start()


@pytest.mark.wip
@pytest.mark.happy
@pytest.mark.xfail(reason="Problems with TXL airport")
def test_no_pandas_group(event, no_pandas_group):
    for adventurer in no_pandas_group:
        event.invite(adventurer)

    for attendee in event.attendees:
        attendee.get_ready()
        attendee.travel_to(event)

    event.start()

Run the tests again:

pytest -v -m "wip and not slow" --count 10
============================ test session starts =============================
collecting ... collected 520 items / 500 deselected / 1 skipped / 19 selected

tests/test_earth.py::test_small_group[1/10] PASSED                      [  5%]
tests/test_earth.py::test_small_group[2/10] PASSED                      [ 10%]
tests/test_earth.py::test_small_group[3/10] PASSED                      [ 15%]
tests/test_earth.py::test_small_group[4/10] PASSED                      [ 20%]
tests/test_earth.py::test_small_group[5/10] PASSED                      [ 25%]
tests/test_earth.py::test_small_group[6/10] PASSED                      [ 30%]
tests/test_earth.py::test_small_group[7/10] PASSED                      [ 35%]
tests/test_earth.py::test_small_group[8/10] PASSED                      [ 40%]
tests/test_earth.py::test_small_group[9/10] PASSED                      [ 45%]
tests/test_earth.py::test_small_group[10/10] PASSED                     [ 50%]
tests/test_earth.py::test_no_pandas_group[1/10] XFAIL                   [ 55%]
tests/test_earth.py::test_no_pandas_group[2/10] XFAIL                   [ 60%]
tests/test_earth.py::test_no_pandas_group[3/10] XPASS                   [ 65%]
tests/test_earth.py::test_no_pandas_group[4/10] XFAIL                   [ 70%]
tests/test_earth.py::test_no_pandas_group[5/10] XFAIL                   [ 75%]
tests/test_earth.py::test_no_pandas_group[6/10] XPASS                   [ 80%]
tests/test_earth.py::test_no_pandas_group[7/10] XFAIL                   [ 85%]
tests/test_earth.py::test_no_pandas_group[8/10] XFAIL                   [ 90%]
tests/test_earth.py::test_no_pandas_group[9/10] XFAIL                   [ 95%]
tests/test_earth.py::test_no_pandas_group[10/10] XFAIL                  [100%]

= 10 passed, 1 skipped, 500 deselected, 8 xfailed, 2 xpassed in 0.24 seconds =

Check code coverage

Now let’s run all of our tests and check the current code coverage. 🌏

pytest --cov earth/
Name                   Stmts   Miss  Cover
------------------------------------------
earth/__init__.py          3      0   100%
earth/adventurers.py      71      0   100%
earth/events.py           24      0   100%
earth/travel.py           31      3    90%
earth/year.py             14      0   100%
------------------------------------------
TOTAL                    143      3    98%

Yay, we’re at 98% code coverage for the earth project! That’s fantastic! 😁

Next steps

We’ve completed our task of increasing the code coverage through automated tests, but we have some more work to do! 🚧

Running all of the tests including the slow test isn’t great for developer productivity. It makes sense to write a custom pytest plugin that skips tests marked as slow by default and only includes slow tests if we run pytest with a custom CLI option.

Also, did you notice that we have three test cases, but apart from the markers and the different fixtures the test functions itself are identical? That’s a lot of redundancy. πŸ˜”

The good news is that pytest comes with a marker which solves this exact problem. Parametrized tests and custom plugins will be our topics in the next part of this tutorial!

I hope you’re enjoying this tutorial as much as I enjoyed giving my talk and writing the first part of this blog post. We’ll do some really cool stuff with pytest in the next part of this tutorial. As usual, I will post a link on my Twitter. Stay tuned! πŸ“

Update: Customizing your pytest test suite (part 2) is now online! πŸ‘¨πŸ»β€πŸ’»


Back to posts