Raphael Aurich
PyBerlin is a new meetup community in Berlin, Germany that strives to be open and inclusive, with talks from a broad range of topics for anyone who is interested in Python.
The PyBerlin organizers invited me to give the opening talk at the first event, which took place last month in February, and asked me to speak about automated testing with pytest. I was super excited and I also felt very honored to be the first speaker for the community. I wanted to prepare something special! 👨🏻💻
New concepts stick much better with me when applied to a real problem rather than when explained with a bunch of trivial examples. So instead of using a slide deck packed with short code snippets to explain the core concepts of writing tests in Python with pytest, I decided to do some live coding on earth, a small example project, and work on a patch for an open GitHub issue.
earth is a library for organizing and running events: You can invite adventurers from around the world, see how they pack their bags and travel by airplane to the event location, and finally greet them and make introductions. 🏕
The GitHub issue reads:
Earth 🌍 is such an important project, but I’m concerned that without adequate test coverage we may be leaving its users vulnerable. Let’s identify the features that are not yet tested and ensure we can maintain excellent coverage in the future.
This blog post is the written form of my PyBerlin talk “Customizing your pytest test suite” If you find it useful, please share it with your colleagues and Python friends! 😁📝
Another version of this talk was live streamed on YouTube and the recording is now available on our Mozilla YouTube channel, if you would rather watch my talk than read this rather lengthy blog post. 📺
Picture taken on Feb 21, 2019 by Tammo Behrends at PyBerlin 📷
About this blog post
I’ve structured this blog post as a tutorial. 📝
We will learn how to…
- get started with writing automated tests with pytest
- use pytest fixtures and markers to organize our tests
- use a number of different CLI options for test selection
- use powerful features of several awesome pytest plugins
- and write our own plugin to customize our test suite
As we go over these topics, I encourage you to try out the code yourself and follow the links to the documentation if you want to learn more about specific features.
Good tutorials are difficult to write. I’ll try my best to give you enough information to complete the steps without going into too much detail. If you spot any typos, find bugs in the code examples, or have any questions, please open a new issue on GitHub. 💻
I will commit code changes to the earth repository on a separate branch named increase-test-coverage and create commits for the steps in this tutorial. Should you get stuck at any point, please check out the commits on that branch and continue from there.
Let’s get started! 😄
Our task
We will work on identifying blind spots in the earth project and increasing its code coverage by developing a series of automated tests. 🌏
We need a local copy of the earth GitHub repository as well as a new Python 3.7 virtual environment with attrs installed. 🐍
Test requirements
Please also install the following Python packages into your activated virtual environment as we will be using these plugins in this tutorial:
pytest
pytest-cov
pytest-emoji
pytest-html
pytest-md
pytest-repeat
pytest-variables
Explore the project
There are a number of tests in the tests directory of the earth project. Have a look around and open a few files in your text editor to make yourself familiar with what the tests are doing. 🕵️♀️
tests/
├── __init__.py
├── adventurers
│ ├── __init__.py
│ ├── test_adventurers_01.py
│ ├── test_adventurers_02.py
│ └── test_adventurers_03.py
├── earth
│ ├── __init__.py
│ ├── test_earth_01.py
│ └── test_earth_02.py
├── events
│ ├── __init__.py
│ ├── test_events_01.py
│ ├── test_events_02.py
│ ├── test_events_03.py
│ └── test_events_04.py
├── old
│ ├── __init__.py
│ └── stuff
│ ├── __init__.py
│ ├── test_stuff_01.py
│ ├── test_stuff_02.py
│ └── test_stuff_03.py
├── travel
│ ├── __init__.py
│ ├── test_travel_01.py
│ ├── test_travel_02.py
│ └── test_travel_03.py
└── year
├── __init__.py
├── test_year_01.py
└── test_year_02.py
7 directories, 25 files
You will find that only tests/year/test_year_02.py imports the earth
library, which indicates that the majority of the existing tests do not use the
earth library and can’t possibly generate code coverage.
Run the tests
Let’s start with running the existing tests as they are:
pytest --verbose
E File "earth/tests/old/stuff/test_stuff_03.py", line 6
E print "hello world"
E ^
E SyntaxError: Missing parentheses in call to 'print'. Did you mean
print("hello world")?
Hmm… a SyntaxError before pytest could run any of the tests? 😕
Skip a single test
When pytest tried to collect the tests, it failed to import the following file which contains a test that uses the Python 2 print statement. That’s not supported under Python 3, which caused a SyntaxError! 😐
tests/old/stuff/test_stuff_03.py
def test_numbers():
assert 1234 == 1234
def test_hello_world():
print "hello world"
def test_foobar():
assert True
Let’s comment out the invalid code, add a comment so that we don’t forget to look into this later and skip the test.
tests/old/stuff/test_stuff_03.py
import pytest
def test_numbers():
assert 1234 == 1234
@pytest.mark.skip(reason="Outdated Python syntax")
def test_hello_world():
# TODO: Can we remove this test?
# print "hello world"
pass
def test_foobar():
assert True
Run the tests (again)
Run pytest again and watch closely what happens:
pytest --verbose
You will find that we don’t see the error anymore, but the 4 tests in
tests/old/stuff/test_stuff_02.py take a fairly long time to complete. 😴
Skip entire test module
Let’s generate a test coverage report using pytest-cov to see if the test module with the slow tests generates test coverage:
pytest --cov earth/
Name Stmts Miss Cover
------------------------------------------
earth/__init__.py 3 0 100%
earth/adventurers.py 71 41 42%
earth/events.py 24 12 50%
earth/travel.py 31 14 55%
earth/year.py 14 0 100%
------------------------------------------
TOTAL 143 67 53%
Then skip the entire test module:
tests/old/stuff/test_stuff_02.py
import time
import unittest
import pytest
pytest.skip("This does not generate code coverage", allow_module_level=True)
class TestStringMethods(unittest.TestCase):
def setUp(self):
# A long time ago in a galaxy far, far away...
time.sleep(2)
def test_upper(self):
self.assertEqual("foo".upper(), "FOO")
def test_upper_bar(self):
self.assertEqual("foo".upper(), "BAR")
def test_isupper(self):
self.assertTrue("FOO".isupper())
self.assertFalse("Foo".isupper())
def test_split(self):
s = "hello world"
self.assertEqual(s.split(), ["hello", "world"])
# check that s.split fails when the separator is not a string
with self.assertRaises(TypeError):
s.split(2)
Let’s generate a new test coverage report. 💻
You will see that the test file was skipped and that the test coverage remained at 53%. This means that the test does not generate any code coverage and is safe to skip.
Run the example
The README file in the earth repo contains an example for how the library can
be used and the very same example is also copied to the example1.py script.
example1.py
from earth import adventurers, Event, Months
def main():
print("Hello adventurers! 🏕")
print("-" * 40)
friends = [
adventurers.new_frog("Bruno"),
adventurers.new_lion("Michael"),
adventurers.new_koala("Brianna"),
adventurers.new_tiger("Julia"),
]
event = Event("PyCon US", "North America", Months.MAY)
for adventurer in friends:
event.invite(adventurer)
print("-" * 40)
for attendee in event.attendees:
attendee.get_ready()
attendee.travel_to(event)
print("-" * 40)
event.start()
if __name__ == "__main__":
main()
Let’s run the example and see what it does:
python example1.py
Hello adventurers! 🏕
----------------------------------------
Bruno accepted our invite! 😃
Michael accepted our invite! 😃
Brianna accepted our invite! 😃
Julia accepted our invite! 😃
----------------------------------------
🐸 Bruno is packing 👜
🐸 Bruno is travelling: South America ✈️ North America
🦁 Michael is packing 👜
🦁 Michael is travelling: Africa ✈️ North America
🐨 Brianna is packing 👜
🐨 Brianna is travelling: Australia ✈️ North America
🐯 Julia is travelling: Asia ✈️ North America
----------------------------------------
Welcome to PyCon US in North America! 🎉
Let's start with introductions...💬
🐸 Hello, my name is Bruno!
🦁 Hello, my name is Michael!
🐨 Hello, my name is Brianna!
🐯 Hello, my name is Julia!
Write a happy path test
The example from the README is super important for the adoption of our earth project. If this code snippet doesn’t work out of the box, chances are users will be frustrated and will look for alternative libraries. We definitely want to have test coverage for that. ⚠️
Let’s create a new test file and write a test based on the example. Note that we won’t perform any assertions in our test, but only test for unhandled exceptions.
I also recommend to add custom pytest markers to tests that we
currently work on as it makes selecting them that much easier. We’ll use
happy because our new test will be a happy path test and wip to
indicate that this test is a work in progress.
tests/test_earth.py
import pytest
from earth import adventurers, Event, Months
@pytest.mark.wip
@pytest.mark.happy
def test_earth():
print("Hello adventurers! 🏕")
print("-" * 40)
friends = [
adventurers.new_frog("Bruno"),
adventurers.new_lion("Michael"),
adventurers.new_koala("Brianna"),
adventurers.new_tiger("Julia"),
]
event = Event("PyCon US", "North America", Months.MAY)
for adventurer in friends:
event.invite(adventurer)
print("-" * 40)
for attendee in event.attendees:
attendee.get_ready()
attendee.travel_to(event)
print("-" * 40)
event.start()
Run only our test
We can now use the markers to select our test and see if it passes:
pytest -m wip
============================ test session starts =============================
collected 50 items / 49 deselected / 1 skipped
tests/test_earth.py .
============= 1 passed, 1 skipped, 49 deselected in 0.11 seconds =============
Yay it works! 😄
Start using pytest fixtures
Now that we know that the test passes, let’s refactor our code and separate out test dependencies from the test implementation using pytest fixtures and also remove the prints as they are not required for the test.
tests/test_earth.py
import pytest
from earth import adventurers, Event, Months
@pytest.fixture
def event():
return Event("PyCon US", "North America", Months.MAY)
@pytest.fixture
def friends():
return [
adventurers.new_frog("Bruno"),
adventurers.new_lion("Michael"),
adventurers.new_koala("Brianna"),
adventurers.new_tiger("Julia"),
]
@pytest.mark.wip
@pytest.mark.happy
def test_earth(event, friends):
for adventurer in friends:
event.invite(adventurer)
for attendee in event.attendees:
attendee.get_ready()
attendee.travel_to(event)
event.start()
Generate an HTML coverage report
It’s a good idea to check whether adding this test changed the code coverage. This time we will run the tests again and generate an HTML coverage report to find out not only the percentage of lines covered by tests, but also which lines are not executed when we run the tests.
pytest --cov earth/ --cov-report html
This creates a new file at htmlcov/index.html 📄
When you open that coverage report in your web browser, you will see that our test coverage increased quite drastically. It is now at 84% - woohoo! 😃
Missing coverage
The HTML coverage report also shows code coverage for individual files. We can
use this information to work on increasing the code coverage for earth even
more. The first file in our report adventurers.py has a coverage of 80%. If
you click on the link in the HTML, you will see a number of red lines, which
means we don’t have coverage for them.
The following functions are not called in any of our tests:
new_panda()new_bear()new_fox()
Let’s do something about that! 🐼🐻🦊
Write another test
We’ll leave the test for the example from the README as is and write a new test
for a larger group of adventurers. Rename the fixture friends to
small_group and rename the test from test_earth to
test_small_group.
Create a new fixture large_group based on small_group but return a
complete list of adventurers and copy test_small_group to a new
test_large_group test.
tests/test_earth.py
import pytest
from earth import adventurers, Event, Months
@pytest.fixture
def event():
return Event("PyCon US", "North America", Months.MAY)
@pytest.fixture
def small_group():
return [
adventurers.new_frog("Bruno"),
adventurers.new_lion("Michael"),
adventurers.new_koala("Brianna"),
adventurers.new_tiger("Julia"),
]
@pytest.fixture
def large_group():
return [
adventurers.new_frog("Bruno"),
adventurers.new_panda("Po"),
adventurers.new_fox("Dave"),
adventurers.new_lion("Michael"),
adventurers.new_koala("Brianna"),
adventurers.new_tiger("Julia"),
adventurers.new_fox("Raphael"),
adventurers.new_fox("Caro"),
adventurers.new_bear("Chris"),
# Bears in warm climates don't hibernate 🐻
adventurers.new_bear("Danny", availability=[*Months]),
adventurers.new_bear("Audrey", availability=[*Months]),
]
@pytest.mark.wip
@pytest.mark.happy
def test_small_group(event, small_group):
for adventurer in small_group:
event.invite(adventurer)
for attendee in event.attendees:
attendee.get_ready()
attendee.travel_to(event)
event.start()
@pytest.mark.wip
@pytest.mark.happy
def test_large_group(event, large_group):
for adventurer in small_group:
event.invite(adventurer)
for attendee in event.attendees:
attendee.get_ready()
attendee.travel_to(event)
event.start()
Run the tests (again)
Let’s run our work in progress tests again and see if the new test passes:
pytest -m wip
============================ test session starts =============================
collected 51 items / 49 deselected / 1 skipped / 1 selected
tests/test_earth.py .F [100%]
================================== FAILURES ==================================
______________________________ test_large_group ______________________________
@pytest.mark.wip
@pytest.mark.happy
def test_large_group(event, large_group):
> for adventurer in small_group:
E TypeError: 'function' object is not iterable
tests/test_earth.py:55: TypeError
======== 1 failed, 1 passed, 1 skipped, 49 deselected in 0.12 seconds ========
Wait, what? Our test fails with a TypeError?! 🤔
Specify fixture names
We forgot to rename small_group in the for loop in test_large_group
🤦♂
However because we define a function with name small_group in the module
scope and everything in Python is an object, Python tries to iterate over our
fixture function instead of raising a NameError.
I recommend overwriting the auto-generated fixture name to avoid this problem.
tests/test_earth.py
import pytest
from earth import adventurers, Event, Months
@pytest.fixture(name="event")
def fixture_event():
return Event("PyCon US", "North America", Months.MAY)
@pytest.fixture(name="small_group")
def fixture_small_group():
return [
adventurers.new_frog("Bruno"),
adventurers.new_lion("Michael"),
adventurers.new_koala("Brianna"),
adventurers.new_tiger("Julia"),
]
@pytest.fixture(name="large_group")
def fixture_large_group():
return [
adventurers.new_frog("Bruno"),
adventurers.new_panda("Po"),
adventurers.new_fox("Dave"),
adventurers.new_lion("Michael"),
adventurers.new_koala("Brianna"),
adventurers.new_tiger("Julia"),
adventurers.new_fox("Raphael"),
adventurers.new_fox("Caro"),
adventurers.new_bear("Chris"),
# Bears in warm climates don't hibernate 🐻
adventurers.new_bear("Danny", availability=[*Months]),
adventurers.new_bear("Audrey", availability=[*Months]),
]
@pytest.mark.wip
@pytest.mark.happy
def test_small_group(event, small_group):
for adventurer in small_group:
event.invite(adventurer)
for attendee in event.attendees:
attendee.get_ready()
attendee.travel_to(event)
event.start()
@pytest.mark.wip
@pytest.mark.happy
def test_large_group(event, large_group):
for adventurer in small_group:
event.invite(adventurer)
for attendee in event.attendees:
attendee.get_ready()
attendee.travel_to(event)
event.start()
Now if we run the tests again, we will see a much more helpful error message. I can’t stress enough how important this is. With Python’s duck-typing capabilities, you might not even realize that your tests accidentally pass only because you forgot to add your fixture to your test’s argument list. ⚠️
pytest -m wip
============================ test session starts =============================
collected 51 items / 49 deselected / 1 skipped / 1 selected
tests/test_earth.py .F [100%]
================================== FAILURES ==================================
______________________________ test_large_group ______________________________
@pytest.mark.wip
@pytest.mark.happy
def test_large_group(event, large_group):
> for adventurer in small_group:
E NameError: name 'small_group' is not defined
tests/test_earth.py:55: NameError
======== 1 failed, 1 passed, 1 skipped, 49 deselected in 0.13 seconds ========
Let’s fix that and update the variable name in test_large_group, before we
run the tests again. 💻
Show slow running tests
If you run the tests, you will notice that the suite now takes a long time to complete. pytest comes with a really neat CLI flag to run the tests and show a sorted list of slow tests. Let’s show only the top 2 slowest tests ⏱
pytest -m wip --durations 2
============================ test session starts =============================
collected 51 items / 49 deselected / 1 skipped / 1 selected
tests/test_earth.py .F [100%]
================================== FAILURES ==================================
______________________________ test_large_group ______________________________
@pytest.mark.wip
@pytest.mark.happy
def test_large_group(event, large_group):
for adventurer in large_group:
event.invite(adventurer)
for attendee in event.attendees:
attendee.get_ready()
attendee.travel_to(event)
> event.start()
tests/test_earth.py:62:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
earth/events.py:34: MissingAttendee
--------------------------- Captured stdout call ----------------------------
Bruno accepted our invite! 😃
Po accepted our invite! 😃
Dave accepted our invite! 😃
Michael accepted our invite! 😃
Brianna accepted our invite! 😃
Julia accepted our invite! 😃
Raphael accepted our invite! 😃
Caro accepted our invite! 😃
Chris is not available in May! 😢
Danny accepted our invite! 😃
Audrey accepted our invite! 😃
🐸 Bruno is packing 👜
🐸 Bruno is travelling: South America ✈️ North America
🐼 Po is eating... 🌱
🐼 Po is eating... 🌱
🐼 Po is eating... 🌱
🐼 Po is eating... 🌱
🐼 Po is packing 👜
🐼 Po is travelling: Asia ✈️ North America
🦊 Dave is packing 👜
🦊 Dave's flight was cancelled 😞 Problems at Berlin Airport 🚧
🦁 Michael is packing 👜
🦁 Michael is travelling: Africa ✈️ North America
🐨 Brianna is packing 👜
🐨 Brianna is travelling: Australia ✈️ North America
🐯 Julia is travelling: Asia ✈️ North America
🦊 Raphael is packing 👜
🦊 Raphael's flight was cancelled 😞 Problems at Berlin Airport 🚧
🦊 Caro is packing 👜
🦊 Caro is travelling: Europe ✈️ North America
🐻 Danny is packing 👜
🐻 Audrey is packing 👜
Welcome to PyCon US in North America! 🎉
Let's start with introductions...💬
🐸 Hello, my name is Bruno!
🐼 Hello, my name is Po!
========================== slowest 2 test durations ==========================
20.01s call tests/test_earth.py::test_large_group
(0.00 durations hidden. Use -vv to show these durations.)
======= 1 failed, 1 passed, 1 skipped, 49 deselected in 20.16 seconds ========
The new test eventually failed after taking more than 20 seconds to complete. ⏱
While the --durations CLI flag might not be super helpful with only two
tests and we already knew that the first new test passed in less than one
second in a previous run, it’s super useful for larger test suites.
Add marker documentation
Let’s add a new custom pytest marker slow to test_large_group. It’s
good practice to write documentation for custom markers and provide a short
description for every marker, that explains the characteristics of tests with
that marker. 📝
Add a pytest configuration file with a markers section:
pytest.ini
[pytest]
markers =
slow: tests that take a long time to complete.
Now if you run pytest --markers you will see the information about pytest’s
markers and also our custom one.
Show output
We know which test is slow, but we don’t really know why that is. By default pytest captures any output sent to stdout and stderr. Let’s run the slow test again with capturing disabled, so that we see prints while the test runs.
pytest -s -m slow
============================ test session starts =============================
collected 51 items / 50 deselected / 1 skipped
tests/test_earth.py Bruno accepted our invite! 😃
Po accepted our invite! 😃
Dave accepted our invite! 😃
Michael accepted our invite! 😃
Brianna accepted our invite! 😃
Julia accepted our invite! 😃
Raphael accepted our invite! 😃
Caro accepted our invite! 😃
Chris is not available in May! 😢
Danny accepted our invite! 😃
Audrey accepted our invite! 😃
🐸 Bruno is packing 👜
🐸 Bruno is travelling: South America ✈️ North America
🐼 Po is eating... 🌱
🐼 Po is eating... 🌱
🐼 Po is eating... 🌱
🐼 Po is eating... 🌱
🐼 Po is packing 👜
🐼 Po is travelling: Asia ✈️ North America
🦊 Dave is packing 👜
🦊 Dave's flight was cancelled 😞 Problems at Berlin Airport 🚧
🦁 Michael is packing 👜
🦁 Michael is travelling: Africa ✈️ North America
🐨 Brianna is packing 👜
🐨 Brianna is travelling: Australia ✈️ North America
🐯 Julia is travelling: Asia ✈️ North America
🦊 Raphael is packing 👜
🦊 Raphael's flight was cancelled 😞 Problems at Berlin Airport 🚧
🦊 Caro is packing 👜
🦊 Caro is travelling: Europe ✈️ North America
🐻 Danny is packing 👜
🐻 Audrey is packing 👜
Welcome to PyCon US in North America! 🎉
Let's start with introductions...💬
🐸 Hello, my name is Bruno!
🐼 Hello, my name is Po!
F
================================== FAILURES ==================================
______________________________ test_large_group ______________________________
@pytest.mark.wip
@pytest.mark.slow
@pytest.mark.happy
def test_large_group(event, large_group):
for adventurer in large_group:
event.invite(adventurer)
for attendee in event.attendees:
attendee.get_ready()
attendee.travel_to(event)
> event.start()
tests/test_earth.py:63:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
def start(self):
print(f"Welcome to {self.name} in {self.location}! 🎉")
print(f"Let's start with introductions...💬")
for attendee in self.attendees:
if attendee.location != self.location:
> raise MissingAttendee(f"Oh no! {attendee.name} is not here! 😟")
E earth.events.MissingAttendee: Oh no! Dave is not here! 😟
earth/events.py:34: MissingAttendee
============ 1 failed, 1 skipped, 50 deselected in 20.17 seconds =============
Our slow test failed, but let’s focus on the test duration for now. If you
tried this out for yourself, you could see that 🐼 Po is eating... 🌱 for
several seconds multiple times!
Random fact from the nature documentary Earth: One Amazing Day 😂
Pandas are very fussy when it comes to their diet. Bamboo is almost the only thing they eat. But because bamboo is not very nutritious, pandas need to eat up to 14 hours a day to get enough energy to feed themselves. 🐼
Back to our tests…
Write another test
We have one test that does not use all of the adventurers.py functions, but
is really fast. We have one test that uses all of the adventurers.py
functions, but is rather slow. The ideal test would be one that uses as many
adventurers.py functions as possible and is really fast. 💨
Let’s write a new pytest fixture no_pandas_group and another test named
test_no_pandas_group:
tests/test_earth.py
import pytest
from earth import adventurers, Event, Months
@pytest.fixture(name="event")
def fixture_event():
return Event("PyCon US", "North America", Months.MAY)
@pytest.fixture(name="small_group")
def fixture_small_group():
return [
adventurers.new_frog("Bruno"),
adventurers.new_lion("Michael"),
adventurers.new_koala("Brianna"),
adventurers.new_tiger("Julia"),
]
@pytest.fixture(name="large_group")
def fixture_large_group():
return [
adventurers.new_frog("Bruno"),
adventurers.new_panda("Po"),
adventurers.new_fox("Dave"),
adventurers.new_lion("Michael"),
adventurers.new_koala("Brianna"),
adventurers.new_tiger("Julia"),
adventurers.new_fox("Raphael"),
adventurers.new_fox("Caro"),
adventurers.new_bear("Chris"),
# Bears in warm climates don't hibernate 🐻
adventurers.new_bear("Danny", availability=[*Months]),
adventurers.new_bear("Audrey", availability=[*Months]),
]
@pytest.fixture(name="no_pandas_group")
def fixture_no_pandas_group():
return [
adventurers.new_frog("Bruno"),
adventurers.new_fox("Dave"),
adventurers.new_lion("Michael"),
adventurers.new_koala("Brianna"),
adventurers.new_tiger("Julia"),
adventurers.new_fox("Raphael"),
adventurers.new_fox("Caro"),
adventurers.new_bear("Chris"),
# Bears in warm climates don't hibernate 🐻
adventurers.new_bear("Danny", availability=[*Months]),
adventurers.new_bear("Audrey", availability=[*Months]),
]
@pytest.mark.wip
@pytest.mark.happy
def test_small_group(event, small_group):
for adventurer in small_group:
event.invite(adventurer)
for attendee in event.attendees:
attendee.get_ready()
attendee.travel_to(event)
event.start()
@pytest.mark.wip
@pytest.mark.slow
@pytest.mark.happy
def test_large_group(event, large_group):
for adventurer in large_group:
event.invite(adventurer)
for attendee in event.attendees:
attendee.get_ready()
attendee.travel_to(event)
event.start()
@pytest.mark.wip
@pytest.mark.happy
def test_no_pandas_group(event, no_pandas_group):
for adventurer in no_pandas_group:
event.invite(adventurer)
for attendee in event.attendees:
attendee.get_ready()
attendee.travel_to(event)
event.start()
Test selection via marker expression
We now have three different tests and added custom markers to them. Let’s run all wip tests that are not slow. 💻
pytest -m "wip and not slow"
============================ test session starts =============================
collected 52 items / 50 deselected / 1 skipped / 1 selected
tests/test_earth.py .. [100%]
============= 2 passed, 1 skipped, 50 deselected in 0.19 seconds =============
Hmm… both tests passed. 🤔
Repeating tests
Remember in our previous test run the slow test failed due to an unhandled exception. According to the error our fox adventurer “Dave” didn’t make it to PyCon. 🦊
earth.events.MissingAttendee: Oh no! Dave is not here! 😟
It’s rather suspicious that the test_no_pandas_group passed, when the only
difference to the test_large_group test is that it doesn’t have any pandas.
This might be because tests with foxes are flaky and only fail under certain
conditions.
We can check this by running our tests multiple times with pytest-repeat. Let’s run our fast, wip tests 10 times.
pytest -v -m "wip and not slow" --count 10
============================ test session starts =============================
collecting ... collected 520 items / 500 deselected / 1 skipped / 19 selected
tests/test_earth.py::test_small_group[1/10] PASSED [ 5%]
tests/test_earth.py::test_small_group[2/10] PASSED [ 10%]
tests/test_earth.py::test_small_group[3/10] PASSED [ 15%]
tests/test_earth.py::test_small_group[4/10] PASSED [ 20%]
tests/test_earth.py::test_small_group[5/10] PASSED [ 25%]
tests/test_earth.py::test_small_group[6/10] PASSED [ 30%]
tests/test_earth.py::test_small_group[7/10] PASSED [ 35%]
tests/test_earth.py::test_small_group[8/10] PASSED [ 40%]
tests/test_earth.py::test_small_group[9/10] PASSED [ 45%]
tests/test_earth.py::test_small_group[10/10] PASSED [ 50%]
tests/test_earth.py::test_no_pandas_group[1/10] FAILED [ 55%]
tests/test_earth.py::test_no_pandas_group[2/10] FAILED [ 60%]
tests/test_earth.py::test_no_pandas_group[3/10] FAILED [ 65%]
tests/test_earth.py::test_no_pandas_group[4/10] FAILED [ 70%]
tests/test_earth.py::test_no_pandas_group[5/10] FAILED [ 75%]
tests/test_earth.py::test_no_pandas_group[6/10] PASSED [ 80%]
tests/test_earth.py::test_no_pandas_group[7/10] FAILED [ 85%]
tests/test_earth.py::test_no_pandas_group[8/10] FAILED [ 90%]
tests/test_earth.py::test_no_pandas_group[9/10] FAILED [ 95%]
tests/test_earth.py::test_no_pandas_group[10/10] FAILED [100%]
======= 9 failed, 11 passed, 1 skipped, 500 deselected in 0.30 seconds =======
As we can see from the CLI report, the test_small_group test passed in 10
out of 10 runs, whereas test_no_pandas_group passed only on 1 out of 10
runs. We might not know the exact reason for why that is, but we can tell from
the stdout in the CLI report that there appear to be problems at TXL
airport…sounds familiar? 🤷♀
Marking flaky tests
We can use the built-in xfail marker from pytest for our flaky test: Tests marked with xfail are expected to fail due to the specified reason. If they fail, they will be reported as XFAIL, however if they unexpectedly pass they will be reported as XPASS.
tests/test_earth.py
import pytest
from earth import adventurers, Event, Months
@pytest.fixture(name="event")
def fixture_event():
return Event("PyCon US", "North America", Months.MAY)
@pytest.fixture(name="small_group")
def fixture_small_group():
return [
adventurers.new_frog("Bruno"),
adventurers.new_lion("Michael"),
adventurers.new_koala("Brianna"),
adventurers.new_tiger("Julia"),
]
@pytest.fixture(name="large_group")
def fixture_large_group():
return [
adventurers.new_frog("Bruno"),
adventurers.new_panda("Po"),
adventurers.new_fox("Dave"),
adventurers.new_lion("Michael"),
adventurers.new_koala("Brianna"),
adventurers.new_tiger("Julia"),
adventurers.new_fox("Raphael"),
adventurers.new_fox("Caro"),
adventurers.new_bear("Chris"),
# Bears in warm climates don't hibernate 🐻
adventurers.new_bear("Danny", availability=[*Months]),
adventurers.new_bear("Audrey", availability=[*Months]),
]
@pytest.fixture(name="no_pandas_group")
def fixture_no_pandas_group():
return [
adventurers.new_frog("Bruno"),
adventurers.new_fox("Dave"),
adventurers.new_lion("Michael"),
adventurers.new_koala("Brianna"),
adventurers.new_tiger("Julia"),
adventurers.new_fox("Raphael"),
adventurers.new_fox("Caro"),
adventurers.new_bear("Chris"),
# Bears in warm climates don't hibernate 🐻
adventurers.new_bear("Danny", availability=[*Months]),
adventurers.new_bear("Audrey", availability=[*Months]),
]
@pytest.mark.wip
@pytest.mark.happy
def test_small_group(event, small_group):
for adventurer in small_group:
event.invite(adventurer)
for attendee in event.attendees:
attendee.get_ready()
attendee.travel_to(event)
event.start()
@pytest.mark.wip
@pytest.mark.slow
@pytest.mark.happy
@pytest.mark.xfail(reason="Problems with TXL airport")
def test_large_group(event, large_group):
for adventurer in large_group:
event.invite(adventurer)
for attendee in event.attendees:
attendee.get_ready()
attendee.travel_to(event)
event.start()
@pytest.mark.wip
@pytest.mark.happy
@pytest.mark.xfail(reason="Problems with TXL airport")
def test_no_pandas_group(event, no_pandas_group):
for adventurer in no_pandas_group:
event.invite(adventurer)
for attendee in event.attendees:
attendee.get_ready()
attendee.travel_to(event)
event.start()
Run the tests again:
pytest -v -m "wip and not slow" --count 10
============================ test session starts =============================
collecting ... collected 520 items / 500 deselected / 1 skipped / 19 selected
tests/test_earth.py::test_small_group[1/10] PASSED [ 5%]
tests/test_earth.py::test_small_group[2/10] PASSED [ 10%]
tests/test_earth.py::test_small_group[3/10] PASSED [ 15%]
tests/test_earth.py::test_small_group[4/10] PASSED [ 20%]
tests/test_earth.py::test_small_group[5/10] PASSED [ 25%]
tests/test_earth.py::test_small_group[6/10] PASSED [ 30%]
tests/test_earth.py::test_small_group[7/10] PASSED [ 35%]
tests/test_earth.py::test_small_group[8/10] PASSED [ 40%]
tests/test_earth.py::test_small_group[9/10] PASSED [ 45%]
tests/test_earth.py::test_small_group[10/10] PASSED [ 50%]
tests/test_earth.py::test_no_pandas_group[1/10] XFAIL [ 55%]
tests/test_earth.py::test_no_pandas_group[2/10] XFAIL [ 60%]
tests/test_earth.py::test_no_pandas_group[3/10] XPASS [ 65%]
tests/test_earth.py::test_no_pandas_group[4/10] XFAIL [ 70%]
tests/test_earth.py::test_no_pandas_group[5/10] XFAIL [ 75%]
tests/test_earth.py::test_no_pandas_group[6/10] XPASS [ 80%]
tests/test_earth.py::test_no_pandas_group[7/10] XFAIL [ 85%]
tests/test_earth.py::test_no_pandas_group[8/10] XFAIL [ 90%]
tests/test_earth.py::test_no_pandas_group[9/10] XFAIL [ 95%]
tests/test_earth.py::test_no_pandas_group[10/10] XFAIL [100%]
= 10 passed, 1 skipped, 500 deselected, 8 xfailed, 2 xpassed in 0.24 seconds =
Check code coverage
Now let’s run all of our tests and check the current code coverage. 🌏
pytest --cov earth/
Name Stmts Miss Cover
------------------------------------------
earth/__init__.py 3 0 100%
earth/adventurers.py 71 0 100%
earth/events.py 24 0 100%
earth/travel.py 31 3 90%
earth/year.py 14 0 100%
------------------------------------------
TOTAL 143 3 98%
Yay, we’re at 98% code coverage for the earth project! That’s fantastic! 😁
Next steps
We’ve completed our task of increasing the code coverage through automated tests, but we have some more work to do! 🚧
Running all of the tests including the slow test isn’t great for developer productivity. It makes sense to write a custom pytest plugin that skips tests marked as slow by default and only includes slow tests if we run pytest with a custom CLI option.
Also, did you notice that we have three test cases, but apart from the markers and the different fixtures the test functions itself are identical? That’s a lot of redundancy. 😔
The good news is that pytest comes with a marker which solves this exact problem. Parametrized tests and custom plugins will be our topics in the next part of this tutorial!
I hope you’re enjoying this tutorial as much as I enjoyed giving my talk and writing the first part of this blog post. We’ll do some really cool stuff with pytest in the next part of this tutorial. As usual, I will post a link on my Twitter. Stay tuned! 📝
Update: Customizing your pytest test suite (part 2) is now online! 👨🏻💻