Thinking in edge cases

Tests that catch nothing

Below, grades.py is lesson 5.8's grade_for with lesson 7.3's range check: it raises a ValueError for scores outside 0 to 100. Four copies of it each have one bug planted. The starting tests try one normal score per grade: the happy pathhappy pathThe normal case, where the input is ordinary and everything goes right.. Press Hunt: it runs your tests on the correct code and on each planted bug, and a bug is caught when a test fails on it.

from grades import grade_for


def test_a():
    assert grade_for(90) == "A"


def test_b():
    assert grade_for(75) == "B"


def test_c():
    assert grade_for(60) == "C"


def test_f():
    assert grade_for(30) == "F"

def grade_for(score):
    if score < 0 or score > 100:
        raise ValueError("score must be 0-100")
    if score >= 85:
        return "A"
    elif score >= 70:
        return "B"
    elif score >= 50:
        return "C"
    else:
        return "F"

import pytest
from grades import grade_for


@pytest.mark.parametrize("score, expected", [
    (0, "F"),
    (49, "F"),
    (50, "C"),
    (69, "C"),
    (70, "B"),
    (84, "B"),
    (85, "A"),
    (100, "A"),
])
def test_grades(score, expected):
    assert grade_for(score) == expected


@pytest.mark.parametrize("score", [-1, 101])
def test_out_of_range(score):
    with pytest.raises(ValueError):
        grade_for(score)

Caught 0 of 4. Every test passes on every bug, because no bug changes 90, 75, 60, or 30. Ten more tests of normal scores would catch nothing new either. Add a test at a boundaryboundaryA value where the answer changes, like 84 and 85 for grade_for. Off-by-one bugs live there., grade_for(85) == "A", and hunt again: the first bug is caught. The other three are about scores that should raise an error, and for those you need one more tool.

Testing that an error is raised

For 101, the right answer isn't a letter: grade_for should raise a ValueError. The first test most people write compares with the error:

from grades import grade_for


def test_too_high():
    assert grade_for(101) == ValueError

def grade_for(score):
    if score < 0 or score > 100:
        raise ValueError("score must be 0-100")
    if score >= 85:
        return "A"
    elif score >= 70:
        return "B"
    elif score >= 50:
        return "C"
    else:
        return "F"

It fails with the very error it hoped for: FAILED test_grades.py::test_too_high - ValueError: score must be 0-100. An error isn't a value the function returns. It stops the test on that line, before == runs (lesson 7.3). To expect an error, pytest has a tool, pytest.raises:

import pytest
from grades import grade_for


def test_too_high():
    with pytest.raises(ValueError):
        grade_for(101)

def grade_for(score):
    if score < 0 or score > 100:
        raise ValueError("score must be 0-100")
    if score >= 85:
        return "A"
    elif score >= 70:
        return "B"
    elif score >= 50:
        return "C"
    else:
        return "F"

Read the with line aloud as "the block below must raise a ValueError". Piece by piece:

• import pytest goes at the top. Forget it and the test fails with NameError: name 'pytest' is not defined.
• with ...: works as it does with open (lesson 8.1): a colon, then an indented block.
• pytest.raises(ValueError) names the kind of error, with no brackets after ValueError: you're naming a type, not making an error.
• The indented block holds only the call that should fail.

When the call raises a ValueError, pytest catches it and the test passes. If it raises nothing, the test fails.

Make the change and run it. pytest says E Failed: DID NOT RAISE <class 'ValueError'>. Change it back to 101. Now go back to the hunt and add two tests like this one, for -1 and 101 (with import pytest at the top). All four bugs are caught. A test of 150 would catch "150 gets an A" but not "101 gets an A": 101 is the boundary, the first score that must be refused.

Four kinds of input

The bugs above live where most bugs live, at an edge caseedge caseAn input at the edge of what the code must handle: a boundary, a value it should refuse, or an empty input.. For each function, test four kinds of input:

• Normal: one ordinary value, the happy path.
• Boundaries: both sides of every place the answer changes, like 84 and 85.
• Bad input: values the code should refuse, with pytest.raises.
• Empty input: "", [], {}: nothing at all.

Empty input is the one people forget. Predict:

def average(scores):
    return sum(scores) / len(scores)


print(average([]))

Traceback (most recent call last):
  File "<your code>", line 5, in <module>
    print(average([]))
          ~~~~~~~^^^^
  File "<your code>", line 2, in average
    return sum(scores) / len(scores)
           ~~~~~~~~~~~~^~~~~~~~~~~~~
ZeroDivisionError: division by zero

A failing test doesn't always mean the code is wrong. The test can be wrong too: a test saying grade_for(70) == "C" fails on correct code, because lesson 2.6's table says 70 is a B. When a test fails, check the requirement first, then decide which to fix. The hunt checks this for you: if your tests fail on the correct code, it says so and counts no catches.

Tests first

You can write the tests before the code. That's test-firsttest-firstWriting the tests for a function before writing the function, then writing it until they pass.: write the tests, watch them fail, then write the code until they pass. It makes you decide what the code should do before you write it. Research on whether it makes programs better is mixed, so treat it as a tool, not a rule.

def parse_amount(text):
    pass

import pytest
from money import parse_amount


def test_amount():
    assert parse_amount("ETB 250") == 250


def test_spaces():
    assert parse_amount(" ETB 40 ") == 40


def test_not_etb():
    with pytest.raises(ValueError):
        parse_amount("250")

def parse_amount(text):
    text = text.strip()
    if not text.startswith("ETB "):
        raise ValueError("amount must start with ETB")
    return int(text[4:])

Your turn: hunt phone bugs

phone.py is lesson 6.5's phone pattern. The starting tests try two good numbers and one obvious word. Use 6.5's list of numbers that should fail until every bug is caught.

from phone import is_phone


def test_ethio_telecom():
    assert is_phone("0911234567")


def test_from_abroad():
    assert is_phone("+251712345678")


def test_word():
    assert not is_phone("hello")

import re

PHONE = r"(?:\+251|0)[79]\d{8}"


def is_phone(number):
    return bool(re.fullmatch(PHONE, number))

import pytest
from phone import is_phone


@pytest.mark.parametrize("number", ["0911234567", "0712345678", "+251912345678"])
def test_valid(number):
    assert is_phone(number)


@pytest.mark.parametrize("number", [
    "0811234567",
    "091123456",
    "09112345678",
    "+2510911234567",
    "0911 234 567",
    "",
])
def test_invalid(number):
    assert not is_phone(number)