Unit Testing

You’ve already seen a very basic testing strategy.

You’ve written some tests using that strategy.

These tests were pretty basic, and a bit awkward in places (testing error conditions in particular).

It gets better

Test Frameworks

So far our tests have been limited to code in an if __name__ == "__main__": block.

  • They are run only when the file is executed
  • They are always run when the file is executed
  • You can’t do anything else when the file is executed without running tests.

This is not optimal.

Python provides testing systems to help.

Standard Library: unittest

The original testing system in Python.

import unittest

More or less a port of JUnit from Java

A bit verbose: you have to write classes & methods (And we haven’t covered that yet!)

But here’s a bit of an introduction, as you will see this in others’ code.

And seeing how verbose it can be will help you appreciate other options.

Using unittest

To use unittest, you need to write subclasses of the unittest.TestCase class:

# in test.py
import unittest

class MyTests(unittest.TestCase):

    def test_tautology(self):
        self.assertEqual(1, 1)

Then you run the tests by using the main function from the unittest module:

# in test.py
if __name__ == '__main__':
    unittest.main()

Testing Your Code

This way, you can write your code in one file and test it from another:

in my_mod.py:

def my_func(val1, val2):
    return val1 * val2

in test_my_mod.py:

import unittest
from my_mod import my_func


class MyFuncTestCase(unittest.TestCase):
    def test_my_func(self):
        test_vals = (2, 3)
        expected = test_vals[0] * test_vals[1]
        actual = my_func(*test_vals)
        self.assertEqual(expected, actual)

if __name__ == '__main__':
    unittest.main()

Advantages of unittest

The unittest module is pretty full featured

It comes with the standard Python distribution, no installation required.

It provides a wide variety of assertions for testing all sorts of situations.

It allows for a setup and tear down workflow both before and after all tests and before and after each test.

It’s well known and well understood.

Disadvantages of unittest

It’s Object Oriented, and quite “heavyweight”.

  • modeled after Java’s JUnit and it shows…

It uses the framework design pattern, so knowing how to use the features means learning what to override.

Needing to override means you have to be cautious.

Test discovery is both inflexible and brittle.

And there is no built-in parameterized testing.

Other Options

There are several other options for running tests in Python.

Nose was the most common test runner when I first started learning testing, but it has been in maintenance mode for a while. Even the nose2 site recommends that you consider pytest.

pytest has become the defacto standard test runner for those that want a more “pythonic” test framework.

pytest is very capable and widely used.

For a great description of the strengths of pytest, see:

The Cleaning Hand of Pytest

So we will use pytest for the rest of this class.

Installing pytest

The first step is to install the package:

$ python3 -m pip install pytest

Once this is complete, you should have a pytest command you can run at the command line:

$ pytest

If you have any tests in your repository, that command will find and run them (If you have followed the proper naming conventions).

Do you?

Pre-existing Tests

Let’s take a look at some examples.

Create a directory to try this out, and download:

test_random_unitest.py

In the directory you created for that file, run:

$ pytest

It should find that test file and run it.

You can also run pytest on a particular test file:

$ pytest test_random_unitest.py

The results you should have seen when you ran pytest above come partly from these files.

Take a few minutes to look these files over.

test_random_unitest.py contains the tests for some of the functions in the built in``random`` module. You really don’t need to test Python’s built in modules – they are already tested! This is just to demonstrate the process.

What is Happening Here?

You should have gotten results that look something like this:

$ pytest
============================= test session starts ==============================
platform darwin -- Python 3.7.0, pytest-3.10.1, py-1.5.4, pluggy-0.7.1
rootdir: /Users/Chris/temp/test_temp, inifile:
plugins: cov-2.6.0
collected 3 items

test_random_unitest.py ...                                               [100%]

=========================== 3 passed in 0.06 seconds ===========================

When you run the pytest command, pytest starts in your current working directory and searches the file system for things that might be tests.

It follows some simple rules:

  • Any python file that starts with test_ or _test is imported.
  • Any functions in them that start with test_ are run as tests.
  • Any classes that start with Test are treated similarly, with methods that begin with test_ treated as tests.

( don’t worry about “classes” part just yet ;-) )

  • Any unittest test cases are run.

pytest

This test running framework is simple, flexible and configurable.

Read the documentation for more information:

http://pytest.org/latest/getting-started.html#getstarted

It will run unittest tests for you, so can be used as a test runner.

But in addition to finding and running tests, it makes writing tests simple, and provides a bunch of nifty utilities to support more complex testing.

Now download this file:

test_random_pytest.py

And run pytest again:

$ pytest
============================= test session starts ==============================
platform darwin -- Python 3.7.0, pytest-3.10.1, py-1.5.4, pluggy-0.7.1
rootdir: /Users/Chris/temp/test_temp, inifile:
plugins: cov-2.6.0
collected 8 items

test_random_pytest.py .....                                              [ 62%]
test_random_unitest.py ...                                               [100%]

=========================== 8 passed in 0.07 seconds ===========================

Note that it ran the tests in both the test files.

Take a look at test_random_pytest.py – It is essentially the same tests – but written in native pytest style – simple test functions.

pytest tests

The beauty of pytest is that it takes advantage of Python’s dynamic nature – you don’t need to use any particular structure to write tests.

Any function named appropriately is a test.

If the function doesn’t raise an error or an assertion, the test passes. It’s that simple.

Let’s take a look at test_random_pytest.py to see how this works.

import random
import pytest

The random module is imported becasue that’s what we are testing. pytest only needs to be imported if you are using its utilities – more on this in a moment.

seq = list(range(10))

Here we create a simple little sequence to use for testing. We put it in the global namespace so other functions can access it.

Now the first tests – simply by naming it test_something, pytest will run it as a test:

def test_choice():
    """
    A choice selected should be in the sequence
    """
    element = random.choice(example_seq)
    assert (element in example_seq)

This is pretty straightforward. We make a random choice from the sequence, and then assert that the selected element is, indeed, in the original sequence.

def test_sample():
    """
    All the items in a sample should be in the sequence
    """
    for element in random.sample(example_seq, 5):
        assert element in example_seq

And this is pretty much the same thing, except that it loops to make sure that every item returned by .sample is in the original sequence.

Note that this will result in 5 separate assertions – that is fine, you can have as many assertions as you like in one test function. But the test will fail on the first failed assertion – so you only want to have closely related assertions in each test function.

def test_shuffle():
    """
    Make sure a shuffled sequence does not lose any elements
    """
    seq = list(range(10))
    random.shuffle(seq)
    seq.sort()  # If you comment this out, it will fail, so you can see output
    print("seq:", seq)  # only see output if it fails
    assert seq == list(range(10))

This test is designed to make sure that random.shuffle only re-arranges the items, but doesn’t add or lose any.

In this case, the global example_seq isn’t used, because shuffle() will change the sequence – tests should never rely on or alter global state. So a new sequence is created for the test. This also allows the test to know exactly what the results should be at the end.

Then the “real work” – calling random.shuffle on the sequence – this should re-arrange the elements without adding or losing any.

Calling .sort() again should put the elements back in the order they started

So we can then test that after shuffling and re-sorting, we have the same sequence back:

assert seq == list(range(10))

If that assertion passes, the test will pass.

Testing for Exceptions

One of the things you might want to test about your code is that it raises an exception when it should – and that the exception it raises is the correct one.

In this example, if you try to call random.shuffle with an immutable sequence, such as a tuple, it should raise a TypeError. Since raising an exception will generally stop the code (and cause a test to fail), we can’t use an assertion to test for this.

pytest provides a “context manager”, pytest.raises, that can be used to test for exceptions. The test will pass if and only if the specified Exception is raised by the enclosed code. You use it like so:

def test_shuffle_immutable():
    """
    Trying to shuffle an immutable sequence raises an Exception
    """
    with pytest.raises(TypeError):
        random.shuffle((1, 2, 3))

The with block is how you use a context manager – it will run the code enclosed, and perform various actions at the end of the code, or when an exception is raised. This is the same with as used to open files. In that case, it is used to assure that the file is properly closed when you are done with it. In this case, the pytest.raises context manager captures any exceptions, and raises an AssertionError if no exception is raised, or if the wrong exception is raised.

In this case, the test will only pass if a TypeError is raised by the call to random.shuffle with a tuple as an argument.

The next test:

def test_sample_too_large():
    """
    Trying to sample more than exist should raise an error
    """
    with pytest.raises(ValueError):
        random.sample(example_seq, 20)

is very similar, except that this time, a ValueError has to be raised for the test to pass.

pytest provides a number of other features for fixtures, parameterized tests, test classes, configuration, shared resources, etc. But simple test functions like this will get you very far.

Test Driven Development

Test Driven Development or “TDD”, is a development process where you write tests to assure that your code works, before you write the actual code.

This is a very powerful approach, as it forces you to think carefully about exactly what your code should do before you start to write it. It also means that you know when you code is working, and you can refactor it in the future with assurance that you haven’t broken it.

Give this exercise a try to get the idea:

Introduction To Unit Testing