3 Ways to Test Any Application Effectively with PyTest
For all but the most trivial of computer programs, you need tests to prove that the program does what you expect. There are many different ways to test software, with terminology that can be at times confusing and overlapping. For every testing method, the most important thing to keep in mind is the scope of what you are trying to test: ask yourself whether you should be testing one small section of code in isolation, a single application or service, or the interaction of many services to form an overall solution? Typically you will need to test at multiple scopes at different stages of the development cycle.
Why PyTest?
Python is an excellent language for writing functional tests, regardless of what language your application is written in.
For unit-testing — testing a unit of code such as a package, module or class in isolation — the framework you choose for unit-testing will be strongly influenced (if not dictated) by your choice of language for the code you are testing. Though useful, unit-tests are not enough on their own however. Even if you perfectly understand and test the interfaces of all your software units, complex and potentially unexpected behaviour can still arise when they are connected together — this is known as “emergent phenomena” and arises often in the natural world as well as theoretical processes such as Conway’s Game of Life. Taming this complexity is where functional testing comes in.
Functional testing involves running an application or service, sending in some controlled input and checking the expected output is received within a reasonable period of time. For a backend service, you might send in a network request on the service’s listening address and await an expected response; for a UI application, you might click on an item in the UI and await some specific text to be rendered on the screen. Though functional testing can be performed manually by a human tester, there are a number of advantages to automated tests — once written, scripted tests are cheap to run over and over again in CI to catch regressions or spot features that work most but not quite all of the time.
Python is an excellent language for writing functional tests, regardless of what language your application is written in. Python is designed to be easy to learn, so even if it is not your main language you should not find it a problem to pick up enough Python to read and write tests with it. Since python is a high-level dynamic language, the code may be very concise and readable without clutter from type declarations or memory management. Finally, Python is very well suited for tasks that require a lot of IO interactions and has many great libraries for networking (e.g. requests, scapy) or controlling user input (e.g. pexpect, selenium).
Python comes with a builtin library for unit-testing, but for both python unit-testing and for general functional testing I highly recommend taking a look at PyTest. Unlike many other frameworks which require you to learn a kind of sub-language of method calls, with PyTest you can simply use the builtin assert statement in the way you would expect. PyTest is able to analyse assert statements in your code and gives useful debug information on why asserts failed, instead of simply telling you “assertion failed”. PyTest also offers fixtures as a powerful tool to handle setting up and tearing down the application and service(s) you are testing and to connect testing clients.
When developing a new feature, you typically will want to write, run and re-run your tests often as you find and fix bugs in your application code. Reducing the time and friction of that iteration loop is important to maintain a fast pace and reduce frustration. Of course, build times of your chosen application language are one concern, but how you use your testing framework matters as well. Here I’ll talk briefly about three different approaches.
From the Terminal
Probably the most common way to run PyTest is from a terminal. For most projects, you can simply run pytest from the root of your repo to run all of your tests in a single batch — PyTest will automatically find all modules with names beginning with test and run functions and methods beginning with test . Running all your tests might take a while, so you can specify individual test scripts by file path like pytest path/to/your/test.py. Individual functions or methods on a class may be specified like pytest path/to/your/test.py::test_func or pytest path/to/your/test.py::TestClass::test_method . The -k parameter can also be used to run all tests that match a given expression, so pytest -k abc will run all tests with “abc” anywhere in the name.
While many developers love their terminal, it can have its annoyances in larger projects when you need to type in longer paths and test names and risk making a typo. Additionally, when many tests are run it can be a bit of a pain to scroll back up through many screens of terminal output to find the failure details of a specific test. A graphical solution can help with both of these limitations.
From Your IDE: PyCharm
The second way to run tests is from your IDE. Shown above is PyCharm (the free Community Edition), a popular and full-featured IDE for Python development. After first setting PyTest as your default test runner in the settings (Tools -> Python Integrated Tools -> Testing) you may run tests from a directory or module in the file explorer pane (left) via the context menu (or keyboard shortcut). Alternatively, you can run a specific test function, class or method by opening a test file and clicking the green run icons in the editor pane (right). Either way will run the test(s) selected and present the results back to you in the test runner pane (bottom). The results are arranged into a tree, allowing you to quickly dig down and find the details for why a specific test failed.
Compared to the terminal, PyCharm presents a friendlier interface both for selecting which tests to run and for exploring the results. On the other hand, if you are not primarily a python developer you might not want to install and learn an entirely new IDE just for writing your tests, it would be less of a context-switch to write your tests in the same editor or IDE you use for your application development. Though some people love their IDEs, they can be complex beasts to learn from scratch and often it can be better to use a program that does one thing well rather than one that represents a multi-functional Swiss army knife.
A Standalone UI For Running PyTests
So, what would be ideal is to have a friendly graphical UI for running your tests, that doesn’t require you to install and learn a Python-specific IDE. This is where pytest-commander can help you.
The tool is controlled through a web app running in your web browser. It provides a tree view of your test modules and the test functions, classes and methods they contain. Tests may be run (and re-run) individually or grouped by their class or module, after which you can dig down and explore the results. Getting started is simple with pip, the python package manager:
pip install pytest-commander
pytest_commander path/to/your/testsWhen the server starts, you should automatically see a browser window open with the web app running, ready for testing.
I’ll admit to having a personal interest here — I developed the tool! So if you have any issue or suggestion for improvement, please don’t hesitate to open an issue on github.
In Summary
We’ve seen how Python and PyTest are suitable for functional testing of applications written in any language. Running tests from your terminal is a quick and easy way to get started, but can start to feel clunky for choosing from and analysing the results of a larger number of tests. IDEs such as PyCharm offer an all-inclusive package for writing Python code, running tests and more, but may not appeal to developers coming from other languages with their own favourite editor or IDE already. So the separate UI provided by pytest-commander can offer a more lightweight solution towards effortless testing.
