Quick useful python profiling

Profiling your process

python -m cProfile -o profile_data.pstats my_program.py

Run python like this and it will create a file full of profiling stats.

Viewing the data

SnakeViz rocks. It's a lovely little visualisation tool that makes it very easy to see where your time is going

pip install snakeviz
snakeviz profile_data.pstats

This will launch a web server and your browser to interactively inspect the results. Enjoy!

Testing interface contracts in Python

I've been writing some interface heavy code in Python recently and have developed a pattern that I really like and have not seen before. I think this is original but I doubt it's the first. I'd like to hear about any similar approaches that have been documented.

Interfaces in Python

They aren't used as widely as they could be, owing to the dynamic nature of Python and it's tendency towards duck typing.

Still, there's some very good support for them in community tooling. I make very good use of pylint for detecting interface completion.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# src/interface/knight.py

class Knight(object):
    def say_ni(self):
        raise NotImplementedError

    def set_able(self, is_able):
        raise NotImplementedError

    def drink(self):
        raise NotImplementedError


class NotAbleToDrinkError(Exception):
    pass

pylint will detect that this is an abstract method and if your subclasses fail to implement it then it says so. This is nice but isn't nearly enough to guarantee correct behaviour.

Interface contracts

The contract is our specification for how an interface must behave. In the case of our Knight, the knights must be able to say "ni" and they must drink whenever they are able.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# src/implementation1/test_knight.py
from unittest import TestCase

from implementation_module import Knight


class TestKnightContract(TestCase):
    def test_can_say_ni_without_issues(self):
        knight = Knight()
        knight.say_ni()

    def test_drinks_when_able(self):
        knight = Knight()
        knight.set_able(True)
        knight.drink()

    def test_does_not_drink_when_unable(self):
        knight = Knight()
        knight.set_able(False)
        with self.assertRaises(NotAbleToDrinkError):
            knight.drink()

That's a simple test for a knight's interface contract. All good so far but it falls down when we have multiple implementations of our interface. We end up writing the same contract out again and again.

DRY - one contract better than repeated tests

Normally if someone recommends deduplicating test code "because it's more DRY" I feel compelled to remind them that DRY is about ideas, not about lines of code. In this case we are dealing with the same idea so we should re-use the same code. There is only one contract that all must obey.

My emerging pattern for implementation of this looks a little unusual - it's defining a class inside a method in order to bind the knight_factory into the closure.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

# src/interfaces/knight_contract.py

from unittest import TestCase

from .knight import NotAbleToDrinkError


def contract_test(knight_factory):
    class TestKnightContract(TestCase):
        def test_can_say_ni_without_issues(self):
            knight = knight_factory()
            knight.say_ni()

        def test_drinks_when_able(self):
            knight = knight_factory()
            knight.set_able(True)
            knight.drink()

        def test_does_not_drink_when_unable(self):
            knight = knight_factory()
            knight.set_able(False)
            with self.assertRaises(NotAbleToDrinkError):
                knight.drink()

    return TestKnightContract


# src/implementation1/test_knight.py

from ..interface.knight_contract import contract_test
from .knight import Knight


class TestKnight(contract_test(Knight)):
    pass

This pattern can now be used to implement the same contract in multiple places. When the contract needs clarifying with another test you can be sure that all implementations follow it.

Note that we subclass to create this test instead of simply TestKnight = contract_test(Knight). That simple assignment would run the tests but it would not report which implementation was failing when something broke.

A slight modification for Django

When writing these contracts for interfaces that can be backed by Django models I found a need to use Django's own TestCase implementation in order to have the test database set up correctly.

It's a small tweak:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# src/interface/knight_contract.py

def contract_test(knight_generator, test_class=TestCase):
    class KnightContractTest(test_class):
        # On with the tests.


# django/tests/test_knight.py

class DjangoKnightTest(contract_test(Knight, test_class=django.test.TestCase)):
    pass

Problems?

As I said as the top, I haven't seen this done before. If there's an issue with this approach that I can't see then I'd love to hear about it.

Attrs library for Python

A quick advertisement.

I love Hynek Schlawack's attrs library. It's had a subtle but greatly clarifying influence on my code since I discovered it.

If you're not familiar with that library then I highly recommend reading about it.

It means you can write simple objects like

@attr.s
class Thing(object):
    value = attr.ib()

Which are equivalent to

class Thing(object):
    def __init__(self, value):
        self.value = value

    def __str__(self):
        # ... The usual thing

    def __eq__(self):
        # ... The usual thing

    # ... All the other dunder methods

The docs have much better examples than I could repeat here. You should check it out.

Django ports & adapters series, laying out the problem

Start from the beginning...

This is my introduction piece to this blog series and lays out how I came to be tackling this problem.

Django is a great system for getting started with a web project but it starts to strain as the project grows. Their ethos is to make the simple things easy and the complex things possible. I'm using Django heavily at the moment and I'm feeling some of these strains. This series is about addressing those issues in a maintainable way using the Ports and Adapters architecture.

It's an opinionated framework in the sense that it makes some choices for you. This is a necessary part of making simple things easy. One of the choices that is made for you is to use the Active record pattern.

Escaping Active Record

Active record is familiar to most Django programmers, by experience if not by name. The models.Model base class is Django's implementation of it. Like any software design choice there are strengths and weaknesses. In accordance with Django's ethos, this choice is aimed at making the simple things easy.

Active record is the practise of building your domain model, domain logic and database model into one object. This is a shortcut that can be taken when the two objects overlap. It's really good when the database table layout and the in-memory domain model are the same - of course you'd use the same object there. Conversely, if the two models are very different then you don't want them to be the same object.

Usually a system will start out simple and grow. It will slowly transition from being well suited to ActiveRecord into something that's really not. This has happened to me.

I've got a professional project where there are dozens of models with hundreds of views. Refactoring the database schema has become hard because there are too many pieces of code dependent on it. Too much complexity is brought in by depending on a detail instead of an abstraction. The views and controllers all depend on the domain model and reference its API. This is good and correct. The problem we have with active record is that the views also depend on the persistence model.

This is tight coupling from one IO mechanism to another and our domain logic is caught in the middle. This puts us in the position of needing to trade off between performance, maintainability and domain model complexity. The performance is necessary and the domain model is out of our control so maintainability suffers if the system continues being developed without addressing this issue.

Have you ever heard a colleague calling user requirements "stupid"? Then some part of the business insists that it's built anyway and everyone gets more tense? This is my litmus test. A code smell without code. The forced complexity from the domain model has nowhere to go when the domain logic depends on other complex details like persistence.

Ports and adapters

Ports and adapters is an architecture pattern very well suited to this situation. It puts domain models and business logic at the core of your program and makes everything else depend on them. These are the details of the real world that don't change nearly as often as database technologies, messaging frameworks, the number of customers your product needs to support or usage patterns of users.

The pattern says that everything outside of the domain model should depend on the domain model. This allows us to contain complexity in the domain model where it is abundant and keep all other concerns separate. In Django's case that starts with the database but will be extended to signals, background task queues and anything else that occurs along the journey.

The dependency of our domain logic on the database has to be inverted so that we can freely change either without worrying about breaking the other. The active record stuff is very useful but has to be abstracted.

It's neatly explained on Ward Cunningham's wiki and informs Bob Martin's "clean architecture".

So, where to?

The next post will show how I started with a domain model built outside Django, built persistence services around it, then brought it in to a Django project whilst retaining the useful properties of the architecture, adding Django models as 'just' another option for persistence.

I'll go on to explore inserting this architecture into an existing system and refactoring the interfaces involved.

Welcome

So ... I'm starting a blog. I made a reckless promise to @hjwp that I'd write up a how-to or two.

I'll be writing about Python, Django, architecture, patterns, development teams and being understood.

It'll be interesting to see if that's still true in a few months