Improve your code quality with Python 3 type hints

Confession #1:

I like dill pickle sandwiches. Yes. Dill pickles. On rye bread. My daughter enjoys them but my son-in-law? He’s not tickled by pickles. “That’s so wrong. A pickle is a side dish. Or a garnish. Something you put on hamburgers. But not a sandwich filling.”

Confession #2:

I wrote Functional Python Programming — Second Edition, available from Packt Publishing. I wrote most of the code examples to include type hints, so I have an idea or two about the appropriate use of hints, and how to use the mypy tool.

These confessions are related — In Python 3, type hints are kind of like dill pickles. Are they a garnish? Are they the sandwich filling? Is including them in legacy code like eating some nasty stale legacy sandwich you made a week ago?(Let’s not slice too deeply into this, okay?)

Type Hinting and Type Checks

Before exploring our pickle metaphor problem and looking at a concrete example, let’s talk about the buzz around type hinting. With type hint, there’s a change in viewpoint that can lead to discomfort and the kind of broad, vague statements that reveal people’s discomfort.

If you follow a variety of Pythonistas on Twitter, you can see some of this in the debates on the merits of type hinting. Some of the key points seem to include:

• It’s hard.
• It’s so hard, only do it if you absolutely need it.
• It’s hard, but it can help.
• It’s too verbose.
• It’s really helpful, and we need to use it more widely.
• It represents a “gap” in the language; the lack of type checking means Python can’t be used for “real” computing.

Let’s look more seriously at a few of these complaints.

Python Lacks Type Checking

This point reflects a weird view in my opinion. Python does have very, very tight run-time type checking. Lots of bad code raises run-time type errors. Python emphasizes a wonderful orthogonality between types and operators, allowing duck typing to work in many cases. I can’t emphasize this enough, Python has very strict type checks.

Python doesn’t attach type information to identifiers: variables don’t have a statically declared type. The objects referenced by variables, however, do have types, and attempts to abuse the objects will lead to run-time problems. I can’t emphasize this enough — no really, I can’t — Python has very strict type checks.

For some, this point isn’t about the lack of type checking in general, it’s more about the lack of “compile-time” checking. Once people learn about the internal type checking, they like to shift the argument forward, “A real language would have prevented that with type checking.” I generally respond with a rhetorical question: “Then why do you unit test?” Python has the same workflow as statically type checked languages, so the error “prevention” idea doesn’t seem to be wonderfully helpful.

In a few cases, a discussion on Python and type hints digresses into the ways some IDE’s can provide syntax suggestions based on declared types of variables. I’m sure this might be helpful, but I have reservations. Mostly it leaves me wondering how reliable code is when the author depended on the IDE to suggest things.

To continue the pickle metaphor problem, when my daughter was little, we played in the kitchen and made intentionally odd sandwiches. I remember potato chips between crackers was something that got us both in trouble with the other adult in the family. But we never used the cat’s food in our sandwiches no matter how many times my daughter asked.

Verbosity

Verbosity in type hints can become a problem. When creating complex objects from built-in types, we often forget to give names to the intermediate object classes. Let’s look at some concrete examples of complex type objects created entirely from the built-in Python collection types.

Let’s say we’re doing some geolocation processing. We’ve got data that looks like the following:

{((12, 13), (14, 15)): 2.8284271247461903,

 ((14, 15), (2, 3)): 16.97056274847714,

 ((2, 3), (5, 7)): 5.0}

This can be described by the following type

Dict[Tuple[Tuple[int, int], Tuple[int, int]], float]

Let’s pretend this structure was created by the following d_map() function. Here’s the version without hints. Adding hints seems hard.

def d_map(points):
    return {
        (p1, p2): hypot(p1[0]-p2[0], p1[1]-p2[1])
        for p1, p2 in points 
    }

The declaration became L. O… N… G… that is, if we make a common mistake.

 def d_map(points: List[Tuple[Tuple[int, int], Tuple[int, int]]]) -> Dict[Tuple[Tuple[int, int], Tuple[int, int]], float]:
       return {
           (p1, p2): hypot(p1[0]-p2[0], p1[1]-p2[1])
           for p1, p2 in points
       }

These hints don’t fully describe what’s happening. The hints we wrote elided important details and didn’t reflect the underlying semantics of the data structure.

One of Python’s strengths is the rich collection of first-class data structures with built-in syntax. We can abbreviate some complex concepts into succinct, expressive code. However, we shouldn’t lose sight of what the succinct code represents. And in this case, it represents some rather complex concepts.

<rant>

Let me play the old programmer card. Wait a second while I set up my lawn chair so I can shake my fist at kids these days. When I was your age we walked to school. It was uphill. Both ways. And, we spent forever trying to get linked lists and simple hash maps to work. I remember struggling with a simple binary tree to summarize a large file into counts grouped by code values. Nowadays, you just slap a collections.Counter into your code like it’s a nothing. It’s not a nothing. It’s serious, sophisticated software engineering. It’s more than Dict[Any, int].

In the long run, I think it’s the simple summary that makes it easy for us to leverage a sophisticated structure without having to write down all of the details.

</rant>

When confronted with a super-long type hint, what can we do? When in doubt, Expose the Intermediate Types. Here’s my rewrite of the above example:

Point = Tuple[int, int]
Leg = Tuple[Point, Point]
Distances = Dict[Leg, float]
def d_map(points: Iterable[Leg]) -> Distances:
    return {
        (p1, p2): hypot(p1[0]-p2[0], p1[1]-p2[1])
        for p1, p2 in points
    }

This provides details in a useful form whereas the original definition avoided describing the input data structure in any detail. Looking at the code, we could deduce two-tuples of numbers were probably the input, but there wasn’t much detail to go on. This revised version gives us reusable type names that can be part of other functions.

I want to call out a kind of quality assurance check for type hints. Long type hints often involve redundancies. I think redundancies are errors: they’re intermediate data structures which should be given proper names.

When we look at this some more, we may rethink using a simple two-tuple to represent a point. The p1[0] syntax is that cheese from the back of the fridge that may have stuff growing on it. We don’t want that on our sandwich. Perhaps this should have been.

class Point(NamedTuple):
    x: int
    y: int

That leads to tiny (almost-but-not-quite trivial) simplifications. Instead of building simple tuples for each point, we can now build named Point tuples and use p1.x and p1.y to make the code much yummier. This change is like switching from bland white bread to proper, crunchy rye bread for your Reuben sandwich. Rye bread has more structure for a hearty sandwich loaded with sauerkraut.

When we look closely at the complaints outlined above, the “it’s hard” complaint feels like “I don’t like sauerkraut on my Reuben sandwich.” Without the sauerkraut (or coleslaw), it’s not a proper Reuben. Analogously, Python code without type hints seems to be missing an essential feature.

No One Wins in Code Golf

One minor consequence of this approach is a tendency to avoid using (), [], and {} to build tuples, lists, mappings, or sets. Yes. This is heresy. I find using tuple(), list(), dict(), and set() is slightly easier because I can replace the type name with an equivalent name as the design matures.

“But,” you may object, “the code is objectively LONGER! You didn’t save me anything! You’re a fraud!”

My first response is, “Correct.” It is objectively longer. My second response is, “Code Golf isn’t a game I want to win at.”

While an obfuscated Python code contest can be fun, I’d rather not muddy working code with intentional obscurity. The Zen of Python advises us “Explicit is better than implicit.” Code that depends on the following behavior is wrong in the larger sense of being intentionally and wickedly obscure.

>>> {1: ‘one’, True: ‘true’}
{1: ‘true’}

I like exposing details within a more complex data structure. The type definitions are potentially reusable, and I think this can add clarity throughout a large application. It can also serve as a seed for writing complete docstrings describing the behavior.

When this kind of declaration is part of a reusable module, the goodness spreads like smiles and hugs throughout the application. Before long, other functions have been tweaked and everyone is sending each other little teddy-bear hug gifts with rainbow cupcakes. (Just please don’t exchange mylar balloons. They’re evil.)

Generic Functions

One of the more difficult problems with providing type hints is detailing generic behaviors. Let’s look at a function to take a sequence of individual points and create pairs of points — legs — of a trip.

def pair_iter_1(points):
    return zip(points[:-1], points[1:])

This requires two (potentially larger) in-memory data structures. My preference is the following.

def pair_iter_2(points):
    point_iter = iter(points)
    p1 = next(point_iter)
    for p2 in point_iter:
        yield p1, p2
        p1 = p2

Both of these functions, pair_iter_1() and pair_iter_2(), rely on Python being very generic with respect to data types. These function work with a wide variety of collections and the underlying types can be almost anything. It can be challenging to find the appropriate abstractions. Here’s an approach that — while technically allowable — is misleading.

def pair_iter(points: Iterable[Any]) -> Iterator[Tuple[Any, Any]]: …

What’s wrong with this? The Any type doesn’t make a very strong claim about the relationship between the outputs and the inputs. The point of type hinting is to make as strong a claim as necessary. Now, we can go too far, and write this:

def pair_iter(points: Iterable[float]) -> Iterator[Tuple[float, float]]: …

This suffers from a needless constraint requiring numbers in an otherwise perfectly generic algorithm. Using type variables allows us to provide a strong-enough constraint. This clarifies the nature of the transformation: the source data is untouched by the restructuring that’s being performed.

_T = TypeVar(“_T”)
def pair_iter(points: Iterable[_T]) -> Iterator[Tuple[_T , _T]]:
    point_iter = iter(points)
    p1 = next(point_iter)
    for p2 in point_iter:
        yield p1, p2
        p1 = p2

The type variable, _T, is bound to the type of the parameter iterable as well as the resulting iterator. This states the intent of this function to provide a resulting structure built around the source objects. It clarifies the absence of any transformation of the underlying objects.

TL;DR

When your type hints seem ungainly and large, consider exposing the intermediate types. It generally helps to break down a big structural type hint into the constituent pieces. I like to think of it this way:

If you had to create a class definition for EVERY variation on list, dict, set, and tuple, what would your new class be named?

This rule helps clarify the roles of type hints. In effect, we’re creating the names for Python data structures, while avoiding the overhead of a full class definition. If you had to describe the underlying meaning of a class — separate from its implementation details — what name would you give it? Picking names is central to creating meaning in software. After addressing dozens and dozens of functional programming examples, the type naming problem appeared frequently. Giving types sensible names made it much easier to get type hints correct. It also taught me picking names is one of the two hardest problems in computing. (The other hardest problem? Cache invalidation and off-by-one errors.)

A complex type definition is like sauerkraut, gari, or kimchee: a pickled concoction that’s more than a garnish or decoration but not a complete meal. Think of it in the same way that yeast-risen bread and fire-roasted meat aren’t a Reuben by themselves. A proper sandwich involves a variety of ingredients, none of which is an afterthought.

Third Confession:

I missed lunch writing this.