Putting More Buzz in Your Python Fizz

Type hint analysis, like unit tests, and static code analysis, all serve to give people an appropriate level of confidence that the code works as expected. They can be a helpful part of a Python program and are one of many tools we use to establish the overall quality of our code.

In this post I want to explore different ways type annotations can help write better software. In order to do this, I’m going to need a problem to solve. You can probably guess from the title what I’m going to tackle: the fizz-buzz problem. Several times. Four times to be precise. With different approaches and therefore distinct kinds of type hints.

In case you haven’t heard of the fizz-buzz problem, it’s based on a party game. People sit in a circle. They count off numbers, starting from “One.” Then the next person says, “Two.” So far, so good, right? Now comes the first exception to the rule. No one says a multiple of three, they say “Fizz” instead. So the next person says “Fizz,” and the person after them says “Four.” Now the next exception: no one says a multiple of five. Instead, they say “Buzz.” So the next person says “Buzz” instead of “Five.” Then “Fizz.” Then hilarity ensues because no one can remember which number comes after a Buzz and a Fizz. Eventually someone figures out it’s “Seven.” Then “Eight,” then “Fizz,” and “Buzz.”

Some backstory on why we’re going to over-engineer this problem. For a few years, I lived on a sailboat. There’s a lot that can go wrong at sea, and the consequences of a failure can be dire. Therefore, many sailors will agree that anything worth engineering is worth over-engineering.

So, in this article I want to over-engineer the fizz-buzz problem four times over. It’s relatively simple, and therefore, we can look at it from a number of perspectives. But before we launch into over-engineering this problem, let’s talk about tools.

On the boat, we use winch handles to give us leverage over lines with heavy sails or chains with heavy anchors. An anchor might weigh 25 kg, and the chain 2.3 kg per meter. Anchoring in 10m of depth nearly doubles what has to be lifted; which is not safe to try by hand. In this situation, winches are essential, and long winch handles are very important.

Now back to coding. By design, Python type hints have little influence on the run-time behavior of our code. They’re mostly used by tools like mypy. The mypy tool does static analysis of the code and the annotations to confirm the code fits the hints.

If you haven’t got it already, you’ll want to add it.

I’m also going to suggest an arrangement for Python project files and folders. Not everyone loves this arrangement, but I think it works out well for most projects.

I’m going to focus on using mypy to check the source, so I’m going to quietly ignore the tests. I’ll leave those as an exercise for the reader.

After installing mypy and setting up the two folders, here’s a first round implementation of fizzbuzz.py

Yes, that script feels like cheating. Instead of stepping through the game algorithm, it has a hard-wired result of nine rounds of play. It does sort of work. It’s a huge pain to test because there’s no function or class a test case can import and exercise. No automated test means it may not be working.

When this is set up, we should be able to enter mypy src to check the type annotations on this little file. There are no explicit type annotations. The one line of code matches the definition of the built-in print() function. So, this line of code looks good to mypy. 

This is a good time to write your own solution to fizz-buzz. Call it fizzbuzz1.py to distinguish it from my not very good initial example.

In order to have something testable, it helps to break the fizz-buzz problem into functions. Here’s a decomposition that seems to solve the problem. Spoiler alert: it has bugs.

This seems to work for numbers from 1 to 10. In spite of the logic problem, let’s add type hints. If you haven’t seen them before, they look like this:

There are two changes to the code to annotate the expected types:

• After the parameter, n, there’s a : int annotation; the hint is n should be an instance of the int type.
• After the function parameter list, there’s a -> str annotation; the hint is the return value from this function should be an instance of the str type.

While some aspects of type hinting can be more complex, this is the essential model for the ways annotations are used. Provide a hint for each parameter to a function or method, and the result of each function or method. This seems to cover a multitude of cases elegantly.

As noted above, the algorithm has a bug. And the type hints also have a problem.

Let’s tackle the hints first. We can run mypy on the src directory and see the following:

Some folks spotted this conflict right away. The type annotation said the fizz_buzz() function returned a str. But. One of the return statements returned an integer value.

This leads to an interesting dilemma. We can ask “which one is right?” When the code and the annotations conflict, we have two paths forward, depending on our intent as designers of this software:

1. The annotation is right: fix the code to match the annotation.
2. The code is right: fix the annotation to match the code.

This dilemma happens a lot. A real lot. More-or-less constantly, in my experience.

Let’s look at option 1 - The annotation is right: fix the code to match the annotation.
The annotation was right all along, but the code didn’t quite implement it correctly. Here’s what the code should be:

We’ve fixed the return statements to create strings, consistent with the annotation. I’m partial to this path but it’s not always the right thing to do.

Let’s look at option 2 - The code is right: fix the annotation to match the code.
The annotation didn’t properly reflect the code. Here’s what the annotation should be:

This introduces a new type constructor, the Union. This builds a composite type where the objects can be either a string or an integer. This describes the results of this function’s implementation.

Python relies on Duck Typing:  ”If it looks like a duck, swims like a duck and quacks like a duck then it probably is a duck.” This means that most Python code is generic with respect to type, and many functions can be described as working with unions of a large number of types.

As a practical matter, our application code tends to be biased toward one or a few types. To clarify our intent, we often want to narrow the domain of possible types, and focus on one that really matters. To an extent, we use type annotations to intentionally set aside the way Python is capable of handling a large number of types so we can focus on just a few types that are relevant to our application.

Sailors make a lot of these nuanced distinctions when we lay hands on the various lines around the boat. We distinguish between sheets, halyards, reefing lines, dock lines, and ground tackle. Yes, they’re all more-or-less cordage of various sizes and colors. In the case of ground tackle, the anchor line may also have a patina of mud. Each line has a specific and often fixed application. When turning the boat, for example, we may need to ease the sheets. Easing the halyards while turning will create havoc. And easing the docklines is pointless.

In the same way, we often want to carefully distinguish the data types permitted by a function in our applications. For this reason, I suggest avoiding the complexities of Union definitions and fixing the function to work with a narrower definition, def fizz_buzz(n: int) -> str:.

This doesn’t uncover the algorithmic problem. It can’t, really, as we’ve failed to account for the number 15. It’s both “Fizz” and “Buzz.” I think this distinction between bugs that can only be found with unit tests, and potential bugs found by mypy, is essential. To solve this problem we need a number of tools, including mypy.

Let’s do some over-engineering, shall we?

Instead of simply printing the number, Fizz, or Buzz, we want to accumulate a mapping between the number and a set of strings. We’re looking to create something like the following:

Ideally, we’ll also fix the bug in the algorithm bug and have 15: {"Fizz", "Buzz"} in the result.

This requires some additional type constructors. The typing module includes List, Set, and Dict definitions we can use. Overall this is a dictionary that maps integers to sets.

We can start with Dict[int, Set] to describe this. Pragmatically, it’s really Set[str], since the set will only contain strings (or be empty.). This leads us to a function signature that looks like this:

We can then use this function to build the mapping, Dict[int, Set[str]].

Before sailing on, feel free to heave to and write your own.

Here’s my solution:

Each function is consistent. They all accept an integer parameter and create a result that is a proper Set[str].  The final mapping uses a dictionary comprehension to create a mapping from the integer to the results of the fizz_buzz(n) function.

When we run mypy on this file, we’ll find that mypy has a question about the fb_map assignment. While we -- as authors of code -- are pretty sure the mapping can be described as Dict[int, Set[str]], mypy is not delighted with leaping to this conclusion.

We need another bit of type annotation machinery.

We’ve put a : Dict[int, Set[str]] into the assignment statement between variable and =.

This clarifies the intent of the dictionary comprehension. It gives mypy enough leverage to make a conclusion on whether or not all the functions are consistent.

The fb_map: Dict[int, Set[str]] assignment statement is kind of complex with a type annotation buried in an already complex statement. Can we simplify this?

Spoiler alert: Yes.

One thing we can do is use type construction to break the complex definition down into simpler parts.

This shows how we can build a new type annotation and give it a name, FBMap. This name lets us simplify the assignment statement, by using only the FBMap type name rather than the long type expression.

While this is simpler, there’s some repetition here that’s undesirable. We repeat Set[str] a lot of times. Do we need to?

Spoiler alert: No.

Consider this decomposition of the mapping type.

We’ve assigned the complex Set[str] to a single name, FzBzState. This isn’t a huge simplification in this case. But, Python lets us build very complex structures that we might want to simplify. Think of a list of tuples of strings and tuples of integers, or something equally bewildering. Because Python permits a lot of complexity, it can help to decompose these complex, bewildering things into some separate definitions.

This leads to a further rethinking. Rather than provide all the code, I’ll summarize:

The intent here is to make sure it’s clear that all of the functions create a FzBzState object, and the final mapping object will include an integer and a FzBzState object. We can see that -- for this specific implementation -- FzBzState is a Set[str]. Having this consistent name makes it possible to consider changing the underlying type, to improve performance or provide a more expressive definition of the objects.

The idea of breaking complex types down into simpler types is very appealing. When we’re facing a complex problem in boat maintenance, it helps to decompose the complex problem into simpler problems we can solve in isolation.

For example, here’s a picture of a too complex bit of plumbing. It’s not clear, but five separate hoses converge through a complex nest of fittings. This needs to be simplified, because if there’s a failure somewhere, this could be very difficult to deal with.

Let’s really over-engineer the fizz-buzz problem by introducing class definitions. And not just any old class definitions. Let’s introduce mutually interdependent class definitions.

We’ll break the fizziness or buzziness of a collection of numbers into two parts:

• A class to define the properties of a given number.
• A collection of those individual number property definitions.

Spoiler Alert: There are problems in the following code.

The FBStatus class definition describes a single number and starts like this:

If you haven’t worked much with type annotations, the __init__() method must return None. We provide the overall map as part of each individual number’s status. I don’t have a brilliant reason why this relationship is essential, but this circularity is a common pattern in complex data structures where navigation of the graph can work “up” as well as “down” the structure.

More code is required to properly load the values into the self.fb set. We’ll come back to this later, once we get the essential class definitions squared away.

The FBMap class definition describes a collection of numbers and looks like this:

The initialization of the mapping creates a dictionary to map integers to FBStatus instances.

This has a nuanced problem. And a few other less subtle problems.

• Inside a method’s body
  we can refer to any object that will be part of the local or global namespace. Method body evaluation happens after all the functions and classes have been defined. This means Python function and class definitions can, generally, be in any order. We often chose an order to help explain the code.
• Outside a method’s body (i.e., in the definition)
  In the definition line we can only have references to names previously defined in the module. This constrains the order for definitions so that a function or method definition can only refer to previously defined classes or functions.

Mypy, however, gives us a way to break this definition order rule. We can provide a string instead of a class name. Mypy will resolve the strings, and this will let us include forward references. Here’s a small change that lets us define FBStatus first with a forward reference to FBMap.

The change is minor. We replaced FBMap with the string "FBMap". This isn’t all, though. Once we have this solved we can move on to the two other problems here.

First Problem - self.parent

The first problem in the __init__() method is that self.parent really needs to be a weakref. That’s outside the scope of the type hint topic, but it’s helpful to use weakref.ref(parent).

Second Problem - setting the self.fb set elements

The second problem in the __init__() method is we never set the value of self.fb to anything useful. We want to create a set of fizz or buzz properties.

Let’s solve both problems and finish the initialization of self.fb:

This shows how we’d like to build the self.fb set as a union of several possible values. We can add “Fizz” to the set, or add “Buzz” to the set, or both, or leave it empty.

Third Not-Really-a-Problem - display the state

While our class is pretty simple, it’s common to have properties or methods to expose the current state of an object. Let’s add one more feature: a property to extract a useful summary from each instance of this class. Here’s the full definition:

This property will give mypy fits. Why? We have a conflict:

• The fizz_buzz property definition claims self.fb is Set[str].
• The __init__() method claims self.fb is Set[Any].

As we noted above, we’ve surfaced a conflict between the code and the hints. Often the code is wrong, but sometimes the hints are wrong. In this case, a little extra annotation will save the day.

One final look at the class definition for FBStatus.

This definition of the FBStatus class provides an important clue to mypy: the set will only contain strings. This additional definition resolves the conflict mypy saw between how self.fb was created initially, and how it was used in the fizz_buzz property.

The “FBMap” string as a forward reference annotation to the FBMap type reminds me of the way “messenger lines” are used in boats. Threading a new halyard inside a 50-foot mast is tricky. There are other lines and electrical wires inside the mast. It’s a long aluminum tube, so we can’t see what we’re doing. However, sailors have a solution for this. We start by tying a light piece of line to the end of the old halyard. When we pull the halyard down, the light piece of line follows it around the various blocks through the dark recesses of the mast. We leave this messenger in place to mark the path. When it’s time to replace the halyard with a new, less chafed line, we bend the new halyard to the messenger, and use it to pull the heavy halyard through the dark recesses of the mast.

Using a string for a type hint is like the messenger line. The real type will be defined eventually. For now, there’s a lightweight placeholder.

Sometimes, the errors from mypy can be confusing. For me, the most common cause for confusion is the mypy errors conflict with one of my closely-held assumptions about the code I’m working on. I thought I knew what I meant. Why can’t mypy see it, too?

The primary tool for clarifying how wrong assumptions can be is the reveal_type() “function”. This has the syntax of a function, but it isn’t a real function. It’s used by mypy to display details.

We might use it like this:

I’ve included a reveal_type(self.fb) in this example to show how it would look. This is something you do before running mypy. It has to be removed, because you can’t even run unit tests with this in place. It can help us see what mypy is seeing about our code.

It’s like spotting a buoy on a tricky river entrance. GPS is fun, and it’s nice to think you know where you are. Nothing beats seeing a big old green can buoy floating in the water, more-or-less where you hoped it would be. A physical landmark provides a lot of confidence that we’re sailing the safe, deep water.

I want to look at another possible over-engineered solution to the fizz-buzz problem. We’ll start by using typing.NamedTuple instances to track fizziness and buzziness of a number. These can have type hints built-in and can be very useful for creating applications with reasonably complete annotations.

First, the new typing.NamedTuple which is much cooler than the older collections.namedtuple. Here’s an example:

This is a Python 3-tuple, with attribute names, n, fizz, and buzz, for the items in the tuple and -- bonus! -- type annotations for each of the items. This is a little nicer than an unnamed tuple of Tuple[int, Optional[str], Optional[str]] because each item has a proper attribute name. The big bonus is providing hints so mypy can examine the code carefully.

The Optional[str] is a handy way to describe a union of two types. This is equivalent to Union[str, None], and reflects a very common Python programming practice. This articulates the way None is commonly used as a placeholder when there’s no useful value.

We can use code like this fb6 = FB(6, "Fizz", None)  to define the FizzBuzz properties of a given number. We can use fb6[0] or fb6.n to get the value, fb6[1] or fb6.fizz to get the fizziness of the number. I’m a fan of named attributes instead of positional attributes.

How do we create these objects? We’ll need a factory of some kind. Here’s a suitable function:

This will build FB tuples with a number and the appropriate properties. I like these kinds of solutions where we can use functions applied to immutable objects.

This isn’t completely compatible with previous definitions, however. It doesn’t use a Set[str] anywhere. We can add that as a property:

I’ve added an as_set property that will transform the fizz and buzz values into a set of strings. Note that the values of self.fizz and self.buzz are Optional[str]. The union of str and None means they may have a None value. We don’t really want to see a None object in the result set, so we explicitly remove it with set subtraction.

This causes problems with mypy, however. The contents of the set appear to be Set[Optional[str]], which doesn’t match the return type of Set[str]. The problem here is that mypy can’t figure out our algorithm for removing None objects. A person can be convinced that there will be no None objects in the resulting set, but mypy isn’t as clever.

There are some places, where mypy can (and does) detect the shift from Optional[str] to str. These places almost always involve an explicit if statement that’s easy to detect and reason about.

Lacking an obvious if statement, we’re forced to label the result with a type that we’re sure our algorithm produces. For this, we’ll need the typing.cast() function:

The use of cast(Set[str], …) tells mypy that the expression really does remove None values, building a Set[str] from what appeared to be Set[Optional[str]].

The cast() function is essentially an annotation with no run-time consequence. It can help to clarify expressions that aren’t perfectly obvious to mypy.

For me, named tuples are like using color-coded sheets to control the sails. Rather than ask a guest to cast off the starboard staysail sheet, it’s a lot easier to ask them to pick up the thinner, green ropey thing. The fatter, green ropey thing is the starboard yankee sheet, and isn’t being used right now. And the two red ropey things are port side sheets, we’ll be using those after we tack.

The complexity of my boat means we’re not really sailing unless we have six different sheets all piled up in the cockpit. As with the attributes of a named tuple, it’s important to have clear names for each one.

The final example of over-engineering is creating instances of dataclasses. There’s a lot of flexibility here, and I’ll stick to one example that’s not too complicated.

This FB class lets us create objects using code like [FB(i) for i in range(15)]. We’ve set this up to show some of the many kinds of initialization alternatives available to dataclasses:

• The default case is shown by the self.n attribute. The core __init__() processing is built for us; this will set the value of the self.n attribute when we use FB(i).
• A common alternative case is to provide a default value. The self.fizz_buzz attribute is not part of the __init__() parameters because we used an assignment statement to provide a default value. The field(init=False) is a very special kind of default, leaving this attribute unset. A kind of no-default default.
• We’ve provided a __post_init__() method to set the value of the self.fizz_buzz attribute after __init__() has set the self.n attribute value.

The __post_init__() method serves to provide a tidy encapsulation of the Fizz Buzz rules inside the class definition. The dataclass lets us provide type hints for the attributes of the class. And it lets us initialize those attributes a variety of ways.

The objects created by the FB class are mutable, unlike the version created with typing.NamedTuple, so a change to the n attribute can lead to an invalid object. Something pathological like:

Is morally corrupt, but valid Python. If we want to be able to set the n attribute, and have the value of the fizz_buzz attribute change also, we will need to create settable properties. I’ll leave this as an exercise for the reader.

The flexible initialization alternatives make dataclasses very handy for creating stateful objects. We can provide type hints to reflect the domains of the attribute values. For me, the flexibility of data classes is like having two masts and five sails on a boat: no matter what conditions, there’s a combination of sails that will provide a safe, controlled ride. It takes some work to get the sails configured, but the number of choices available means the results will often be delightful.

Put more buzz in your fizz with type hints.

There are a lot of ways software development can go wrong. We can misunderstand the users and their problem. We can misunderstand the data or the appropriate algorithm. We may have code that doesn’t quite match our intent, or perhaps, our intent is a little vague.

I’m a fan of thinking through the type hints and using this to inform the code and the unit tests. I think it’s helpful to clearly articulate what the structures need to be before trying to write the code that builds or consumes those data structures.

I’ve had some conversation with people who wonder if type hints are effectively redundant; since unit tests confirm the software works, type hints aren’t telling us anything new. I reject the idea of redundancy as a problem.

I submit that type hints do tell us something new. Python’s duck typing flexibility means a number of mistakes can pass a suite of unit tests that fail to test all the obscure edge cases. An expression like a+b works for float, integer, string, list, tuple, and even bytes. While it’s common to test the expected types, we rarely test the unexpected types. The static analysis of mypy can help narrow the domain of types under consideration and provide assurance the tests cover all possible cases.

I can’t say anything bad about having multiple tools for the same job. My boat has four separate pumps to drain water from the bilge - two electric and two manual. When the consequences of a problem are dire, it seems sensible to have multiple tools, each with a distinct focus. If I’m going to write high-quality software, I want to use as many tools as possible to be sure things work the way I expect.