Zero-Cost Exceptions and Error Interfaces

I wouldn't have thought that 5 lines of code can inspire great conversations about permission versus forgiveness, early exit, and interfaces.

But that's exactly what I learned a few weeks ago when we set up an interesting poll on Twitter about implementing a relatively simple read function.

A lot has happened in the 27 days since this poll. This post won't comment on the changes around Twitter but will highlight some related features in the newly released Python 3.11. 🙂 It's time to revisit those 5 lines of code.

The Poll

The original question was: Which version of this code do you prefer?

Version A)

def read(self, id_: str) -> Item:
    result = self.items.get(id_)
    if not result:
        raise ItemNotFound(id_)
    return result

Version B)

def read(self, id_: str) -> Item:
    if result := self.items.get(id_):
        return result
    else:
        raise ItemNotFound(id_)

@PamphileRoy quickly suggested a 3rd option:

Version C)

def read(self, id_: str) -> Item:
    try:
        return self.items[id_]
    except KeyError as e:
        raise ItemNotFound(id_) from e

Look Before You Leap vs Ask for Forgiveness

The first fundamental question is: Should we use an if to check whether this item exists or rather a try-except block?

This is an area where code adhering to the semantics is also more performant.

The guideline:

• If the condition is likely to happen. => Use if.
• If the condition is unlikely to happen. => Use a try-except.

It might be intuitive why this is the guideline from a semantic point of view: Use exceptions for stuff that's considered an "exceptional" case. And if branches for the various happy paths.

But why is this guideline also wise from the performance point of view?

• Handling exceptions is expensive.
• Checking for exceptions is cheap.

If you're interested in some measurements about the costs of conditional statements and exception handling, check out this blog post by Sebastian Witowski. Or his talk Writing Faster Python 3 at EuroPython in 2022.

In the example above, we expect id_ to be a valid ID. Providing an invalid ID is an error case. So, we should use a try-except.

Python 3.11: Zero-Cost Exception Handling

On 24 October, Python 3.11 was released. Which makes the above guideline even more valid than before. While in earlier Python versions, the cost of checking for exceptions was low, in 3.11, it's practically zero.

This is due to a change in the compiler:

• In Python 3.10, try statements are compiled to a SETUP_FINALLY instruction.
• In Python 3.11, try statements are compiled to a NOP (no operation) instruction.

If you're curious how jump tables made this change possible, check out Real Python's article about Cool New Features in Python 3.11

The feature zero-cost exceptions was inspired by Java and C++. The idea behind it: While it makes sense to handle various errors, you probably want to optimize your code's speed in the happy path.

Why Not Just Return None?

Returning None is a quick solution, but it has several drawbacks:

• Callers of the function need to introduce None-checks. As we've seen above, these are more expensive than try-except blocks.
• These None-checks are also quite error-prone because None isn't the only False-equivalent return value.

For more examples of how returning None can go wrong, see Item 20 in the book Effective Python

Which Error To Raise?

Now that we've agreed on raising an error, the next question is: What type of error should this be? We'll discuss 3 solutions:

• Pass the error to the caller.
• Re-raise the error with a note.
• Raise a new error.

Pass the Error to the Caller

An interesting suggestion was not to use a try-except block at all. If a Exception occurs, let the caller of our function handle it.

def read(self, id_: str) -> Item:
    return self.items[id_]

This can be a great strategy, if the called code (in our example: self.items[id_]) already throws a quite specific exception.

It's less practical if the called code raises a very general exception, like a KeyError. The caller of our API might have a hard time figuring out where in the call hierarchy this KeyError happened.

Python 3.11: Add Notes to Errors

In Python 3.11, a further possibility was introduced: You can add notes to exceptions.

Let's assume that the called code raises a specific error. Instead of wrapping it into a new type of error, you can add a note to it:

def read(self, id_: str) -> Item:
    try:
        return self.items.get(id_)
    except SomeSpecificError as e:
        e.add_note(f"ID not found: {id_}")
        raise e

Raise a New Error from the Previous Error

In our example, we don't get a specific exception, just a rather generic KeyError. In that case, it makes sense to define a new exception class and raise that instead. Use the from syntax to include information about the original error in the stack trace:

def read(self, id_: str) -> Item:
    try:
        return self.items[id_]
    except KeyError as e:
        raise ItemNotFound(id_) from e

Define an Exception Hierarchy for Your API

In the book Effective Python, Brett Slatkin suggests: "Define a Root Exception to Insulate Callers from APIs" (Item 87)

1. Define a root exception for your API (This inherits from the built-in Exception class.)
2. All exceptions raised by the API inherit from this root exception

class ItemApiError(Exception):
    """Base class for all exceptions raised by the Item API."""


class ItemNotFoundError(ItemApiError):
    """No item found with the provided ID."""

This approach means a bit more upfront work when designing the API but gives a lot of flexibility to the consumers of the API. They can easily:

• catch all exceptions provided by the API with a single except block
• handle the various exceptions provided by the API in different ways

Conclusion

Python provides several ways to handle errors, and 3.11 introduced further great features in this area. If your code happens to be more complex than this tiny example (and especially if it contains async operations), make sure to check out exception groups as well.

Two questions to consider when dealing with errors:

• For if vs try decisions: Is this a branch of the happy path or an error case?
• For deciding which exception to raise: Which information does the caller need to handle this exception properly?

Further Resources

Twitter

The Twitter poll Thanks to everyone who contributed to this insightful conversation:

from Hacker News https://ift.tt/Uqbj2lP