I recently added type annotations to my
epyc
and
epydemic
libraries. Making these work
while not sacrificing interoperability a wide range of Python versions
is quite delicate.
Python is inherently a dynamically-typed language. The parameters to
methods and functions are checked as they are called, leading to
TypeError
and ValueError
exceptions if they’re used
incorrectly. Dignifying this by calling it “duck
typing”
doesn’t alter the fundamental weakness — from the perspective of a
programming language person — that this opens-up in the heart of a
codebase. Unit testing isn’t a proper substitute for static checks.
Parts of the Python community have recently embraced this view too,
leading to the
typing
module
available from Python 3.5. This allows (but doesn’t require)
programmers to add type annotations to code that can then be checked
by external typecheckers. There are a range, including
mypy
and
pylance
(which is built into Microsoft’s VS Code).
As with any emerging feature in any language, the problem then becomes one of backwards compatibility. I wanted to retain compatibility with previous versions of Python prior to my current Python 3.8 — ideally all the way back to Python 3.5. That turned out to be a version too far, but getting support for Python 3.6 and later proved possible.
Basic type annotations as they are intended
We should of course start with the features we want. Type annotations can be attached to method declarations and object attributes:
class A: name : str = 'my name' def __init__(self, v : str =None): self.value = v def value(self) -> str: return self.v
Hopefully self-explanatory, and note that it doesn’t need any special imports. If we were to extend things a little, for example to use lists or dicts, then we need some more machinery:
from typing import List, Dict, Any class B: names : List[str] = [] values : List[Any] = [] def addPair(self, n: str, v : Any) -> Any: names.append(n) values.append(v) return v def asDict(self) -> Dict[str, Any]: d = dict() for i in range(len(names)): d[names[i]] = values[i] return d
Note that List[]
and Dict[]
are type constructors taking
their element types in square brackets. A typechecker can now work out
before execution that, for example (b.asDict())[1]
is type-incorrect.
Self types
How about a method that returns an instance of self
(or similar)?
The obvious approach is:
class C: def meAgain(self) -> C: return self
Surprisingly this will fail, as C
isn’t in scope in its own
definition. This is such a natural thing, though, that it’s going to
be supported in future versions and, from Python 3.8, is available as
an import from the __future__
module:
from __future__ import annotations class C: def meAgain(self) -> C: # now typechecks return self
(Remember that __future__
imports have to appear as the first code
line of a source file.)
Backporting type annotations
This all works fine in Python 3.8, so if that’s all you care about you
can stop here. However, a lot of Python scripts want (or need) to
continue to work with earlier versions, preferably without the
disaster-waiting-to-happen of maintaining parallel codebases. So can
we use typing
for earlier versions?
Type annotations first appeared in Python 3.5, but the ideas have been
evolving. If all you use are “normal” types such as builtins, class names, List[]
and so on, then everything works fine. But I
discovered that there are three exceptions.
1. Final[]
(and other) types
Suppose you have an attribute that you want to be constant. The
Final[]
type constructor lets you denote this:
from typing import List, Final class D VERSION : Final[str] = '1.2.1'
Final[]
wasn’t added to typing
until Python 3.8, though, so
running this code on Python 3.7 (for example) will fail. However,
there’s another module, typing_extensions
, that backports many new
features which can be imported if needed:
import sys if sys.version_info >= (3, 8): from typing import List, Final else: from typing import List from typing_extensions import Final class D VERSION : Final[str] = '1.2.1'
(The
documentation for typing_extensions
specifies the types it provides. They’re all
very specialised apart from Final[]
.) One the one hand this kind
of conditional importing is messy and inelegant; on the other hand, it
makes something possible that otherwise isn’t.
Of course you need to have typing_extensions
available to be
imported. This means that it has to be included in the
requirements.txt
and/or the setup.py
files for the
module. This requires using version annotations in those files. For
requirements.txt
we can “guard” the import with the Python
versions for which it applied:
typing_extensions; python_version <= '3.7'
For setup.py
we similarly include it as an “extra” requirement,
guarded by the version:
from setuptools import setup with open('README.rst') as f: longDescription = f.read() setup(name = 'my_module', version = ..., packages = ..., package_data = { 'my_module': [ 'py.typed' ] }, zip_safe = False, install_requires = [ ... ], extra_requires = { ':python_version < 3.8': [ 'typing_extensions' ] }, )
(All the ...
s are whatever you’d normally have in your setup script.)
Notice the package_data
attribute. This points to a file called
py.typed
which sits in the source tree and does … absolutely
nothing, except indicate that the module has type annotations that
can then be used by other modules that import it.
2. Self types
Methods that return instances of their own class need a __future__
import that’s only available from Python 3.8 onwards: used with
earlier versions, it will be rejected. Fortunately there’s a
workaround, which is that type annotations can be strings rather than
more general objects. So to annotate self types portably, replace the
__fiture__
import and use of the class name with a use of the
string name of the class name:
# Too far into the future... #from __future__ import annotations class C: def meAgain(self) -> 'C': return self
I find this very messy — but again, if you want portability, it’s a way round the problem that hopefully only appears in a few places within the code.
3. Variable annotations
Type annotations were introduced in Python 3.5, but annotations for variables only came in with Python 3.6. So code of this form will fail with a syntax error when run on Python 3.5:
class E: v : str = 'my value'
And as far as I can tell there’s nothing to be done about this: the syntax simply isn’t available. If you desperately need Python 3.5 then you need to not annotate variables; otherwise, your backport stops with Python 3.6. In my case I chose the latter option, as I had a lot of constant strings that needed to be annotated. Your constraints might lead to a different choice.
Conclusion
Feelings vary on the usefulness and Pythonic qualities of Python type annotations. For me, they help eliminate logic errors and improve the effectiveness of testing. The typecheckers do an impressive job of type inference too, which reduces the need for excessive annotations: I almost never use them for variables within methods, for example, just for method signatures and attributes. The backporting isn’t perfect but at least it’s possible, and it extends the usefulness of codebases without overly complicating things. I find that quite a Pythonic idea.