Philosophy

Let’s dig into why meta-programming function signatures is a good idea, and then we’ll cover some principals.

The why: intuitive design

Python lacks tooling to dynamically create callable signatures (without resorting to exec()). While this sounds esoteric, it’s actually a big source of error and frustration for authors and users. Have you ever encountered a function that looks like this: execute(*args, **kwargs)?

How about a real world example: the stdlib logging module. Inspecting one of the six logging methods (i.e. logging.debug(), logging.info(), logging.warning(), logging.error(), logging.critical(), and logging.exception()) we get a meaningless signature:

Inspecting any of the six logging methods we get a very limited understanding of how to use the function:

  • logging.debug(msg, *args, **kwargs),
  • logging.info(msg, *args, **kwargs),
  • logging.warning(msg, *args, **kwargs),
  • logging.error(msg, *args, **kwargs),
  • logging.critical(msg, *args, **kwargs), and
  • logging.exception(msg, *args, **kwargs).

Furthermore, the docstring messages available via the builtin help() provide minimally more insight:

>>> import logging
>>> help(logging.warning)
Help on function warning in module logging:

warning(msg, *args, **kwargs)
    Log a message with severity 'WARNING' on the root logger.
    If the logger has no handlers, call basicConfig() to add a console handler with a pre-defined format.

Of course the basic function of logging.warning() is to output a string, so it’d be excusable if you guessed that *args and **kwargs serve the same function as str.format(). Let’s try it:

>>> logging.warning('{user} changed a password', user='dave')
TypeError: _log() got an unexpected keyword argument 'user'

It’s arguable that this signature is worse than useless for code consumers - it’s led to an incorrect inference of behavior. If we look at the extended, online documentation for logging.warning(), we’re redirected further to the online documentation for logging.debug() which clarifies the role of the var-positional argument *args and var-keyword argument **kwargs [1].

logging.debug(msg, *args, **kwargs)

Logs a message with level DEBUG on the root logger. The msg is the message format string, and the args are the arguments which are merged into msg using the string formatting operator. (Note that this means that you can use keywords in the format string, together with a single dictionary argument.)

There are three keyword arguments in kwargs which are inspected: exc_info which, if it does not evaluate as false, causes exception information to be added to the logging message. If an exception tuple (in the format returned by sys.exc_info()) is provided, it is used; otherwise, sys.exc_info() is called to get the exception information.

The second optional keyword argument is stack_info, which defaults to False. If true, stack information is added to the logging message, including the actual logging call. Note that this is not the same stack information as that displayed through specifying exc_info: The former is stack frames from the bottom of the stack up to the logging call in the current thread, whereas the latter is information about stack frames which have been unwound, following an exception, while searching for exception handlers.

You can specify stack_info independently of exc_info, e.g. to just show how you got to a certain point in your code, even when no exceptions were raised. The stack frames are printed following a header line which says:

…

The third optional keyword argument is extra which can be used to pass a dictionary which is used to populate the __dict__ of the LogRecord created for the logging event with user-defined attributes. These custom attributes can then be used as you like. For example, they could be incorporated into logged messages. For example:

…

That’s a bit of documentation, but it uncovers why our attempt at supplying keyword arguments raises a TypeError. The string formatting that the logging methods provide has no relation to the string formatting provided by str.format() from PEP 3101 (introduced in Python 2.6 and Python 3.0).

In fact, there is significant amounts of documentation clarifying formatting style compatibility with the logging methods. But, let’s have some empathy; the Python core developers certainly don’t want to repeat themselves six times – once for each logging level – right?

This example illuminates the problem that forge sets out to solve: writing, testing and maintaining signatures requires too much effort. Left to their own devices, authors instead resort to hacks like signing a function with a var-keyword parameter (e.g. **kwargs). But is there method madness? Code consumers (collaborators and users) are left in the dark, asking “what parameters are really accepted; what should I pass?”.

The how: magic-free manipulation

Modern Python (3.5+) advertises a callable signature by looking for:

  1. a __signature__ attribute on your callable
  2. devising a signature from the __code__ attribute of the callable

And it allows for type-hints on parameters and return-values by looking for:

  1. an __annotations__ attribute on the callable with a return key
  2. devising a signature from the __code__ attribute of the callable

When you call a function wrapped with forge, the following occurs:

  1. arguments are associated with the public-facing parameters
  2. pre-bound parameters are added to the arguments mapping
  3. default values are applied for missing parameters
  4. converters (as supplied) are applied to the default or provided values
  5. validators (as supplied) are called with the converted values
  6. the arguments are mapped onto the wrapped function’s signature
  7. the wrapped function is called with the mapped attributes

The what: applying the knowledge

Looking back on the code for logging.debug(), let’s try and improve upon this implementation by wrapping the standard logging methods with enough information to provide basic direction for end-users.

import logging
import forge

make_explicit = forge.sign(
    forge.arg('msg'),
    *forge.args('substitutions'),
    exc_info=forge.kwarg(default=False),
    stack_info=forge.kwarg(default=False),
    extras=forge.kwarg(factory=dict),
)
debug = make_explicit(logging.debug)
info = make_explicit(logging.info)
warning = make_explicit(logging.warning)
error = make_explicit(logging.error)
critical = make_explicit(logging.critical)
exception = make_explicit(logging.exception)

assert forge.stringify_callable(debug) == \
    'debug(msg, *substitutions, exc_info=False, stack_info=False, extras=<Factory dict>)'

We’ve aided our intuition about how to use these functions.

Forge provides a sane middle-ground for well-intentioned, albeit lazy package authors and pragmatic, albeit lazy package consumers to communicate functionality and intent.

The bottom-line: signatures shouldn’t be this hard

After a case-study with logging where we enhanced the code with context, let’s consider the modern state of Python signatures beyond the stdlib.

Codebases you the broadly adopted sqlalchemy or graphene could benefit, as could third party corporate APIs which expect you to identify subtleties.

Driving developers from their IDE to your documentation is an dark pattern. Be a good community member – write cleanly and clearly.

Design principals

The API emulates usage.
forge provides an API for making function signatures more literate. Therefore, the library is designed in a literate way. Users are encouraged to supply positional-only and positional-or-keyword parameters as positional arguments, the var-positional parameter for positional-expansion (e.g. *forge.args), keyword-only parameters as keyword arguments, and the var-keyword parameter for keyword expansion (e.g. **forge.kwargs).
Minimal API impact.

Your callable, and it’s underlying code is 100% unmodified, organic. You can even get the original function by accessing the function’s __wrapped__ attribute.

Function in, function out: no hybrid instance-callables produced. classmethod`(), staticmethod`(), and property`() are all supported.

Performance matters.

forge was written from the ground up with an eye on performance, so it does the heavy lifting once, upfront rather than every time it’s called.

All classes use __slots__ for speeder attribute access. PyPy 6.0.0+ has first class support.

Immutable and flexible.
forge classes are immutable, but also flexible enough to support dynamic usage. You can share an FParameter or FSignature without fearing for your previously signed classes.
Type-Hints available.
forge supports the use of type-hints by providing an API for supplying types on parameters. In addition, forge itself is written with type-hints.
100% Coverage and Linted
forge maintains 100% code-coverage through unit testing. Code is also linted with mypy and pylint during automated testing upon every git push.

What forge is not

forge isn’t an interface to the wild-west that is exec() or eval().

All forge does is:

  1. takes your new signature,
  2. wraps your old callable,
  3. routes calls between the two

The mapper is built prior to execution (for speed). It’s available for inspection, but immutable (at __mapper__`). The callable remains unmodified and intact (at :attr:__wrapped__).

Common names: forge.arg and forge.kwarg

Based on a quick, informal poll of #python, many developers don’t know the formal parameter names. Given a function that looks like:

def func(a, b=3, *args, c=3, **kwargs):
    pass
  • a is often referred to as an argument (an arg), and
  • c is often referred to as a keyword argument (a kwarg),
  • b is usually bucketed as either of the above,
  • *args is simply referred to as args, and
  • **kwargs is simply referred to as kwargs.

Officially, that’s inaccurate. - a and b are positional-or-keyword parameters, - c is a keyword-only parameter, - args is a var-positional parameter, and - kwargs is a var-keyword parameter.

Python developers are often pragmatic developers, so forge was written in a supportive manner. Therefore, the following synonyms are defined:

Use whichever variant you please.

Footnotes

[1]Logging module documentation