API Reference

Package Layout

mitogen Package

mitogen.core

mitogen.master

mitogen.parent

mitogen.fakessh

_images/fakessh.svg

Message Class

Router Class

Connection Methods

Router.buildah(container=None, buildah_path=None, username=None, **kwargs)

Construct a context on the local machine over a buildah invocation. Accepts all parameters accepted by local(), in addition to:

Parameters

  • container (str) – The name of the Buildah container to connect to.

  • buildah_path (str) – Filename or complete path to the buildah binary. PATH will be searched if given as a filename. Defaults to buildah.

  • username (str) – Username to use, defaults to unset.

Router.fork(on_fork=None, on_start=None, debug=False, profiling=False, via=None)

Construct a context on the local machine by forking the current process. The forked child receives a new identity, sets up a new broker and router, and responds to function calls identically to children created using other methods.

The use of this method is strongly discouraged. It requires Python 2.6 or newer, as older Pythons made no effort to reset threading state upon fork.

For long-lived processes, local() is always better as it guarantees a pristine interpreter state that inherited little from the parent. Forking should only be used in performance-sensitive scenarios where short-lived children must be spawned to isolate potentially buggy code, and only after accounting for all the bad things possible as a result of, at a minimum:

  • Files open in the parent remaining open in the child, causing the lifetime of the underlying object to be extended indefinitely.

    • From the perspective of external components, this is observable in the form of pipes and sockets that are never closed, which may break anything relying on closure to signal protocol termination.

    • Descriptors that reference temporary files will not have their disk space reclaimed until the child exits.

  • Third party package state, such as urllib3’s HTTP connection pool, attempting to write to file descriptors shared with the parent, causing random failures in both parent and child.

  • UNIX signal handlers installed in the parent process remaining active in the child, despite associated resources, such as service threads, child processes, resource usage counters or process timers becoming absent or reset in the child.

  • Library code that makes assumptions about the process ID remaining unchanged, for example to implement inter-process locking, or to generate file names.

  • Anonymous MAP_PRIVATE memory mappings whose storage requirement doubles as either parent or child dirties their pages.

  • File-backed memory mappings that cannot have their space freed on disk due to the mapping living on in the child.

  • Difficult to diagnose memory usage and latency spikes due to object graphs becoming unreferenced in either parent or child, causing immediate copy-on-write to large portions of the process heap.

  • Locks held in the parent causing random deadlocks in the child, such as when another thread emits a log entry via the logging package concurrent to another thread calling fork(), or when a C extension module calls the C library allocator, or when a thread is using the C library DNS resolver, for example via socket.gethostbyname().

  • Objects existing in Thread-Local Storage of every non-fork() thread becoming permanently inaccessible, and never having their object destructors called, including TLS usage by native extension code, triggering many new variants of all the issues above.

  • Pseudo-Random Number Generator state that is easily observable by network peers to be duplicate, violating requirements of cryptographic protocols through one-time state reuse. In the worst case, children continually reuse the same state due to repeatedly forking from a static parent.

fork() cleans up Mitogen-internal objects, in addition to locks held by the logging package, reseeds random.random(), and the OpenSSL PRNG via ssl.RAND_add(), but only if the ssl module is already loaded. You must arrange for your program’s state, including any third party packages in use, to be cleaned up by specifying an on_fork function.

The associated stream implementation is mitogen.fork.Stream.

Parameters

  • on_fork (function) – Function invoked as on_fork() from within the child process. This permits supplying a program-specific cleanup function to break locks and close file descriptors belonging to the parent from within the child.

  • on_start (function) – Invoked as on_start(econtext) from within the child process after it has been set up, but before the function dispatch loop starts. This permits supplying a custom child main function that inherits rich data structures that cannot normally be passed via a serialization.

  • via (mitogen.core.Context) – Same as the via parameter for local().

  • debug (bool) – Same as the debug parameter for local().

  • profiling (bool) – Same as the profiling parameter for local().

Router.local(remote_name=None, python_path=None, debug=False, connect_timeout=None, profiling=False, via=None)

Construct a context on the local machine as a subprocess of the current process. The associated stream implementation is mitogen.master.Stream.

Parameters

  • remote_name (str) –

    The argv[0] suffix for the new process. If remote_name is test, the new process argv[0] will be mitogen:test.

    If unspecified, defaults to <username>@<hostname>:<pid>.

    This variable cannot contain slash characters, as the resulting argv[0] must be presented in such a way as to allow Python to determine its installation prefix. This is required to support virtualenv.

  • python_path (str|list) –

    String or list path to the Python interpreter to use for bootstrap. Defaults to sys.executable for local connections, and python for remote connections.

    It is possible to pass a list to invoke Python wrapped using another tool, such as ["/usr/bin/env", "python"].

  • debug (bool) – If True, arrange for debug logging (enable_debug()) to be enabled in the new context. Automatically True when enable_debug() has been called, but may be used selectively otherwise.

  • unidirectional (bool) – If True, arrange for the child’s router to be constructed with unidirectional routing enabled. Automatically True when it was enabled for this router, but may still be explicitly set to False.

  • connect_timeout (float) – Fractional seconds to wait for the subprocess to indicate it is healthy. Defaults to 30 seconds.

  • profiling (bool) – If True, arrange for profiling (profiling) to be enabled in the new context. Automatically True when profiling is True, but may be used selectively otherwise.

  • via (mitogen.core.Context) –

    If not None, arrange for construction to occur via RPCs made to the context via, and for ADD_ROUTE messages to be generated as appropriate.

    # SSH to the remote machine.
remote_machine = router.ssh(hostname='mybox.com')

# Use the SSH connection to create a sudo connection.
remote_root = router.sudo(username='root', via=remote_machine)

Router.doas(username=None, password=None, doas_path=None, password_prompt=None, incorrect_prompts=None, **kwargs)

Construct a context on the local machine over a doas invocation. The doas process is started in a newly allocated pseudo-terminal, and supports typing interactive passwords.

Accepts all parameters accepted by local(), in addition to:

Parameters

  • username (str) – Username to use, defaults to root.

  • password (str) – The account password to use if requested.

  • doas_path (str) – Filename or complete path to the doas binary. PATH will be searched if given as a filename. Defaults to doas.

  • password_prompt (bytes) – A string that indicates doas is requesting a password. Defaults to Password:.

  • incorrect_prompts (list) – List of bytestrings indicating the password is incorrect. Defaults to (b”doas: authentication failed”).

Raises

mitogen.doas.PasswordError – A password was requested but none was provided, the supplied password was incorrect, or the target account did not exist.

Router.docker(container=None, image=None, docker_path=None, **kwargs)

Construct a context on the local machine within an existing or temporary new Docker container using the docker program. One of container or image must be specified.

Accepts all parameters accepted by local(), in addition to:

Parameters

  • container (str) – Existing container to connect to. Defaults to None.

  • username (str) – Username within the container to setuid() to. Defaults to None, which Docker interprets as root.

  • image (str) – Image tag to use to construct a temporary container. Defaults to None.

  • docker_path (str) – Filename or complete path to the Docker binary. PATH will be searched if given as a filename. Defaults to docker.

Router.jail(container, jexec_path=None, **kwargs)

Construct a context on the local machine within a FreeBSD jail using the jexec program.

Accepts all parameters accepted by local(), in addition to:

Parameters

  • container (str) – Existing container to connect to. Defaults to None.

  • username (str) – Username within the container to setuid() to. Defaults to None, which jexec interprets as root.

  • jexec_path (str) – Filename or complete path to the jexec binary. PATH will be searched if given as a filename. Defaults to /usr/sbin/jexec.

Router.kubectl(pod, kubectl_path=None, kubectl_args=None, **kwargs)

Construct a context in a container via the Kubernetes kubectl program.

Accepts all parameters accepted by local(), in addition to:

Parameters

  • pod (str) – Kubernetes pod to connect to.

  • kubectl_path (str) – Filename or complete path to the kubectl binary. PATH will be searched if given as a filename. Defaults to kubectl.

  • kubectl_args (list) – Additional arguments to pass to the kubectl command.

Router.lxc(container, lxc_attach_path=None, **kwargs)

Construct a context on the local machine within an LXC classic container using the lxc-attach program.

Accepts all parameters accepted by local(), in addition to:

Parameters

  • container (str) – Existing container to connect to. Defaults to None.

  • lxc_attach_path (str) – Filename or complete path to the lxc-attach binary. PATH will be searched if given as a filename. Defaults to lxc-attach.

Router.lxd(container, lxc_path=None, **kwargs)

Construct a context on the local machine within a LXD container using the lxc program.

Accepts all parameters accepted by local(), in addition to:

Parameters

  • container (str) – Existing container to connect to. Defaults to None.

  • lxc_path (str) – Filename or complete path to the lxc binary. PATH will be searched if given as a filename. Defaults to lxc.

Router.podman(container=None, podman_path=None, username=None, **kwargs)

Construct a context on the local machine over a podman invocation. Accepts all parameters accepted by local(), in addition to:

Parameters

  • container (str) – The name of the Podman container to connect to.

  • podman_path (str) – Filename or complete path to the podman binary. PATH will be searched if given as a filename. Defaults to podman.

  • username (str) – Username to use, defaults to unset.

Router.setns(container, kind, username=None, docker_path=None, lxc_info_path=None, machinectl_path=None, **kwargs)

Construct a context in the style of local(), but change the active Linux process namespaces via calls to setns(2) before executing Python.

The namespaces to use, and the active root file system are taken from the root PID of a running Docker, LXC, LXD, or systemd-nspawn container.

The setns method depends on the built-in ctypes module, and thus does not support Python 2.4.

A program is required only to find the root PID, after which management of the child Python interpreter is handled directly.

Parameters

  • container (str) – Container to connect to.

  • kind (str) – One of docker, lxc, lxd or machinectl.

  • username (str) – Username within the container to setuid() to. Defaults to root.

  • docker_path (str) – Filename or complete path to the Docker binary. PATH will be searched if given as a filename. Defaults to docker.

  • lxc_path (str) – Filename or complete path to the LXD lxc binary. PATH will be searched if given as a filename. Defaults to lxc.

  • lxc_info_path (str) – Filename or complete path to the LXC lxc-info binary. PATH will be searched if given as a filename. Defaults to lxc-info.

  • machinectl_path (str) – Filename or complete path to the machinectl binary. PATH will be searched if given as a filename. Defaults to machinectl.

Router.su(username=None, password=None, su_path=None, password_prompt=None, incorrect_prompts=None, **kwargs)

Construct a context on the local machine over a su invocation. The su process is started in a newly allocated pseudo-terminal, and supports typing interactive passwords.

Accepts all parameters accepted by local(), in addition to:

Parameters

  • username (str) – Username to pass to su, defaults to root.

  • password (str) – The account password to use if requested.

  • su_path (str) – Filename or complete path to the su binary. PATH will be searched if given as a filename. Defaults to su.

  • password_prompt (bytes) – The string that indicates su is requesting a password. Defaults to Password:.

  • incorrect_prompts (str) – Strings that signal the password is incorrect. Defaults to (“su: sorry”, “su: authentication failure”).

Raises

mitogen.su.PasswordError – A password was requested but none was provided, the supplied password was incorrect, or (on BSD) the target account did not exist.

Router.sudo(username=None, sudo_path=None, password=None, **kwargs)

Construct a context on the local machine over a sudo invocation. The sudo process is started in a newly allocated pseudo-terminal, and supports typing interactive passwords.

Accepts all parameters accepted by local(), in addition to:

Parameters

  • username (str) – Username to pass to sudo as the -u parameter, defaults to root.

  • sudo_path (str) – Filename or complete path to the sudo binary. PATH will be searched if given as a filename. Defaults to sudo.

  • password (str) – The password to use if/when sudo requests it. Depending on the sudo configuration, this is either the current account password or the target account password. mitogen.sudo.PasswordError will be raised if sudo requests a password but none is provided.

  • set_home (bool) – If True, request sudo set the HOME environment variable to match the target UNIX account.

  • preserve_env (bool) – If True, request sudo to preserve the environment of the parent process.

  • selinux_type (str) – If not None, the SELinux security context to use.

  • selinux_role (str) – If not None, the SELinux role to use.

  • sudo_args (list) – Arguments in the style of sys.argv that would normally be passed to sudo. The arguments are parsed in-process to set equivalent parameters. Re-parsing ensures unsupported options cause mitogen.core.StreamError to be raised, and that attributes of the stream match the actual behaviour of sudo.

Router.ssh(hostname, username=None, ssh_path=None, ssh_args=None, port=None, check_host_keys='enforce', password=None, identity_file=None, identities_only=True, compression=True, **kwargs)

Construct a remote context over an OpenSSH ssh invocation.

The ssh process is started in a newly allocated pseudo-terminal to support typing interactive passwords and responding to prompts, if a password is specified, or check_host_keys=accept. In other scenarios, BatchMode is enabled and no PTY is allocated. For many-target configurations, both options should be avoided as most systems have a conservative limit on the number of pseudo-terminals that may exist.

Accepts all parameters accepted by local(), in addition to:

Parameters

  • username (str) – The SSH username; default is unspecified, which causes SSH to pick the username to use.

  • ssh_path (str) – Absolute or relative path to ssh. Defaults to ssh.

  • ssh_args (list) – Additional arguments to pass to the SSH command.

  • port (int) – Port number to connect to; default is unspecified, which causes SSH to pick the port number.

  • check_host_keys (str) –

    Specifies the SSH host key checking mode. Defaults to enforce.

    • ignore: no host key checking is performed. Connections never fail due to an unknown or changed host key.

    • accept: known hosts keys are checked to ensure they match, new host keys are automatically accepted and verified in future connections.

    • enforce: known host keys are checked to ensure they match, unknown hosts cause a connection failure.

  • password (str) – Password to type if/when ssh requests it. If not specified and a password is requested, mitogen.ssh.PasswordError is raised.

  • identity_file (str) –

    Path to an SSH private key file to use for authentication. Default is unspecified, which causes SSH to pick the identity file.

    When this option is specified, only identity_file will be used by the SSH client to perform authenticaion; agent authentication is automatically disabled, as is reading the default private key from ~/.ssh/id_rsa, or ~/.ssh/id_dsa.

  • identities_only (bool) – If True and a password or explicit identity file is specified, instruct the SSH client to disable any authentication identities inherited from the surrounding environment, such as those loaded in any running ssh-agent, or default key files present in ~/.ssh. This ensures authentication attempts only occur using the supplied password or SSH key.

  • compression (bool) – If True, enable ssh compression support. Compression has a minimal effect on the size of modules transmitted, as they are already compressed, however it has a large effect on every remaining message in the otherwise uncompressed stream protocol, such as function call arguments and return values.

  • ssh_debug_level (int) – Optional integer 0..3 indicating the SSH client debug level.

Raises

  • mitogen.ssh.PasswordError – A password was requested but none was specified, or the specified password was incorrect.

  • mitogen.ssh.HostKeyError – When check_host_keys is set to either accept, indicates a previously recorded key no longer matches the remote machine. When set to enforce, as above, but additionally indicates no previously recorded key exists for the remote machine.

Context Class

Receiver Class

Sender Class

Select Class

Channel Class

Broker Class

Fork Safety

Utility Functions

mitogen.core.now()

A reference to time.time() on Python 2, or time.monotonic() on Python >3.3. We prefer time.monotonic() when available to ensure timers are not impacted by system clock changes.

A random assortment of utility functions useful on masters and children.

@mitogen.utils.with_router

Decorator version of run_with_router(). Example:

@with_router
def do_stuff(router, arg):
    pass

do_stuff(blah, 123)

Exceptions