Release Notes¶

To avail of fixes in an unreleased version, please download a ZIP file directly from GitHub.

In progress (unreleased)¶

v0.3.18 (2024-11-07)¶

#1083 ansible_mitogen: Templated become method (e.g. ansible_become_method).
#1083 ansible_mitogen: Templated become flag (e.g. ansible_become_method, become keyword).

v0.3.17 (2024-11-07)¶

#1182 CI: Fix incorrect world readable/writable file permissions on SSH key mitogen__has_sudo_pubkey.key during Ansible tests.
#1083 ansible_mitogen: Templated SSH private key file (e.g. ansible_private_key_file).
#1083 ansible_mitogen: Templated SSH host key checking (e.g. ansible_host_key_checking, ansible_ssh_host_key_checking).
#1083 ansible_mitogen: Templated host address (e.g. ansible_host, ansible_ssh_host)
#1184 Test templated SSH host key checking in task vars

v0.3.16 (2024-11-05)¶

#1083 ansible_mitogen: Templated become executable (e.g. become_exe).
#1083 ansible_mitogen: Templated become executable arguments (e.g. become_flags).
#1083 ansible_mitogen: Templated ssh executable (ansible_ssh_executable).
#1083 ansible_mitogen: Fixed templated connection options during a meta: reset_connection task.
#1129 CI: Migrated macOS 12 runners to macOS 13, due to EOL.

v0.3.15 (2024-10-28)¶

#905 ansible_mitogen: Support templated SSH command arguments (e.g. ansible_ssh_args, ansible_ssh_extra_args).
#692 tests: Fix and re-enable several sudo tests
#1083 ansible_mitogen: Support templated become password (e.g. ansible_become_pass, ansible_sudo_pass)

v0.3.14 (2024-10-16)¶

#1159 CI: Reduce number of Jobs by parameterizing Mitogen Docker SSH tests
#1083 ansible_mitogen: Support templated become username.

v0.3.13 (2024-10-09)¶

#1138 CI: Complete migration from Azure DevOps Pipelines to GitHub Actions
#1116 ansible_mitogen: Support for templated variable ansible_ssh_user.
#978 ansible_mitogen: Support templated Ansible SSH port.
#1073 Python 3.13 support

v0.3.12 (2024-10-07)¶

#1106 ansible_mitogen: Support for ansible_ssh_password connection variable, and templated SSH connection password.
#1136 tests: Improve Ansible fail_msg formatting.
#1137 tests: Ignore inventory files of inactive tests & benchmarks
#1138 CI: Add re-actors/alls-green GitHub Actions job to simplify branch protections configuration.

v0.3.11 (2024-09-30)¶

#1127 mitogen: Consolidate mitogen backward compatibility fallbacks and polyfills into mitogen.core
#1127 ansible_mitogen: Remove backward compatibility fallbacks for Python 2.4 & 2.5.
#1127 ansible_mitogen: Remove fallback imports for Ansible releases before 2.10
#1127 ansible_mitogen: Consolidate Python 2 & 3 compatibility
#1128 CI: Start migration from Azure DevOps to GitHub Actions

v0.3.10 (2024-09-20)¶

#950 Fix Solaris/Illumos/SmartOS compatibility with become
#1087 Fix mitogen.core.StreamError when Ansible template module is called with a dest: filename that has an extension
#1110 Fix mitogen.core.StreamError when Ansible copy module is called with a file larger than 124 kibibytes (ansible_mitogen.connection.Connection.SMALL_FILE_LIMIT)
#905 Initial support for templated ansible_ssh_args, ansible_ssh_common_args, and ansible_ssh_extra_args variables. NB: play or task scoped variables will probably still fail.
#694 CI: Fixed a race condition and some resource leaks causing some of intermittent failures when running the test suite.

v0.3.9 (2024-08-13)¶

#1097 Respect ansible_facts.discovered_interpreter_python when executing non new-style modules (e.g. JSONARGS style, WANT_JSON style).
#1074 Support Ansible 10 (ansible-core 2.17)

v0.3.8 (2024-07-30)¶

#952 Fix Ansible –ask-become-pass, add test coverage
#957 Fix Ansible exception when executing against 10s of hosts “ValueError: filedescriptor out of range in select()”
#1066 Support Ansible ansible_host_key_checking & ansible_ssh_host_key_checking
#1090 CI: Migrate macOS integration tests to macOS 12, drop Python 2.7 jobs

v0.3.7 (2024-04-08)¶

#1021 Support for Ansible 8 (ansible-core 2.15)
tests: Replace uses of include: & import:, unsupported in Ansible 9
#1053 Support for Ansible 9 (ansible-core 2.16)

v0.3.6 (2024-04-04)¶

#974 Support Ansible 7
#1046 Raise TypeError in <mitogen.util.cast()>() when casting a string subtype to bytes() or str() fails. This is potentially an API breaking change. Failures previously passed silently.
#1046 Add <ansible_mitogen.util.cast()>(), to cast ansible.utils.unsafe_proxy.AnsibleUnsafe objects in Ansible 7+.

v0.3.5 (2024-03-17)¶

#987 Support Python 3.11
#885 Fix PermissionError in importlib when becoming an unprivileged user with Python 3.x
#1033 Support PEP 451, required by Python 3.12
#1033 Support Python 3.12

v0.3.4 (2023-07-02)¶

#929 Support Ansible 6 and ansible-core 2.13
#832 Fix runtime error when using the ansible.builtin.dnf module multiple times
#925 ansible_mitogen.connection.Connection no longer tries to close the connection on destruction. This is expected to reduce cases of mitogen.core.Error: An attempt was made to enqueue a message with a Broker that has already exitted. However it may result in resource leaks.
#659 Removed mitogen.compat.simplejson, not needed with Python 2.7+, contained Python 3.x syntax errors
#983 CI: Removed PyPI faulthandler requirement from tests
#1001 CI: Fixed Debian 9 & 11 tests

v0.3.3 (2022-06-03)¶

#906 Support packages dynamically inserted into sys.modules, e.g. distro >= 1.7.0 as ansible.module_utils.distro.
#918 Support Python 3.10
#920 Support Ansible podman connection plugin
#836 mitogen.utils.with_router() decorator preserves the docstring in addition to the name.
#936 fetch module no longer emits [DEPRECATION WARNING]: The ‘_remote_checksum()’ method is deprecated.

v0.3.2 (2022-01-12)¶

#891 Correct Framework :: Ansible Trove classifier

v0.3.1 (unreleased)¶

#874 Support for Ansible 5 (ansible-core 2.12)
#774 Fix bootstrap failures on macOS 11.x and 12.x, involving Python 2.7 wrapper
#834 Support for Ansible 3 and 4 (ansible-core 2.11)
#869 Continuous Integration tests are now run with Tox
#869 Continuous Integration tests now cover CentOS 6 & 8, Debian 9 & 11, Ubuntu 16.04 & 20.04
#860 Add initial support for podman connection (w/o Ansible support yet)
#873 python -c … first stage no longer uses platform` to detect the macOS release
#876 python -c … first stage no longer contains tab characters, to reduce size
#878 Continuous Integration tests now correctly perform comparisons of 2 digit versions
#878 Kubectl connector fixed with Ansible 2.10 and above

v0.3.0 (2021-11-24)¶

This release separates itself from the v0.2.X releases. Ansible’s API changed too much to support backwards compatibility so from now on, v0.2.X releases will be for Ansible < 2.10 and v0.3.X will be for Ansible 2.10+. See here for details.

#827 NewStylePlanner: detect ansible_collections imports
#770 better check for supported Ansible version
#731 ansible 2.10 support
#652 support for ansible collections import hook
#847 Removed historic Continuous Integration reverse shell

v0.2.10 (2021-11-24)¶

#597 mitogen does not support Ansible 2.8 Python interpreter detection
#655 wait_for_connection gives errors
#672 cannot perform relative import error
#673 mitogen fails on RHEL8 server with bash /usr/bin/python: No such file or directory
#676 mitogen fail to run playbook without “/usr/bin/python” on target host
#716 fetch fails with “AttributeError: ‘ShellModule’ object has no attribute ‘tmpdir’”
#756 ssh connections with check_host_keys=’accept’ would timeout, when using recent OpenSSH client versions.
#758 fix initilialisation of callback plugins in test suite, to address a KeyError in ansible.plugins.callback.CallbackBase.v2_runner_on_start()
#775 Test with Python 3.9
#775 Add msvcrt to the default module deny list

v0.2.9 (2019-11-02)¶

This release contains minimal changes required for Ansible 2.9.

#633: meta: reset_connection could fail to reset a connection when become: true was set on the playbook.
#660: Ansible 2.9 is supported.

Thanks!¶

Mitogen would not be possible without the support of users. A huge thanks for bug reports, testing, features and fixes in this release contributed by

Can Ozokur and Olexandr Minzak.

v0.2.8 (2019-08-18)¶

This release includes Ansible 2.8 and SELinux support, fixes for two deadlocks, and major internal design overhauls in preparation for future functionality.

Enhancements¶

#556, #587: Ansible 2.8 is supported. Become plugins (#631) and interpreter discovery (#630) are not yet handled.
#419, #470: file descriptor usage is approximately halved, as it is no longer necessary to separately manage read and write sides to work around a design problem.
#419: setup for all connections happens almost entirely on one thread, reducing contention and context switching early in a run.
#419: Connection setup is better pipelined, eliminating some network round-trips. Most infrastructure is in place to support future removal of the final round-trips between a target booting and receiving function calls.
#595: the buildah() connection method is available to manipulate Buildah containers, and is exposed to Ansible as the buildah connection plug-in.
#615: a modified fetch module implements streaming transfer even when become is active, avoiding excess CPU and memory spikes, and improving performance. A representative copy of two 512 MiB files drops from 55.7 seconds to 6.3 seconds, with peak memory usage dropping from 10.7 GiB to 64.8 MiB. 1
Operon no longer requires a custom library installation, both Ansible and Operon are supported by a single Mitogen release.
The MITOGEN_CPU_COUNT variable shards the connection multiplexer into per-CPU workers. This may improve throughput for large runs involving file transfer, and is required for future functionality. One multiplexer starts by default, to match existing behaviour.
d6faff06, 807cbef9, e93762b3, 50bfe4c7: locking is avoided on hot paths, and some locks are released before waking a thread that must immediately acquire the same lock.

Mitogen for Ansible¶

#363: fix an obscure race matching Permission denied errors from some versions of su(1) running on heavily loaded machines.
#410: Uses of unix(7) sockets are replaced with traditional pipe(7) pairs when SELinux is detected, to work around a broken heuristic in common SELinux policies that prevents inheriting unix(7) sockets across privilege domains.
#467: an incompatibility running Mitogen under Molecule was resolved.
#547, #598: fix a deadlock during initialization of connections, async tasks, tasks using custom module_utils, mitogen_task_isolation: fork modules, and modules present on an internal blacklist. This would manifest as a timeout or hang, was easily hit, had been present since 0.2.0, and likely impacted many users.
#549: the open file limit is increased to the permitted hard limit. It is common for distributions to ship with a higher hard limit than the default soft limit, allowing “too many open files” errors to be avoided more often in large runs without user intervention.
#558, #582: on Ansible 2.3 a directory was unconditionally deleted after the first module belonging to an action plug-in had executed, causing the unarchive module to fail.
#578: the extension could crash while rendering an error due to an incorrect format string.
#590: the importer can handle modules that replace themselves in sys.modules with completely unrelated modules during import, as in the case of Ansible 2.8 ansible.module_utils.distro.
#591: the working directory is reset between tasks to ensure os.getcwd() cannot fail, in the same way AnsibleModule resets it during initialization. However this restore happens before the module executes, ensuring code that calls os.getcwd() prior to AnsibleModule initialization, such as the Ansible 2.7 pip module, cannot fail due to the actions of a prior task.
#593: the SSH connection method exposes mitogen_ssh_keepalive_interval and mitogen_ssh_keepalive_count variables, and the default timeout for an SSH server has been increased from 15*3 seconds to 30*10 seconds.
#600: functionality to reflect changes to /etc/environment did not account for Unicode file contents. The file may now use any single byte encoding.
#602: connection configuration is more accurately inferred for meta: reset_connection, the synchronize module, and for any action plug-ins that establish additional connections.
#598, #605: fix a deadlock managing a shared counter used for load balancing, present since 0.2.4.
#615: streaming is implemented for the fetch module and other actions that transfer files from targets to the controller. Previously files were sent in one message, requiring them to fit in RAM and be smaller than an internal message size sanity check. Transfers from controller to targets have been streaming since 0.2.0.
7ae926b3: the lineinfile module leaked writable temporary file descriptors between Ansible 2.7.0 and 2.8.2. When lineinfile created or modified a script, and that script was later executed, the execution could fail with “text file busy”. Temporary descriptors are now tracked and cleaned up on exit for all modules.

Core Library¶

Log readability is improving and many repr() strings are more descriptive. The old pseudo-function-call format is migrating to readable output where possible. For example, “Stream(ssh:123).connect()” might be written “connecting to ssh:123”.
In preparation for reducing default log output, many messages are delivered to per-component loggers, including messages originating from children, enabling logging aggregation to function as designed. An importer message like:
```
12:00:00 D mitogen.ctx.remotehost mitogen: loading module "foo"
```
Might instead be logged to the mitogen.importer.[remotehost] logger:
```
12:00:00 D mitogen.importer.[remotehost] loading module "foo"
```
Allowing a filter or handler for mitogen.importer to select that logger in every process. This introduces a small risk of leaking memory in long-lived programs, as logger objects are internally persistent.
bytearray() was removed from the list of supported serialization types. It was never portable between Python versions, unused, and never made much sense to support.
#170: to improve subprocess management and asynchronous connect, a TimerList interface is available, accessible as Broker.timers in an asynchronous context.
#419: the internal Stream has been refactored into many new classes, modularizing protocol behaviour, output buffering, line-oriented input parsing, option handling and connection management. Connection setup is internally asynchronous, laying most groundwork for fully asynchronous connect, proxied Ansible become plug-ins, and in-process SSH.
#169, #419: zombie subprocess reaping has vastly improved, by using timers to efficiently poll for a child to exit, and delaying shutdown while any subprocess remains. Polling avoids process-global configuration such as a SIGCHLD handler, or signal.set_wakeup_fd() available in modern Python.
#256, #419: most os.dup() use was eliminated, along with most manual file descriptor management. Descriptors are trapped in os.fdopen() objects at creation, ensuring a leaked object will close itself, and ensuring every descriptor is fused to a closed flag, preventing historical bugs where a double close could destroy unrelated descriptors.
#533: routing accounts for a race between a parent (or cousin) sending a message to a child via an intermediary, where the child had recently disconnected, and DEL_ROUTE propagating from the intermediary to the sender, informing it that the child no longer exists. This condition is detected at the intermediary and a dead message is returned to the sender.

Previously since the intermediary had already removed its route for the child, the route messages upwards rule would be triggered, causing the message (with a privileged src_id/auth_id) to be sent upstream, resulting in a bad auth_id error logged at the first upstream parent, and a possible hang due to a request message being dropped.
#586: fix import of __main__ on later versions of Python 3 when running from the interactive console.
#606: fix example code on the documentation front page.
#612: fix various errors introduced by stream refactoring.
#615: when routing fails to deliver a message for some reason other than the sender cannot or should not reach the recipient, and no reply-to address is present on the message, instead send a dead message to the original recipient. This ensures a descriptive message is delivered to a thread sleeping on the reply to a function call, where the reply might be dropped due to exceeding the maximum configured message size.
#624: the number of threads used for a child’s automatically initialized service thread pool has been reduced from 16 to 2. This may drop to 1 in future, and become configurable via a Router option.
a5536c35: avoid quadratic buffer management when logging lines received from a child’s redirected standard IO.
49a6446a: the empty() methods of Latch, Receiver and Select are obsoleted by a more general size() method. empty() will be removed in 0.3
ecc570cb: previously mitogen.select.Select.add() would enqueue one wake event when adding an existing receiver, latch or subselect that contained multiple buffered items, causing get() calls to block or fail even though data existed to return.
5924af15: [security] unidirectional routing, where contexts may optionally only communicate with parents and never siblings (so that air-gapped networks cannot be unintentionally bridged) was not inherited when a child was initiated directly from another child. This did not effect Ansible, since the controller initiates any new child used for routing, only forked tasks are initiated by children.

Thanks!¶

Mitogen would not be possible without the support of users. A huge thanks for bug reports, testing, features and fixes in this release contributed by Andreas Hubert, Anton Markelov, Dan, Dave Cottlehuber, Denis Krienbühl, El Mehdi CHAOUKI, Florent Dutheil, James Hogarth, Jordan Webb, Julian Andres Klode, Marc Hartmayer, Nigel Metheringham, Orion Poplawski, Pieter Voet, Stefane Fermigier, Szabó Dániel Ernő, Ulrich Schreiner, Vincent S. Cojot, yen, Yuki Nishida, @alexhexabeam, @DavidVentura, @dbiegunski, @ghp-rr, @migalsp, @rizzly, @SQGE, and @tho86.

Footnotes

1: Peak RSS of controller and target as measured with /usr/bin/time -v ansible-playbook -c local using the reproduction supplied in #615.

v0.2.7 (2019-05-19)¶

This release primarily exists to add a descriptive error message when running on Ansible 2.8, which is not yet supported.

Fixes¶

#557: fix a crash when running on machines with high CPU counts.
#570: the firewalld module internally caches a dbus name that changes across firewalld restarts, causing a failure if the service is restarted between firewalld module invocations.
#575: fix a crash when rendering an error message to indicate no usable temporary directories could be found.
#576: fix a crash during startup on SuSE Linux 11, due to an incorrect version compatibility check in the Mitogen code.
#581: a mitogen_mask_remote_name Ansible variable is exposed, to allow masking the username, hostname and process ID of ansible-playbook running on the controller machine.
#587: display a friendly message when running on an unsupported version of Ansible, to cope with potential influx of 2.8-related bug reports.

Thanks!¶

Mitogen would not be possible without the support of users. A huge thanks for bug reports, testing, features and fixes in this release contributed by Orion Poplawski, Thibaut Barrère, @Moumoutaru, and @polski-g.

v0.2.6 (2019-03-06)¶

Fixes¶

#542: some versions of OS X ship a default Python that does not support select.poll(). Restore the 0.2.3 behaviour of defaulting to Kqueue in this case, but still prefer select.poll() if it is available.
#545: an optimization introduced in #493 caused a 64-bit integer to be assigned to a 32-bit field on ARM 32-bit targets, causing runs to fail.
#548: mitogen_via= could fail when the selected transport was set to smart.
#550: avoid some broken TTY-related ioctl() calls on Windows Subsystem for Linux 2016 Anniversary Update.
#554: third party Ansible action plug-ins that invoked _make_tmp_path() repeatedly could trigger an assertion failure.
#555: work around an old idiom that reloaded sys in order to change the interpreter’s default encoding.
ffae0355: needless information was removed from the documentation and installation procedure.

Core Library¶

#535: to support function calls on a service pool from another thread, mitogen.select.Select additionally permits waiting on mitogen.core.Latch.
#535: mitogen.service.Pool.defer allows any function to be enqueued for the thread pool from another thread.
#535: a new mitogen.os_fork module provides a os.fork() wrapper that pauses thread activity during fork. On Python<2.6, mitogen.core.Broker and mitogen.service.Pool automatically record their existence so that a os.fork() monkey-patch can automatically pause them for any attempt to start a subprocess.
ca63c26e: mitogen.core.Latch.put()’s obj argument was made optional.

Thanks!¶

Mitogen would not be possible without the support of users. A huge thanks for bug reports, testing, features and fixes in this release contributed by Fabian Arrotin, Giles Westwood, Matt Layman, Percy Grunwald, Petr Enkov, Tony Finch, @elbunda, and @zyphermonkey.

v0.2.5 (2019-02-14)¶

Fixes¶

#511, #536: changes in 0.2.4 to repair delegate_to handling broke default ansible_python_interpreter handling. Test coverage was added.
#532: fix a race in the service used to propagate Ansible modules, that could easily manifest when starting asynchronous tasks in a loop.
#536: changes in 0.2.4 to support Python 2.4 interacted poorly with modules that imported simplejson from a controller that also loaded an incompatible newer version of simplejson.
#537: a swapped operator in the CPU affinity logic meant 2 cores were reserved on 1<n<4 core machines, rather than 1 core as desired. Test coverage was added.
#538: the source distribution includes a LICENSE file.
#539: log output is no longer duplicated when the Ansible log_path setting is enabled.
#540: the stderr stream of async module invocations was previously discarded.
#541: Python error logs originating from the boto package are quiesced, and only appear in -vvv output. This is since EC2 modules may trigger errors during normal operation, when retrying transiently failing requests.
748f5f67, 21ad299d, 8ae6ca1d, 7fd0d349: the ansible_ssh_host, ansible_ssh_user, ansible_user, ansible_become_method, and ansible_ssh_port variables more correctly match typical behaviour when mitogen_via= is active.
2a8567b4: fix a race initializing a child’s service thread pool on Python 3.4+, due to a change in locking scheme used by the Python import mechanism.

Thanks!¶

Mitogen would not be possible without the support of users. A huge thanks for bug reports, testing, features and fixes in this release contributed by Carl George, Guy Knights, and Josh Smift.

v0.2.4 (2019-02-10)¶

Mitogen for Ansible¶

This release includes a huge variety of important fixes and new optimizations. It is 35% faster than 0.2.3 on a synthetic 64 target run that places heavy load on the connection multiplexer.

Enhancements¶

#76, #351, #352: disconnect propagation has improved, allowing Ansible to cancel waits for responses from abruptly disconnected targets. This ensures a task will reliably fail rather than hang, for example on network failure or EC2 instance maintenance.
#369, #407: Connection.reset() is implemented, allowing meta: reset_connection to shut down the remote interpreter as documented, and improving support for the reboot module.
09aa27a6: the mitogen_host_pinned strategy wraps the host_pinned strategy introduced in Ansible 2.7.
#477: Python 2.4 is fully supported by the core library and tested automatically, in any parent/child combination of 2.4, 2.6, 2.7 and 3.6 interpreters.
#477: Ansible 2.3 is fully supported and tested automatically. In combination with the core library Python 2.4 support, this allows Red Hat Enterprise Linux 5 targets to be managed with Mitogen. The simplejson package need not be installed on such targets, as is usually required by Ansible.
#412: to simplify diagnosing connection configuration problems, Mitogen ships a mitogen_get_stack action that is automatically added to the action plug-in path. See The mitogen_get_stack Action for more information.
152effc2, bd4b04ae: a CPU affinity policy was added for Linux controllers, reducing latency and SMP overhead on hot paths exercised for every task. This yielded a 19% speedup in a 64-target job composed of many short tasks, and should easily be visible as a runtime improvement in many-host runs.
2b44d598: work around a defective caching mechanism by pre-heating it before spawning workers. This saves 40% runtime on a synthetic repetitive task.
0979422a: an expensive dependency scanning step was redundantly invoked for every task, bottlenecking the connection multiplexer.
eaa990a97: a new mitogen_ssh_compression variable is supported, allowing Mitogen’s default SSH compression to be disabled. SSH compression is a large contributor to CPU usage in many-target runs, and severely limits file transfer. On a “shell: hostname” task repeated 500 times, Mitogen requires around 800 bytes per task with compression, rising to 3 KiB without. File transfer throughput rises from ~25MiB/s when enabled to ~200MiB/s when disabled.
#260, a18a083c: brokers no longer wait for readiness indication to transmit, and instead assume transmission will succeed. As this is usually true, one loop iteration and two poller reconfigurations are avoided, yielding a significant reduction in interprocess round-trip latency.
#415, #491, #493: the interface employed for in-process queues changed from kqueue(2) / epoll(7) to poll(2), which requires no setup or teardown, yielding a 38% latency reduction for inter-thread communication.

Fixes¶

#251, #359, #396, #401, #404, #412, #434, #436, #465: connection delegation and delegate_to: handling suffered a major regression in 0.2.3. The 0.2.2 behaviour has been restored, and further work has been made to improve the compatibility of connection delegation’s configuration building methods.
#323, #333: work around a Windows Subsystem for Linux bug that caused tracebacks to appear during shutdown.
#334: the SSH method tilde-expands private key paths using Ansible’s logic. Previously the path was passed unmodified to SSH, which expanded it using pwd.getpwnam(). This differs from os.path.expanduser(), which uses the HOME environment variable if it is set, causing behaviour to diverge when Ansible was invoked across user accounts via sudo.
#364: file transfers from controllers running Python 2.7.2 or earlier could be interrupted due to a forking bug in the tempfile module.
#370: the Ansible reboot module is supported.
#373: the LXC and LXD methods print a useful hint on failure, as no useful error is normally logged to the console by these tools.
#374, #391: file transfer and module execution from 2.x controllers to 3.x targets was broken due to a regression caused by refactoring, and compounded by #426.
#400: work around a threading bug in the AWX display callback when running with high verbosity setting.
#409: the setns method was silently broken due to missing tests. Basic coverage was added to prevent a recurrence.
#409: the LXC and LXD methods support mitogen_lxc_path and mitogen_lxc_attach_path variables to control the location of third pary utilities.
#410: the sudo method supports the SELinux --type and --role options.
#420: if a Connection was constructed in the Ansible top-level process, for example while executing meta: reset_connection, resources could become undesirably shared in subsequent children.
#426: an oversight while porting to Python 3 meant no automated 2->3 tests were running. A significant number of 2->3 bugs were fixed, mostly in the form of Unicode/bytes mismatches.
#429: the sudo method can now recognize internationalized password prompts.
#362, #435: the previous fix for slow Python 2.x subprocess creation on Red Hat caused newly spawned children to have a reduced open files limit. A more intrusive fix has been added to directly address the problem without modifying the subprocess environment.
#397, #454: the previous approach to handling modern Ansible temporary file cleanup was too aggressive, and could trigger early finalization of Cython-based extension modules, leading to segmentation faults.
#499: the allow_same_user Ansible configuration setting is respected.
#527: crashes in modules are trapped and reported in a manner that matches Ansible. In particular, a module crash no longer leads to an exception that may crash the corresponding action plug-in.
dc1d4251: the synchronize module could fail with the Docker transport due to a missing attribute.
599da068: fix a race when starting async tasks, where it was possible for the controller to observe no status file on disk before the task had a chance to write one.
2c7af9f04: Ansible modules were repeatedly re-transferred. The bug was hidden by the previously mandatorily enabled SSH compression.

Core Library¶

#76: routing records the destination context IDs ever received on each stream, and when disconnection occurs, propagates mitogen.core.DEL_ROUTE messages towards every stream that ever communicated with the disappearing peer, rather than simply towards parents. Conversations between nodes anywhere in the tree receive mitogen.core.DEL_ROUTE when either participant disconnects, allowing receivers to wake with mitogen.core.ChannelError, even when one participant is not a parent of the other.
#109, 57504ba6: newer Python 3 releases explicitly populate sys.meta_path with importer internals, causing Mitogen to install itself at the end of the importer chain rather than the front.
#310: support has returned for trying to figure out the real source of non-module objects installed in sys.modules, so they can be imported. This is needed to handle syntax sugar used by packages like plumbum.
#349: an incorrect format string could cause large stack traces when attempting to import built-in modules on Python 3.
#387, #413: dead messages include an optional reason in their body. This is used to cause mitogen.core.ChannelError to report far more useful diagnostics at the point the error occurs that previously would have been buried in debug log output from an unrelated context.
#408: a variety of fixes were made to restore Python 2.4 compatibility.
#399, #437: ignore a DeprecationWarning to avoid failure of the su method on Python 3.7.
#405: if an oversized message is rejected, and it has a reply_to set, a dead message is returned to the sender. This ensures function calls exceeding the configured maximum size crash rather than hang.
#406: mitogen.core.Broker did not call mitogen.core.Poller.close() during shutdown, leaking the underlying poller FD in masters and parents.
#406: connections could leak FDs when a child process failed to start.
#288, #406, #417: connections could leave FD wrapper objects that had not been closed lying around to be closed during garbage collection, causing reused FD numbers to be closed at random moments.
#411: the SSH method typed “y” rather than the requisite “yes” when check_host_keys=”accept” was configured. This would lead to connection timeouts due to the hung response.
#414, #425: avoid deadlock of forked children by reinitializing the mitogen.service pool lock.
#416: around 1.4KiB of memory was leaked on every RPC, due to a list of strong references keeping alive any handler ever registered for disconnect notification.
#418: the mitogen.parent.iter_read() helper would leak poller FDs, because execution of its finally block was delayed on Python 3. Now callers explicitly close the generator when finished.
#422: the fork method could fail to start if sys.stdout was opened in block buffered mode, and buffered data was pending in the parent prior to fork.
#438: a descriptive error is logged when stream corruption is detected.
#439: descriptive errors are raised when attempting to invoke unsupported function types.
#444: messages regarding unforwardable extension module are no longer logged as errors.
#445: service pools unregister the mitogen.core.CALL_SERVICE handle at shutdown, ensuring any outstanding messages are either processed by the pool as it shuts down, or have dead messages sent in reply to them, preventing peer contexts from hanging due to a forgotten buffered message.
#446: given thread A calling mitogen.core.Receiver.close(), and thread B, C, and D sleeping in mitogen.core.Receiver.get(), previously only one sleeping thread would be woken with mitogen.core.ChannelError when the receiver was closed. Now all threads are woken per the docstring.
#447: duplicate attempts to invoke mitogen.core.Router.add_handler() cause an error to be raised, ensuring accidental re-registration of service pools are reported correctly.
#448: the import hook implementation now raises ModuleNotFoundError instead of ImportError in Python 3.6 and above, to cope with an upcoming version of the subprocess module requiring this new subclass to be raised.
#453: the loggers used in children for standard IO redirection have propagation disabled, preventing accidental reconfiguration of the logging package in a child from setting up a feedback loop.
#456: a descriptive error is logged when mitogen.core.Broker.defer() is called after the broker has shut down, preventing new messages being enqueued that will never be sent, and subsequently producing a program hang.
#459: the beginnings of a mitogen.master.Router.get_stats() call has been added. The initial statistics cover the module loader only.
#462: Mitogen could fail to open a PTY on broken Linux systems due to a bad interaction between the glibc grantpt() function and an incorrectly mounted /dev/pts filesystem. Since correct group ownership is not required in most scenarios, when this problem is detected, the PTY is allocated and opened directly by the library.
#479: Mitogen could fail to import __main__ on Python 3.4 and newer due to a breaking change in the pkgutil API. The program’s main script is now handled specially.
#481: the version of sudo that shipped with CentOS 5 replaced itself with the program to be executed, and therefore did not hold any child PTY open on our behalf. The child context is updated to preserve any PTY FD in order to avoid the kernel sending SIGHUP early during startup.
#523: the test suite didn’t generate a code coverage report if any test failed.
#524: Python 3.6+ emitted a DeprecationWarning for mitogen.utils.run_with_router().
#529: Code coverage of the test suite was not measured across all Python versions.
16ca111e: handle OpenSSH 7.5 permission denied prompts when ~/.ssh/config rewrites are present.
9ec360c2: a new mitogen.core.Broker.defer_sync() utility function is provided.
f20e0bba: mitogen.service.FileService.register_prefix() permits granting unprivileged access to whole filesystem subtrees, rather than single files at a time.
8f85ee03: mitogen.core.Router.myself() returns a mitogen.core.Context referring to the current process.
824c7931: exceptions raised by the import hook were updated to include probable reasons for a failure.
57b652ed: a stray import meant an extra roundtrip and ~4KiB of data was wasted for any context that imported mitogen.parent.

Thanks!¶

Mitogen would not be possible without the support of users. A huge thanks for bug reports, testing, features and fixes in this release contributed by Alex Willmer, Andreas Krüger, Anton Stroganov, Berend De Schouwer, Brian Candler, dsgnr, Duane Zamrok, Eric Chang, Gerben Meijer, Guy Knights, Jesse London, Jiří Vávra, Johan Beisser, Jonathan Rosser, Josh Smift, Kevin Carter, Mehdi, Michael DeHaan, Michal Medvecky, Mohammed Naser, Peter V. Saveliev, Pieter Avonts, Ross Williams, Sergey, Stéphane, Strahinja Kustudic, Tom Parker-Shemilt, Younès HAFRI, @killua-eu, @myssa91, @ohmer1, @s3c70r, @syntonym, @trim777, @whky, and @yodatak.

v0.2.3 (2018-10-23)¶

Mitogen for Ansible¶

Enhancements¶

#315, #392: Ansible 2.6 and 2.7 are supported.
#321, #336: temporary file handling was simplified, undoing earlier damage caused by compatibility fixes, improving 2.6 compatibility, and avoiding two network roundtrips for every related action (assemble, aws_s3, copy, patch, script, template, unarchive, uri). See Temporary Files for a complete description.
#376, #377: the kubectl connection type is now supported. Contributed by Yannig Perré.
084c0ac0: avoid a roundtrip in copy and template due to an unfortunate default.
7458dfae: avoid a roundtrip when transferring files smaller than 124KiB. Copy and template actions are now 2-RTT, reducing runtime for a 20-iteration template loop over a 250 ms link from 30 seconds to 10 seconds compared to v0.2.2, down from 120 seconds compared to vanilla.
#337: To avoid a scaling limitation, a PTY is no longer allocated for an SSH connection unless the configuration specifies a password.
d62e6e2a: many-target runs executed the dependency scanner redundantly due to missing synchronization, wasting significant runtime in the connection multiplexer. In one case work was reduced by 95%, which may manifest as faster runs.
5189408e: threads are cooperatively scheduled, minimizing GIL contention, and reducing context switching by around 90%. This manifests as an overall improvement, but is easily noticeable on short many-target runs, where startup overhead dominates runtime.
The faulthandler module is automatically activated if it is installed, simplifying debugging of hangs. See Diagnosing Hangs for details.
The MITOGEN_DUMP_THREAD_STACKS environment variable’s value now indicates the number of seconds between stack dumps. See Diagnosing Hangs for details.

Fixes¶

#251, #340: Connection Delegation could establish connections to the wrong target when delegate_to: is present.
#291: when Mitogen had previously been installed using pip or setuptools, the globally installed version could conflict with a newer version bundled with an extension that had been installed using the documented steps. Now the bundled library always overrides over any system-installed copy.
#324: plays with a custom module_utils would fail due to fallout from the Python 3 port and related tests being disabled.
#331: the connection multiplexer subprocess always exits before the main Ansible process, ensuring logs generated by it do not overwrite the user’s prompt when -vvv is enabled.
#332: support a new sys.excepthook()-based module exit mechanism added in Ansible 2.6.
#338: compatibility: changes to /etc/environment and ~/.pam_environment made by a task are reflected in the runtime environment of subsequent tasks. See Process Environment Emulation for a complete description.
#343: the sudo --login option is supported.
#344: connections no longer fail when the controller’s login username contains slashes.
#345: the IdentitiesOnly yes option is no longer supplied to OpenSSH by default, better matching Ansible’s behaviour.
#355: tasks configured to run in an isolated forked subprocess were forked from the wrong parent context. This meant built-in modules overridden via a custom module_utils search path may not have had any effect.
#362: to work around a slow algorithm in the subprocess module, the maximum number of open files in processes running on the target is capped to 512, reducing the work required to start a subprocess by >2000x in default CentOS configurations.
#397: recent Mitogen master versions could fail to clean up temporary directories in a number of circumstances, and newer Ansibles moved to using atexit to effect temporary directory cleanup in some circumstances.
b9112a9c, 2c287801: OpenSSH 7.5 permission denied prompts are now recognized. Contributed by Alex Willmer.
A missing check caused an exception traceback to appear when using the ansible command-line tool with a missing or misspelled module name.
Ansible since >=2.7 began importing __main__ from ansible.module_utils.basic, causing an error during execution, due to the controller being configured to refuse network imports outside the ansible.* namespace. Update the target implementation to construct a stub __main__ module to satisfy the otherwise seemingly vestigial import.

Core Library¶

A new mitogen.parent.CallChain class abstracts safe pipelining of related function calls to a target context, cancelling the chain if an exception occurs.
#305: fix a long-standing minor race relating to the logging framework, where no route for Message.. would frequently appear during startup.
#313: mitogen.parent.Context.call() was documented as capable of accepting static methods. While possible on Python 2.x the result is ugly, and in every case it should be trivial to replace with a classmethod. The documentation was fixed.
#337: to avoid a scaling limitation, a PTY is no longer allocated for each OpenSSH client if it can be avoided. PTYs are only allocated if a password is supplied, or when host_key_checking=accept. This is since Linux has a default of 4096 PTYs (kernel.pty.max), while OS X has a default of 127 and an absolute maximum of 999 (kern.tty.ptmx_max).
#339: the LXD connection method was erroneously executing LXC Classic commands.
#345: the SSH connection method allows optionally disabling IdentitiesOnly yes.
#356: if the master Python process does not have sys.executable set, the default Python interpreter used for new children on the local machine defaults to "/usr/bin/python".
#366, #380: attempts by children to import __main__ where the main program module lacks an execution guard are refused, and an error is logged. This prevents a common and highly confusing error when prototyping new scripts.
#371: the LXC connection method uses a more compatible method to establish an non-interactive session. Contributed by Brian Candler.
af2ded66: add mitogen.fork.on_fork() to allow non-Mitogen managed process forks to clean up Mitogen resources in the child.
d6784242: the setns method always resets HOME, SHELL, LOGNAME and USER environment variables to an account in the target container, defaulting to root.
830966bf: the UNIX listener no longer crashes if the peer process disappears in the middle of connection setup.

Thanks!¶

Mitogen would not be possible without the support of users. A huge thanks for bug reports, testing, features and fixes in this release contributed by Alex Russu, Alex Willmer, atoom, Berend De Schouwer, Brian Candler, Dan Quackenbush, dsgnr, Jesse London, John McGrath, Jonathan Rosser, Josh Smift, Luca Nunzi, Orion Poplawski, Peter V. Saveliev, Pierre-Henry Muller, Pierre-Louis Bonicoli, Prateek Jain, RedheatWei, Rick Box, nikitakazantsev12, Tawana Musewe, Timo Beckers, and Yannig Perré.

v0.2.2 (2018-07-26)¶

Mitogen for Ansible¶

#291: ansible_*_interpreter variables are parsed using a restrictive shell-like syntax, supporting a common idiom where ansible_python_interpreter is set to /usr/bin/env python.
#299: fix the network_cli connection type when the Mitogen strategy is active. Mitogen cannot help network device connections, however it should still be possible to use device connections while Mitogen is active.
#301: variables like $HOME in the remote_tmp setting are evaluated correctly.
#303: the Doas become method is supported. Contributed by Mike Walker.
#309: fix a regression to process environment cleanup, caused by the change in v0.2.1 to run local tasks with the correct environment.
#317: respect the verbosity setting when writing to Ansible’s log_path, if it is enabled. Child log filtering was also incorrect, causing the master to needlessly wake many times. This nets a 3.5% runtime improvement running against the local machine.
The mitogen_ssh_debug_level variable is supported, permitting SSH debug output to be included in Mitogen’s -vvv output when both are specified.

Core Library¶

#291: the python_path parameter may specify an argument vector prefix rather than a string program path.
#300: the broker could crash on OS X during shutdown due to scheduled kqueue(2) filter changes for descriptors that were closed before the IO loop resumes. As a temporary workaround, kqueue’s bulk change feature is not used.
#303: the Doas become method is now supported. Contributed by Mike Walker.
#307: SSH login banner output containing the word ‘password’ is no longer confused for a password prompt.
#319: SSH connections would fail immediately on Windows Subsystem for Linux, due to use of TCSAFLUSH with termios.tcsetattr(). The flag is omitted if WSL is detected.
#320: The OS X poller could spuriously wake up due to ignoring an error bit set on events returned by the kernel, manifesting as a failure to read from an unrelated descriptor.
#342: The network_cli connection type would fail due to a missing internal SSH plugin method.
Standard IO forwarding accidentally configured the replacement stdout and stderr write descriptors as non-blocking, causing subprocesses that generate more output than kernel buffer space existed to throw errors. The write ends are now configured as blocking.
When mitogen.core.enable_profiling() is active, mitogen.service threads are profiled just like other threads.
The ssh_debug_level parameter is supported, permitting SSH debug output to be redirected to a Mitogen logger when specified.
Debug logs containing command lines are printed with the minimal quoting and escaping required.

Thanks!¶

Mitogen would not be possible without the support of users. A huge thanks for the bug reports and pull requests in this release contributed by Alex Russu, Andy Freeland, Ayaz Ahmed Khan, Colin McCarthy, Dan Quackenbush, Duane Zamrok, Gonzalo Servat, Guy Knights, Josh Smift, Mark Janssen, Mike Walker, Orion Poplawski, falbanese, Tawana Musewe, and Zach Swanson.

v0.2.1 (2018-07-10)¶

Mitogen for Ansible¶

#297: compatibility: local actions set their working directory to that of their defining playbook, and inherit a process environment as if they were executed as a subprocess of the forked task worker.

v0.2.0 (2018-07-09)¶

Mitogen 0.2.x is the inaugural feature-frozen branch eligible for fixes only, except for problem areas listed as in-scope below. While stable from a development perspective, it should still be considered “beta” at least for the initial releases.

In Scope

Python 3.x performance improvements
Subprocess reaping improvements
Major documentation improvements
PyPI/packaging improvements
Test suite improvements
Replacement CI system to handle every supported OS
Minor deviations from vanilla Ansible behaviour
Ansible raw action support

The goal is a tick/tock model where even-numbered series are a maturation of the previous unstable series, and unstable series are released on PyPI with --pre enabled. The API and user visible behaviour should remain unchanged within a stable series.

Mitogen for Ansible¶

Support for Ansible 2.3 - 2.7.x and any mixture of Python 2.6, 2.7 or 3.6 on controller and target nodes.
Drop-in support for many Ansible connection types.
Preview of Connection Delegation feature.
Built-in file transfer compatible with connection delegation.

Core Library¶

Synchronous connection establishment via OpenSSH, sudo, su, Docker, LXC and FreeBSD Jails, local subprocesses and os.fork(). Parallel connection setup is possible using multiple threads. Connections may be used from one or many threads after establishment.
UNIX masters and children, with Linux, MacOS, FreeBSD, NetBSD, OpenBSD and Windows Subsystem for Linux explicitly supported.
Automatic tests covering Python 2.6, 2.7 and 3.6 on Linux only.