Staging
v0.5.1
https://github.com/python/cpython
Revision 976da903a746a5455998e9ca45fbc4d3ad3479d8 authored by Alexey Izbyshev on 24 October 2020, 00:47:01 UTC, committed by GitHub on 24 October 2020, 00:47:01 UTC
* bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe

When used to run a new executable image, fork() is not a good choice
for process creation, especially if the parent has a large working set:
fork() needs to copy page tables, which is slow, and may fail on systems
where overcommit is disabled, despite that the child is not going to
touch most of its address space.

Currently, subprocess is capable of using posix_spawn() instead, which
normally provides much better performance. However, posix_spawn() does not
support many of child setup operations exposed by subprocess.Popen().
Most notably, it's not possible to express `close_fds=True`, which
happens to be the default, via posix_spawn(). As a result, most users
can't benefit from faster process creation, at least not without
changing their code.

However, Linux provides vfork() system call, which creates a new process
without copying the address space of the parent, and which is actually
used by C libraries to efficiently implement posix_spawn(). Due to sharing
of the address space and even the stack with the parent, extreme care
is required to use vfork(). At least the following restrictions must hold:

* No signal handlers must execute in the child process. Otherwise, they
  might clobber memory shared with the parent, potentially confusing it.

* Any library function called after vfork() in the child must be
  async-signal-safe (as for fork()), but it must also not interact with any
  library state in a way that might break due to address space sharing
  and/or lack of any preparations performed by libraries on normal fork().
  POSIX.1 permits to call only execve() and _exit(), and later revisions
  remove vfork() specification entirely. In practice, however, almost all
  operations needed by subprocess.Popen() can be safely implemented on
  Linux.

* Due to sharing of the stack with the parent, the child must be careful
  not to clobber local variables that are alive across vfork() call.
  Compilers are normally aware of this and take extra care with vfork()
  (and setjmp(), which has a similar problem).

* In case the parent is privileged, special attention must be paid to vfork()
  use, because sharing an address space across different privilege domains
  is insecure[1].

This patch adds support for using vfork() instead of fork() on Linux
when it's possible to do safely given the above. In particular:

* vfork() is not used if credential switch is requested. The reverse case
  (simple subprocess.Popen() but another application thread switches
  credentials concurrently) is not possible for pure-Python apps because
  subprocess.Popen() and functions like os.setuid() are mutually excluded
  via GIL. We might also consider to add a way to opt-out of vfork() (and
  posix_spawn() on platforms where it might be implemented via vfork()) in
  a future PR.

* vfork() is not used if `preexec_fn != None`.

With this change, subprocess will still use posix_spawn() if possible, but
will fallback to vfork() on Linux in most cases, and, failing that,
to fork().

[1] https://ewontfix.com/7

Co-authored-by: Gregory P. Smith [Google LLC] <gps@google.com>
1 parent 16ee68d
History
Tip revision: 976da903a746a5455998e9ca45fbc4d3ad3479d8 authored by Alexey Izbyshev on 24 October 2020, 00:47:01 UTC
bpo-35823: subprocess: Use vfork() instead of fork() on Linux when safe (GH-11671)
Tip revision: 976da90
File Mode Size
.azure-pipelines
.github
Doc
Grammar
Include
Lib
Mac
Misc
Modules
Objects
PC
PCbuild
Parser
Programs
Python
Tools
m4
.gitattributes -rw-r--r-- 1.9 KB
.gitignore -rw-r--r-- 1.8 KB
.travis.yml -rw-r--r-- 8.4 KB
CODE_OF_CONDUCT.md -rw-r--r-- 630 bytes
LICENSE -rw-r--r-- 13.6 KB
Makefile.pre.in -rw-r--r-- 67.7 KB
README.rst -rw-r--r-- 10.0 KB
aclocal.m4 -rw-r--r-- 15.2 KB
config.guess -rwxr-xr-x 43.1 KB
config.sub -rwxr-xr-x 35.4 KB
configure -rwxr-xr-x 501.3 KB
configure.ac -rw-r--r-- 169.3 KB
install-sh -rwxr-xr-x 15.0 KB
netlify.toml -rw-r--r-- 82 bytes
pyconfig.h.in -rw-r--r-- 44.6 KB
setup.py -rw-r--r-- 108.7 KB

README.rst

back to top