|
| 1 | +================== |
| 2 | +Subprocess support |
| 3 | +================== |
| 4 | + |
| 5 | +Normally coverage writes the data via a pretty standard atexit handler. However, if the subprocess doesn't exit on its |
| 6 | +own then the atexit handler might not run. Why that happens is best left to the adventurous to discover by waddling |
| 7 | +though the Python bug tracker. |
| 8 | + |
| 9 | +pytest-cov supports subprocesses and multiprocessing, and works around these atexit limitations. However, there are a |
| 10 | +few pitfalls that need to be explained. |
| 11 | + |
| 12 | +If you use ``multiprocessing.Pool`` |
| 13 | +=================================== |
| 14 | + |
| 15 | +**pytest-cov** automatically registers a multiprocessing finalizer. The finalizer will only run reliably if the pool is |
| 16 | +closed. Closing the pool basically signals the workers that there will be no more work, and they will eventually exit. |
| 17 | +Thus one also needs to call `join` on the pool. |
| 18 | + |
| 19 | +If you use ``multiprocessing.Pool.terminate`` or the context manager API (``__exit__`` |
| 20 | +will just call ``terminate``) then the workers can get SIGTERM and then the finalizers won't run or complete in time. |
| 21 | +Thus you need to make sure your ``multiprocessing.Pool`` gets a nice and clean exit: |
| 22 | + |
| 23 | +.. code-block:: python |
| 24 | +
|
| 25 | + from multiprocessing import Pool |
| 26 | +
|
| 27 | + def f(x): |
| 28 | + return x*x |
| 29 | +
|
| 30 | + if __name__ == '__main__': |
| 31 | + p = Pool(5) |
| 32 | + try: |
| 33 | + print(p.map(f, [1, 2, 3])) |
| 34 | + finally: |
| 35 | + p.close() # Marks the pool as closed. |
| 36 | + p.join() # Waits for workers to exit. |
| 37 | +
|
| 38 | +
|
| 39 | +If you must use the context manager API (e.g.: the pool is managed in third party code you can't change) then you can |
| 40 | +register a cleaning SIGTERM handler like so: |
| 41 | + |
| 42 | +.. code-block:: python |
| 43 | +
|
| 44 | + from multiprocessing import Pool |
| 45 | +
|
| 46 | + def f(x): |
| 47 | + return x*x |
| 48 | +
|
| 49 | + if __name__ == '__main__': |
| 50 | + try: |
| 51 | + from pytest_cov.embed import cleanup_on_sigterm |
| 52 | + except ImportError: |
| 53 | + pass |
| 54 | + else: |
| 55 | + cleanup_on_sigterm() |
| 56 | +
|
| 57 | + with Pool(5) as p: |
| 58 | + print(p.map(f, [1, 2, 3])) |
| 59 | +
|
| 60 | +If you use ``multiprocessing.Process`` |
| 61 | +====================================== |
| 62 | + |
| 63 | +There's similar issue when using the ``Process`` objects. Don't forget to use ``.join()``: |
| 64 | + |
| 65 | +.. code-block:: python |
| 66 | +
|
| 67 | + from multiprocessing import Process |
| 68 | +
|
| 69 | + def f(name): |
| 70 | + print('hello', name) |
| 71 | +
|
| 72 | + if __name__ == '__main__': |
| 73 | + try: |
| 74 | + from pytest_cov.embed import cleanup_on_sigterm |
| 75 | + except ImportError: |
| 76 | + pass |
| 77 | + else: |
| 78 | + cleanup_on_sigterm() |
| 79 | +
|
| 80 | + p = Process(target=f, args=('bob',)) |
| 81 | + try: |
| 82 | + p.start() |
| 83 | + finally: |
| 84 | + p.join() # necessary so that the Process exists before the test suite exits (thus coverage is collected) |
| 85 | +
|
| 86 | +.. _cleanup_on_sigterm: |
| 87 | + |
| 88 | +If you got custom signal handling |
| 89 | +================================= |
| 90 | + |
| 91 | +**pytest-cov 2.6** has a rudimentary ``pytest_cov.embed.cleanup_on_sigterm`` you can use to register a SIGTERM handler |
| 92 | +that flushes the coverage data. |
| 93 | + |
| 94 | +**pytest-cov 2.7** adds a ``pytest_cov.embed.cleanup_on_signal`` function and changes the implementation to be more |
| 95 | +robust: the handler will call the previous handler (if you had previously registered any), and is re-entrant (will |
| 96 | +defer extra signals if delivered while the handler runs). |
| 97 | + |
| 98 | +For example, if you reload on SIGHUP you should have something like this: |
| 99 | + |
| 100 | +.. code-block:: python |
| 101 | +
|
| 102 | + import os |
| 103 | + import signal |
| 104 | +
|
| 105 | + def restart_service(frame, signum): |
| 106 | + os.exec( ... ) # or whatever your custom signal would do |
| 107 | + signal.signal(signal.SIGHUP, restart_service) |
| 108 | +
|
| 109 | + try: |
| 110 | + from pytest_cov.embed import cleanup_on_signal |
| 111 | + except ImportError: |
| 112 | + pass |
| 113 | + else: |
| 114 | + cleanup_on_signal(signal.SIGHUP) |
| 115 | +
|
| 116 | +Note that both ``cleanup_on_signal`` and ``cleanup_on_sigterm`` will run the previous signal handler. |
| 117 | + |
| 118 | +Alternatively you can do this: |
| 119 | + |
| 120 | +.. code-block:: python |
| 121 | +
|
| 122 | + import os |
| 123 | + import signal |
| 124 | +
|
| 125 | + try: |
| 126 | + from pytest_cov.embed import cleanup |
| 127 | + except ImportError: |
| 128 | + cleanup = None |
| 129 | +
|
| 130 | + def restart_service(frame, signum): |
| 131 | + if cleanup is not None: |
| 132 | + cleanup() |
| 133 | +
|
| 134 | + os.exec( ... ) # or whatever your custom signal would do |
| 135 | + signal.signal(signal.SIGHUP, restart_service) |
| 136 | +
|
| 137 | +If you use Windows |
| 138 | +================== |
| 139 | + |
| 140 | +On Windows you can register a handler for SIGTERM but it doesn't actually work. It will work if you |
| 141 | +`os.kill(os.getpid(), signal.SIGTERM)` (send SIGTERM to the current process) but for most intents and purposes that's |
| 142 | +completely useless. |
| 143 | + |
| 144 | +Consequently this means that if you use multiprocessing you got no choice but to use the close/join pattern as described |
| 145 | +above. Using the context manager API or `terminate` won't work as it relies on SIGTERM. |
| 146 | + |
| 147 | +However you can have a working handler for SIGBREAK (with some caveats): |
| 148 | + |
| 149 | +.. code-block:: python |
| 150 | +
|
| 151 | + import os |
| 152 | + import signal |
| 153 | +
|
| 154 | + def shutdown(frame, signum): |
| 155 | + # your app's shutdown or whatever |
| 156 | + signal.signal(signal.SIGBREAK, shutdown) |
| 157 | +
|
| 158 | + try: |
| 159 | + from pytest_cov.embed import cleanup_on_signal |
| 160 | + except ImportError: |
| 161 | + pass |
| 162 | + else: |
| 163 | + cleanup_on_signal(signal.SIGBREAK) |
| 164 | +
|
| 165 | +The `caveats <https://stefan.sofa-rockers.org/2013/08/15/handling-sub-process-hierarchies-python-linux-os-x/>`_ being |
| 166 | +roughly: |
| 167 | + |
| 168 | +* you need to deliver ``signal.CTRL_BREAK_EVENT`` |
| 169 | +* it gets delivered to the whole process group, and that can have unforeseen consequences |
0 commit comments