Windows: coverage through multiprocessing not catched correctly

[Makman2]

Hi ;)

Recently I'm trying to get the test coverage to 100% on a windows machine, but it doesn't want to. My code under test uses an event-loop driven by a concurrent.futures.ProcessPoolExecutor. Coverage is 100% on a Linux machine, but not on Windows.

PR:
coala/coala#4475
Build on AppVeyor:
https://ci.appveyor.com/project/coala/coala/build/1.0.7774
Same build but on Linux (Travis):
https://travis-ci.org/coala/coala/builds/254572218?utm_source=github_status&utm_medium=notification
Relevant Python module/components under test:
https://github.com/coala/coala/tree/8f4113a2f471acf4c0121199c6ebb9b29dfa84ff/coalib/core, specificially

• https://github.com/coala/coala/blob/8f4113a2f471acf4c0121199c6ebb9b29dfa84ff/coalib/core/FileBear.py
• https://github.com/coala/coala/blob/8f4113a2f471acf4c0121199c6ebb9b29dfa84ff/coalib/core/ProjectBear.py

The missing lines are the ones inside def execute_task.

setup.cfg (and yes, coverage is enabled ;D):
https://github.com/coala/coala/blob/8f4113a2f471acf4c0121199c6ebb9b29dfa84ff/setup.cfg#L32

Exchanging the ProcessPoolExecutor with a ThreadPoolExecutor does the trick and produces full coverage. However it would be nice to have everything being tested with a ProcessPoolExecutor (especially due to possible pickling issues on Windows for subprocesses).

Not proven assumption:
I have the feeling that other parts of the code only run through a subprocess is measured properly, only those two modules under test don't work.

[ionelmc]

I suspect this is related to #139. A problem with Windows it's impossible to get 100% clean process termination (no sigterm, you either tell it to exit via ipc and wait, or you kill it).

[ionelmc]

How do you shutdown the executor in coala?

[Makman2]

Thanks for the quick response ;)

I'm waiting until all remaining tasks are processed. I wait with event_loop.run_forever() until event_loop.stop() is called. After that event_loop.close() is called for cleanup.
(See https://github.com/coala/coala/blob/8f4113a2f471acf4c0121199c6ebb9b29dfa84ff/coalib/core/Core.py#L360)

[Makman2]

Is this considered a "good" shutdown?

[ionelmc]

I doubt your ioloop magically figures out that executor.shutdown() needs to be run at some point ... try fixing that first.

[Makman2]

Damn yeah, that makes totally sense... I've actually forgotten that I always pass the executor explicitly to event_loop.run_in_executor, so how shall it know about my executor's lifetime...

I'm currently trying that out on the CI, I hope that does the trick :D

[Makman2]

No effect, tried to shutdown the executor manually at different places (inside the done-callback of my futures before and after event_loop.stop(), immediately before event_loop.close()), and also tried to register the executor as default (with event_loop.set_default_executor) with the hope that the event-loop takes the shutdown over completely.

[Makman2]

Any other ideas? :/

(I did workaround the problem now by executing my tests also with a ThreadPoolExecutor, where coverage data is correctly merged)

[ionelmc]

@Makman2 seems the branch is gone? can you republish it somewhere so I can try it?

[Makman2]

Oh I accidentally removed it... my bad sorry :/
Recreated problem on coala/coala#4589
(The PR shuts down the executor, but without effect like mentioned above)

[Makman2]

Corresponding Windows build:
https://ci.appveyor.com/project/coala/coala/build/1.0.9309/job/bvh96djsd64dfmmw

[ionelmc]

Well I've tried to run it locally but it has failed tests and the whole suite timeout (no coverage report). Please provide a more minimal reproducer.

[Makman2]

You can only run the relevant core tests by doing py.test -k core. Then you don't need to run the complete test suite ;) (note that you will get many files not covered 100%, but the relevant files mentioned above - FileBear, ProjectBear and DependencyBear - have the same missing lines as running the whole test suite)