A pytest plugin to run async tests 'concurrently'

[u/Royal-Fail3273]

What My Project Does

System/Integration tests sometimes can take really long time, due to spending huge amount of time waiting for external services responding. Pytest-asyncio make async tests testable, but run them sequentially. Pytest-xdist spinup multiple processes, blew up our fragile server during tests collection :(

• This plugin is to solve this by running asynchronous tests in true parallel, enabling faster execution for high I/O or network-bound test suites.
• Also give you high flexibility to specify tests that can run in parallel.
• Compatibility with Pytest-asyncio if you are already deep in rabbit hole.

Target Audience

The plugin is mainly targeted system/Integration tests, which is heavily bounded by I/O, network or other external dependency.

This plugin would more or less Break Test Isolation Principle. So make sure your tests is ok to run concurrently before you use this plugin.

Comparison

As mentioned above, unlike pytest-asyncio, which runs async tests sequentially, pytest-asyncio-concurrent takes advantage of Python's asyncio capabilities to execute tests concurrently by specifying async group.

Try this out!

Welcome to try this out, and let me know your feedback!

Github link: https://github.com/czl9707/pytest-asyncio-concurrent
PyPI: pip install pytest-asyncio-concurrent

[u/RonnyPfannschmidt]

The plugin heavily monkeypatches pytest privates that may change

It seems highly fragile and as far as I'm aware doesn't coordinate with pytest core

[u/andrewthetechie]

Cool idea but eyeballing the code, this is super fragile. You're monkeypatching private methods in pytest, which are not guaranteed as part of the api contract from pytest itself. Potentially, any minor version bump of pytest could break this entire thing.

A xdist controller https://pytest-xdist.readthedocs.io/en/stable/how-it-works.html might be easier to maintain long-term.

[u/Royal-Fail3273 → u/andrewthetechie]

Thanks for the suggestion.
Have to admit, making the testing framework work asynchronously is harder than I initially thought, and finally end up with two places of monkey patching it to workaround. Will definitely need some work to refract to get rid of these fragile pieces.

[u/andrewthetechie → u/Royal-Fail3273]

This is a hard problem! That's why there wasn't a ready made solution.

Keep at it. I can see the usefulness of what you're doing.

I would suggest it might be a more efficient use of time to fix up your CI server and use xdist. but hey I'm not your boss :D

[u/riksi]

Don't you need to add something like a thread/coroutine/contextvar_id somewhere like pytest-xdist has worker_id? So you can use 1 db for each thread_id and concurrently run tests even with external things.

Did you think about using the pytest-xdist framework and adding another "executor"?

[u/riksi → u/riksi]

Having this contextvar_id be available globally, you'll be able to make wrappers for ~most things.

[u/Royal-Fail3273 → u/riksi]

I will take this into consideration, thanks :)

[u/Royal-Fail3273 → u/riksi]

The reason of why not pytest-xdist is kinda funny. Pytest has a CPU usage spike during beginning stages (which is ok in most cases), when using xdist with 16 processes on our fragile server, the huge spike just blew up the whole machine.

[u/riksi → u/Royal-Fail3273]

The cpu spike is probably just starting the processes. I meant to create another executor for pytest-xdist and it will spawn asyncio workers and send tests there instead of spawning processes.

[u/JustPlainRude → u/Royal-Fail3273]

You can specify the number of workers with -n.