We hit a problem where one mutant made a test allocate memory without stopping. It went from 2 GB to over 60 GB in about ten seconds. That was enough to run our whole CI machine out of memory. The Linux OOM killer then killed the GitHub Actions runner agent, so the job failed with "The runner has received a shutdown signal" and nothing pointed at Stryker or at the test. It took us a long time to find.
Stryker already protects against a mutant that runs too long, viatimeoutMS and timeoutFactor. There is no equivalent for a mutant that allocates too much. In our case the two were racing: the timeout was about 10 seconds, and filling the machine took 8 to 12 seconds. Whichever won decided if the run survived. That is why it looked flaky.
Would you consider a memory limit for test runner children? Something like testRunnerMemoryLimitMB. If a worker passes it, kill the worker and mark the mutant the same way a timeout is marked.
Things we tried that did not work:
--max-old-space-size via testRunnerNodeArgs. The memory was not in the V8 old space, so the cap had no effect. We tested 512 MB and the process still reached 60 GB.- Lowering
concurrency. The runaway just used the memory that freeing up workers gave it. With concurrency: 1 it still died.
We hit a problem where one mutant made a test allocate memory without stopping. It went from 2 GB to over 60 GB in about ten seconds. That was enough to run our whole CI machine out of memory. The Linux OOM killer then killed the GitHub Actions runner agent, so the job failed with "The runner has received a shutdown signal" and nothing pointed at Stryker or at the test. It took us a long time to find.
Stryker already protects against a mutant that runs too long, via
timeoutMSandtimeoutFactor. There is no equivalent for a mutant that allocates too much. In our case the two were racing: the timeout was about 10 seconds, and filling the machine took 8 to 12 seconds. Whichever won decided if the run survived. That is why it looked flaky.Would you consider a memory limit for test runner children? Something like
testRunnerMemoryLimitMB. If a worker passes it, kill the worker and mark the mutant the same way a timeout is marked.Things we tried that did not work:
--max-old-space-sizeviatestRunnerNodeArgs. The memory was not in the V8 old space, so the cap had no effect. We tested 512 MB and the process still reached 60 GB.concurrency. The runaway just used the memory that freeing up workers gave it. Withconcurrency: 1it still died.