Integration tests

[PerchunPak]

We should add some kind of integration tests on release, to prevent such issues as #433. On another hand, it would be hard work, and possibly will not find anything even in long term.

We also don't have unit tests for some parts of our code (like Bedrock part and CLI as well, partially tracked by #371).

[kevinkjt2000]

Running a few containerized Minecraft servers would be a way to tackle this. I recommend itzg's Minecraft docker images; I've been using those for a while to run servers and they work great. There's an image that runs bedrock too.

[PerchunPak]

A list of server implementations to test.

Java ones (itzg's docker):

• Vanilla
• Bukkit
• Spigot
• Paper
• PurPur (7.5% "Bukkit and forks" users use it, more than Bukkit has)
• Forge
• Fabric
• Sponge
• Mohist
• Magma

Bedrock (itzg's docker):

• Vanilla
• Geyser
• Nukkit
• PMMP
• PowerNukkit

Proxies:

• BungeeCord
• Waterfall
• Flamecord
• Velocity
• Aegis (maybe, it's not free. But possibly they provide some kind of public server to test - yep, they provide hubplay.pl)

I tried to include all popular server cores. Maybe it's a lot, and we should cut some of them.

[ItsDrike]

Could be worth including a forge server, or other modded server too

[PerchunPak]

They're under PurPur.

[PerchunPak]

Duplicating my messages from Discord for history.

---

I have just found that DigitalOcean provides free hosting for some open-source projects. We could try to apply and host there our integration tests
https://www.digitalocean.com/open-source/credits-for-projects

Although, they provide only $60/year (which is $5/month) and we probably could only buy 512mb RAM VPS

Another idea is to use swap, but it would drastically decrease speed (although if we would test so many versions and cores as we currently want, speed wouldn't really matter)

As it's integration tests are blocking for releases, we could leverage it by automatically creating patch notes (and so GitHub release) or by manually triggering integration tests before manually creating GitHub release

Any thoughts? If not, just add 👍 and I will email DigitalOcean.

[ItsDrike]

Do we even need something like that? I think I would honestly be happy enough with just a local shell/python script based tool that spins up the mc servers as docker containers and runs some tests against them, even if we just run it manually for now before releases, they don't happen that often and it would still be a pretty major benefit to have that. (Assuming it doesn't take like an hour to run until we test every server.)

I also think we might be able to just integrate it into the GitHub CI. There should be support for running side services from docker images (ref). We could run the mc server in this way.

[PerchunPak]

Huh, they must have bumped the specs for GitHub Actions runners
https://docs.github.com/en/actions/reference/runners/github-hosted-runners#standard-github-hosted-runners-for-public-repositories

It has 16 GB of RAM now, so we don't have to buy anything