[test] Separate test runner configuration from main compiler config

[sbc100]

Followup to #27216

Move test-only configuration settings (NODE_JS_TEST, V8_ENGINE, SPIDERMONKEY_ENGINE, JS_ENGINES, WASMER, WASMTIME, WASM_ENGINES) out of the main compiler config file (.emscripten) and into a new test/config.py file.

We are making this change for two reasons:

1. To keep test suite configuration separate from core compiler code.
2. To prepare for converting the main compiler configuration file to a simpler non-Python format (e.g. emscripten.conf).

If any test-only settings are found in the main config file, the compiler now errors out requesting that they be moved to test/config.py.

Also, add some docs about how to configure that test suite, which were previously missing.

See: #27219

[sbc100]

This change does run the risk of making bisecting across this change (with local test-config settings in place) as little annoying.

However, most users likely don't use test config settings (the defaults are normally good enough) and commenting out these settings when running a bisect would likely be fine in most cases too.

[kripken]

I'm not sure I follow the motivation. Right now, I have one config file to edit. What do I gain by having two?

Also, each config file has some logic for managing it, like we autogenerate .emscripten - more files means more complexity?

[sbc100]

As as laid out in the PR description the motivation is really twofold:

1. I want to simplify the compiler config so it no longer depend on python (Convert config file format from python to something simpler. #27219). The allows us to move in the direction of a native clang runner (Ship native binary launchers to reduce Python startup overhead #26453), only using python for linking.
2. Separation of test and runtime code keeps us honest. We don't want the complexity of test configuration in the normal compiler, and we want the testing to be as blackbox as possible. The less blurred these lines are the better IMHO. This work has been onging for a long time (See Separate test config from core compiler. NFC #27216 for example) and this is just the next step.

A couple of notes on the impact of this:

1. Most users should never need to touch either config files. Obviously emsdk user never touch the config directly. But even non-emsdk users should not need to write a config file these days in we look for all tools in the PATH by default.
2. Most emscripten developers don't need to modify the test config since testing with node JS alone is normally sufficient.

There is not reason that the compiler and test framework should share config file or a config file format. Imagine if I proposed to the llvm project that clang and lit should share config file format? Or, in binaryen, if I proposed that check.py and wasm-opt should share a config file?

[sbc100]

I do admint there are couple of wrinkles here that I still need to figure out. For example there is one part of the config that is shared and that is the default node JS version. I'm trying to figure out way to deal with that now.

[kripken]

There is not reason that the compiler and test framework should share config file or a config file format. Imagine if I proposed to the llvm project that clang and lit should share config file format? Or, in binaryen, if I proposed that check.py and wasm-opt should share a config file?

Counterpoint, some projects have a single config file for many components. Firefox has one config file for all parts of the browser, not one for the JS engine and one for WebGL and one for networking etc. A single config file means only one file needs to be managed, which is nice.

So the question is whether the testing stuff is really entirely separate from the main project. I don't feel strongly about that actually. But I think the other benefits you mention are independent of this? For example, changing the config file format sounds like a good idea regardless of how many config files we have.

Do we even need a test config file, btw? If it is just for the node version, perhaps we can just use the path/an env var?

[sbc100]

There is not reason that the compiler and test framework should share config file or a config file format. Imagine if I proposed to the llvm project that clang and lit should share config file format? Or, in binaryen, if I proposed that check.py and wasm-opt should share a config file?

Counterpoint, some projects have a single config file for many components. Firefox has one config file for all parts of the browser, not one for the JS engine and one for WebGL and one for networking etc. A single config file means only one file needs to be managed, which is nice.

But does the firefox test framework also get configured via this one config file? I honestly don't know.

So the question is whether the testing stuff is really entirely separate from the main project. I don't feel strongly about that actually. But I think the other benefits you mention are independent of this? For example, changing the config file format sounds like a good idea regardless of how many config files we have.

Indeed. Changing the config file format to make simpler is the primary goal.

Do we even need a test config file, btw? If it is just for the node version, perhaps we can just use the path/an env var?

Indeed, I did consider that, but there are some cases where there is some actual complexity needed for the test config, at least for my test config. Sometimes I want to setup mulitple JS_ENGINES in an array, for example.

We could argue that having mulitple JS_ENGINES in the same test run is almost never used..

The nice thing about this PR is that it basically means don't need a test config file... its completely optional and most people probably won't need/use one.

[kripken]

But does the firefox test framework also get configured via this one config file? I honestly don't know.

I don't either. A unit test framework like gtest would use the main config file I'd guess, but maybe not an external one.

The nice thing about this PR is that it basically means don't need a test config file... its completely optional and most people probably won't need/use one.

But for those that do, they'd need to figure out where that file is and how it is managed etc. - that's the complexity I'm not happy about.

[sbc100]

But does the firefox test framework also get configured via this one config file? I honestly don't know.

I don't either. A unit test framework like gtest would use the main config file I'd guess, but maybe not an external one.

I asked AI about how the firefox tests are run:

"""
When developing Firefox itself, the test runner ecosystem is managed via the ./mach command-line tool, with specific test behavior configured through manifest files (.ini), environment variables, and runtime JavaScript APIs.
Firefox core testing relies on several test suites (flavors), most notably Mochitest (for functional/integration tests) and Reftest (for layout rendering tests).
"""

Specifically, they do not use the firefox config file it seems. i.e. The firefox config file is not interpreted by the test runner.