improve wasm support

[FreyLuis]

I found a few issues while setting up WASM benchmarks & would like to fix those. Since those would include breaking changes, I’m asking about them here before starting any work.

• Currently --expose_wasm is the default parameter for javaScriptEngineArguments, but the flag was removed in 2024.
• The JS entry point is hardcoded to be AppBundle/test-main.js (or AppBundle/main.js with .net 5 / 6). The user then has to create the file under the correct path & still needs to set the WasmMainJSPath as an msbuild argument. The path shouldn’t be hardcoded & BenchmarkDotNet could autogenerate the file.
• The default parameter for the .net monikers are still .net 5. IMHO, those should either be removed or replaced with factory methods (CreateWasm10_0(string javaScriptEngine, string javaScriptEngineArguments, bool aot)).

The docs are also outdated (already tracked by #1818):

• The current example only describes how to set benchmarks up using a custom dotnet runtime. This is unnecessary and looks scary to anyone not working on dotnet itself.
• The example also recommends using v8 directly via jsvu, which does not provide standard JavaScript APIs. Dotnet relies on these APIs (for creating GUID’s, it uses crypto.getRandomValues() for example). IMHO, recommending node might be the better choice.

Would you accept such a PR, or is someone already working on something similar?

[timcassell]

• The default parameter for the .net monikers are still .net 5. IMHO, those should either be removed or replaced with factory methods (CreateWasm10_0(string javaScriptEngine, string javaScriptEngineArguments, bool aot)).

I have mentioned in the past that the default moniker should probably use the current runtime version, but no one seemed interested in doing it.

We are definitely open to accepting PRs that update the wasm toolchain. Just make sure you validate any changes against dotnet/performance.

[timcassell]

• Currently --expose_wasm is the default parameter for javaScriptEngineArguments, but the flag was removed in 2024.

That probably explains my issue in #2675. Maybe we can enable Windows support with your updates.

[FreyLuis]

@timcassell Sure, I was already wondering what blocks windows support. I believe that the run-v8.sh script generated with the WasmGenerateRunV8Script MsBuild Property also contains the flag, but BenchmarkDotNet doesn't use the script, so this isn't an issue here.

Would it be fine to completely remove the wasmDataDir command line argument? I believe it's not actually an option in the WebAssembly SDK and just used here in dotnet/performance to set the main.js path. Instead, we could add a wasmMainJsPath option.

[timcassell]

It sounds fine to me, but I'm no wasm expert.

[timcassell]

• The default parameter for the .net monikers are still .net 5. IMHO, those should either be removed or replaced with factory methods (CreateWasm10_0(string javaScriptEngine, string javaScriptEngineArguments, bool aot)).

After a bit more thought, I think they should just be removed. All the other monikers (except old mono) are version-specific.

[timcassell]

At least points 2 and 3 are resolved by #2998 and #3010. Still open for the other things.

[FreyLuis]

@timcassell Sorry, I have not been able to find the time to test this against dotnet/performance since December. I have a branch that adds windows support, allows any wasmruntime from the path, adds a custom argument formatter option, etc.

I’ll update this to the current master and test it against dotnet/performance this week.

[timcassell]

recommending node might be the better choice

I also think we should make Node the default. I'm trying to add WebSocket IPC in #2958 to match the other toolchains, and it's apparently impossible in v8. I got it working with Node. @LoopedBard3 @ilonatommy @maraf @pavelsavara would switching to Node work for dotnet/performance?

[pavelsavara]

I also think we should make Node the default.

The real supported target of our product is browsers, not shells. So Node vs V8 makes only small difference from that angle.

I'm trying to add WebSocket IPC in #2958 to match the other toolchains, and it's apparently impossible in v8. I got it working with Node.

That is a very large PR ... what's the motivation for WS ?

Indeed, V8 doesn't have WS support and many other things.
Note that WS and HTTP behavior of NodeJS doesn't match the browser behavior 100% and we had many troubles when testing edge cases in the past. We don't test BCL with NodeJS in CI at the moment.

My final objection with NodeJS is that it is much more complex VM than V8 and from perf perspective, there is more opportunity for noise in the measurements. I mean, when you run dotnet on WASM, you are running 2 layers of VM. NodeJS has possibly different perf profile from both browser and pure V8.

All that said, I can understand NodeJS appeal and can live with the change. Probably there would be transition when the measured numbers do not match.

[timcassell]

That is a very large PR ... what's the motivation for WS ?

My intent is to add cooperative cancellation. The way we abort benchmarks today is harsh, like thread aborts of old (#2159).

It also allows diagnosers to hook in which are totally ignored today.

The real supported target of our product is browsers, not shells.
We don't test BCL with NodeJS in CI at the moment.

Do you test it with browsers, or just with v8? We could also possibly do headless browser to match real world performance more closely, though it's a question of which browser should be the default.

[pavelsavara]

Do you test it with browsers, or just with v8?

We test with browsers and only few smoke tests are V8 or Node.

We could also possibly do headless browser to match real world performance more closely

(http servers, selenium, process exit codes, chrome bugs, lifting log files from the browser, hangs, ...)

Unless you want to deal with reliability troubles on 10x rate, I would advise against it.
The perf pipeline with V8 is often broken months at a time ...

though it's a question of which browser should be the default.

Only real desktop market share is Chromium browsers. For testing on Safari , you would need iOS farm. We don't have that.

[pavelsavara]

My intent is to add cooperative cancellation

Web sockets on browser emit incoming event only when you yield to browser event loop.
We only have single thread. So, unless you actually await Task.Yield() from the every loop that you want to be canceled, you will not receive that event. And yielding is very disruptive from perf perspective.

[timcassell]

(http servers, selenium, process exit codes, chrome bugs, lifting log files from the browser, hangs, ...)

Unless you want to deal with reliability troubles on 10x rate, I would advise against it. The perf pipeline with V8 is often broken months at a time ...

I guess node is the best option, then.

Web sockets on browser emit incoming event only when you yield to browser event loop. We only have single thread. So, unless you actually await Task.Yield() from the every loop that you want to be canceled, you will not receive that event. And yielding is very disruptive from perf perspective.

Right. So we would yield in the engine after each iteration (not each invocation, so it won't affect the measurement). And if the user's benchmark is truly async, it will necessarily yield anyway. Obviously long running synchronous benchmarks wouldn't be able to receive the cancellation signal immediately, but those type of benchmarks are niche. In a single threaded environment, it's just best effort.