bug: WSS handler socket leak

[mendelskiv93]

Summary:
WSS handler fails to reject malformed connections gracefully, causing TCP socket exhaustion and service degradation.

Incident Details:

• When: 2025-10-24 02:04 UTC
• Duration: ~5 hours (self-resolved)
• Impact: All WSS endpoints unresponsive, TLS handshakes timing out
• Scope: Multiple production nodes simultaneously affected

Evidence:

• TCP socket count: 200 → 1,600+ (8x increase)
• Log volume: 11K → 127K lines/hour (11x increase)
• Message throughput: Unchanged (~25K/hour)
• WSS errors: 1,733 AsyncStream Error: "Incomplete data sent or received" in one hour
• P2P functionality: Unaffected

Root Cause:
Malformed/incomplete WSS connection attempts trigger errors in wstransport.nim (lines 294, 296):

Http Error: "Timed out parsing request"
AsyncStream Error: "Incomplete data sent or received"

Connections are accepted but never properly closed, causing socket leak and Recv-Q buildup.

Expected Behavior:
WSS handler should reject invalid connections quickly and close sockets properly, preventing resource exhaustion.

Version:

• Image: harbor.status.im/wakuorg/nwaku:deploy-status-prod (2 months old)

[weboko]

I can confirm that yesterday we observed this issue on waku.sandbox.

Can we prioritise this bug, @Ivansete-status ?

[Ivansete-status]

I'll check, this is happening in v0.36.0

[Ivansete-status]

EDITED: disregard this comment. I think it doesn't replicates the original Incomplete data sent or received issue.

It can be replicated by running this script for a few minutes. After that, ctrl+C the script but then, it still appear many ESTABLISHED and WAIT1 connections:

import WebSocket from 'ws';

const TARGET = 'wss://node-01.do-ams3.waku.test.status.im:8000';
const COUNT  = 1000;
const clients = [];

for (let i = 0; i < COUNT; i++) {
  const ws = new WebSocket(TARGET, { perMessageDeflate: false });
  
  ws.on('open', () => {
    console.log(`Socket ${i} opened`);
    // optional: send an initial message
    ws.send(`Hello jamon ${i}`);
    // periodic ping
    setInterval(() => ws.ping(), 30000);
  });

  ws.on('close', () => console.log(`Socket ${i} closed`));
  ws.on('error', (err) => console.log(`Socket ${i} error: ${err.message}`));

  clients.push(ws);
}

[Ivansete-status]

@mendelskiv93 - is it easy to replicate the issue from your end?

[mendelskiv93]

@mendelskiv93 - is it easy to replicate the issue from your end?

Yeah, we reproduced it. Sent ~500 malformed WebSocket requests and saw Recv-Q jump to 100+ with the same AsyncStream errors from production. I guess it's not a leak, connections do clean up eventually, but the handler is too slow at rejecting bad requests, so during sustained attacks they pile up and legitimate clients can't connect. Our production incident hit 1,600+ sockets over 4 hours before recovering.

[Ivansete-status]

Thanks for the detailed description @mendelskiv93 !
Is there an "easy" way to enforce incorrect WebSocket requests. That will help a lot to iterate over the issue :)

[mendelskiv93]

Here is what I used to reproduce it:

for i in {1..500}; do
  (echo -ne "GET / HTTP/1.1\r\nUpgrade: websocket\r\n" | timeout 3 openssl s_client -connect localhost:443 -quiet 2>/dev/null) &
done

Monitor with docker exec nim-waku-boot netstat -tlnp | grep 443. The second column should show Recv-Q building up.

[Ivansete-status]

@richard-ramos - the following may help with that issue: status-im/nim-websock#178