smp-server: fix several memory leaks#1826
Open
shumvgolove wants to merge 4 commits into
Open
Conversation
shumvgolove
changed the title
Add an exception-guarded periodic thread that logs a single greppable "LEAKDIAG" line censusing every growable in-memory structure: live threads, per-client endThreads and subscriptions (by SubThread state), subscriber maps, ntf store, store entity/loaded counts, and proxy agent maps with in-flight sentCommands. Interval via SMP_LEAKDIAG_SEC (default 60), no RTS flags required. Adds pClientSentCommandsCount and getAgentLeakStats accessors.
Standalone smp-mem-bench executable that starts an in-process SMP server and drives churn workloads, reporting GHC live-heap residency per checkpoint after a forced major GC. Store selectable via BENCHSTORE (pgmsg/pgjournal/journal). Phases: plain, svc, svcrace, ntf, conc, svcsubs, getp, link, and leak repros - stuck (delivery threads blocked forever on a full sndQ), certchurn (serviceLocks/services grow per distinct service certificate), and ntfexp (NtfStore keys retained after notifications expire).
Service subscription counters (totalServiceSubs, serviceSubsCount, ntfServiceSubsCount) are TVar (Int64, IdsHash). modifyTVar' only forces the pair to WHNF, so `n +/- n'` and `idsHash <> idsHash'` stay unevaluated and accumulate an unbounded thunk chain under subscription and delivery churn (the IdsHash chain also retains a bytestring per update) - a space leak proportional to the number of updates. Force both components in addServiceSubs/subtractServiceSubs. Verified with the load bench: svc churn drops from +5.4 KiB/iter (linear) to flat.
deleteExpiredNtfs trimmed each notifier's message list but never removed the outer NtfStore map key, so one empty entry per notifier queue that ever received a notification was retained forever (grows with the active notifier set, never shrinks). Remove the outer key when its list becomes empty, and make storeNtf fully atomic so it cannot race the removal and write a notification to an orphaned TVar. Verified with the load bench (ntfexp): after expiry ntfStore_keys drops from the queue count to 0 instead of staying flat.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Leak 1: service subscription counters
A regression since v7.0.0-beta.1.
Messaging services changed the counters
totalServiceSubs,serviceSubsCount,ntfServiceSubsCountfromTVar Int64toTVar (Int64, IdsHash):modifyTVar'only forces the tuple to WHNF, son + n'andh <> h'stay as thunks. Every subscribe/deliver/unsubscribe adds a layer to a
thunk chain in the TVar (and the
IdsHash <>chain retains a bytestringper update) - a space leak proportional to the number of updates. With the
old
TVar Int64,modifyTVar' (+1)fully evaluated the counter.Fix: force both components in
addServiceSubs/subtractServiceSubs.Leak 2: notification store keys
deleteExpiredNtfsemptied each notifier's message list but never removedthe outer
NtfStorekey, so one empty entry per notifier queue that everreceived a notification was retained forever.
Fix: remove the key when its list becomes empty, and make
storeNtfatomic so it cannot race that removal and lose a notification.