Enhance onchain transaction management#628
Enhance onchain transaction management#628Camillarhi wants to merge 2 commits intolightningdevkit:mainfrom
Conversation
|
👋 Thanks for assigning @tnull as a reviewer! |
39922cc to
c228760
Compare
|
🔔 1st Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 2nd Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 3rd Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 4th Reminder Hey @tnull! This PR has been waiting for your review. |
tnull
left a comment
There was a problem hiding this comment.
Just took an initial look and added a few high-level comments. I have yet to review any of the actual RBF logic changes.
However, as noted elsewhere, please don't make all the changes as single huge commit, but rather break the PR up in logical commits that all have descriptive commit messages outlining what the change is, why it's necessary, etc. For guidance you can have a look at https://cbea.ms/git-commit/
It would also make sense to not include the changes for #452 in this initial PR directly, but do it in a separate PR to keep the diff more manageable and reviewable.
a52b1e5 to
3523954
Compare
3523954 to
609d0a0
Compare
|
Thanks for the PR. |
src/wallet/mod.rs
Outdated
There was a problem hiding this comment.
I'm not fully sure about this flow: since with RBF we can’t guarantee which transaction will eventually confirm (usually the last one, but not always), removing the previous payment here might cause issues. What happens if the earlier tx ends up confirming instead of the latest — would the wallet balance be deducted while the store still shows the payment as pending? If so, could this lead to a confusing UX for the user?
There was a problem hiding this comment.
Excellent catch. You're right to be concerned about the potential for state inconsistency.
However, the update_payment_store method acts as a reconciler that syncs the payment store with the actual state of the BDK wallet. Since the payment store is designed to be a reflection of the wallet's state.
There was a problem hiding this comment.
We might potentially still end up with multiple entries for the same payment though, right, as we'd not drop the RBF entry once the original transaction confirms? There would also be no 'history' of all the bumps that happened.
There was a problem hiding this comment.
Yes, having multiple entries is a possibility. Regarding dropping the entry once the original confirms, that's what I plan on doing in this issue #452 .
As for the history of bumps, a solution could be introducing another status replaced with a replacement ID that indicates the transaction was bumped and replaced.
There was a problem hiding this comment.
I think one way to simplify tracking RBF transactions is to add two fields to each payment: is_rbf (a boolean indicating whether the transaction is part of an RBF sequence) and origin_payment_id (optional, pointing to the original payment in the sequence). The original transaction would have is_rbf = true and origin_payment_id = None. Each RBF attempt would also have is_rbf = true and set origin_payment_id to the original transaction’s ID. If a new RBF is created from an existing RBF, it can reuse the same origin_payment_id, linking it back to the original.
With this setup, handling confirmations becomes straightforward:
- If an RBF transaction confirms, find the original transaction via
origin_payment_idand clean up all other RBFs linked to it. - When a transaction with is_rbf = true and origin_payment_id = None (i.e., an original transaction) confirms, find all RBF transactions with origin_payment_id equal to its ID and remove them.
This keeps data consistent and ensures that extra lookups are only needed when is_rbf is true, making RBF handling explicit and efficient.
For example, with A as the original transaction and B, C, D as RBFs:
graph TD
A[A<br/>is_rbf=true<br/>origin=None]
B[B<br/>is_rbf=true<br/>origin=A.payment_id]
C[C<br/>is_rbf=true<br/>origin=A.payment_id]
D[D<br/>is_rbf=true<br/>origin=A.payment_id]
A --> B
A --> C
A --> D
There was a problem hiding this comment.
I think there’s a design consideration for transactions that won’t confirm because an RBF replaced them. When an RBF transaction confirms, the original and any other RBFs in the sequence become invalid. In my opinion, a more user-friendly approach is to mark them as failed and reference the txid of the confirming transaction, which makes it clear why these transactions didn’t confirm while keeping the history intact.
There was a problem hiding this comment.
Thank you for this feedback. This aligns well with the introduction of a replacement id as mentioned above.
Regarding retaining the history of bumped transactions and marking them as failed when any confirms, this could lead to increasing clutter of failed transactions if RBF is used frequently.
My proposed approach during node sync and store updates is when a confirmed transaction is encountered, if it's an RBF transaction, we can confirm that transaction and clear/remove the history of bumps from the store. This prevents unnecessary accumulation of failed transaction records while maintaining proper transaction state tracking.
609d0a0 to
8b7cf78
Compare
Thanks for the review! I've split the PR into two separate commits as suggested |
|
🔔 1st Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 2nd Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 3rd Reminder Hey @tnull! This PR has been waiting for your review. |
tnull
left a comment
There was a problem hiding this comment.
Did another pass and added some comments. Still have to take an even closer look at the RBF logic and think through edge cases.
src/wallet/mod.rs
Outdated
There was a problem hiding this comment.
We might potentially still end up with multiple entries for the same payment though, right, as we'd not drop the RBF entry once the original transaction confirms? There would also be no 'history' of all the bumps that happened.
06e7ff3 to
6e6e2aa
Compare
|
🔔 1st Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 2nd Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 3rd Reminder Hey @tnull! This PR has been waiting for your review. |
There was a problem hiding this comment.
@Camillarhi Excuse the delay here - last few weeks have been super busy pushing towards an LDK 0.2 beta release. Will see to get back to review on this in a proper ASAP, it might take another week or so though.
In the meantime I want to note that recently there has been some movement towards #446 as BDK now added events (see bitcoindevkit/bdk_wallet#6 / bitcoindevkit/bdk_wallet#310) which will ship as part of the next BDK 2.2 release. So I do wonder if we should generally first move to update our payment store based on the events emitted from syncing (also allowing for #448 to land), and then basing the changes here on top. This might be in particular interesting as bitcoindevkit/bdk_wallet#310 included a TxReplaced event which might allow us to detect and track (all?) RBF transactions belonging to a 'payment' more easily.
Any thoughts on this?
tnull
left a comment
There was a problem hiding this comment.
Thanks for the rebase, here are some updated comments.
aa1558a to
8631a4f
Compare
tnull
left a comment
There was a problem hiding this comment.
Let me know when this is ready for the next round of review. Unfortunately seems to need yet-another rebase already. Sorry!
Sure, I will. |
8dd9382 to
764538a
Compare
|
🔔 1st Reminder Hey @tnull! This PR has been waiting for your review. |
|
🔔 2nd Reminder Hey @tnull! This PR has been waiting for your review. |
764538a to
00de553
Compare
|
🔔 3rd Reminder Hey @tnull! This PR has been waiting for your review. |
tnull
left a comment
There was a problem hiding this comment.
Already looks pretty good, some comments, and it will need a rebase now that the broadcaster interface changed.
src/wallet/mod.rs
Outdated
There was a problem hiding this comment.
nit: Changes like this are unrelated and in this case will be reverted by the weekly cargo +nightly fmt job anyways.
| ConfirmationStatus::Unconfirmed, | ||
| ); | ||
| // We fetch payment details here since the replacement has updated the stored state | ||
| let payment = |
There was a problem hiding this comment.
Hmm, but that would mean we expect this get to always return Some at this point, right? Can we add a debug_assert checking that it does?
There was a problem hiding this comment.
Hmm seems we hitting some of the new debug_asserts in test_rbf_via_direct_block_insertion:
Payment b25fa25838fc1654906457e86846d380996624ff8750c4910049be4350710f1c expected in store during WalletEvent::TxReplaced but not found
Might be good to understand what's happening here. Possibly some of the fixes in #769 could also be related, but not sure.
There was a problem hiding this comment.
Thanks, I will check out what's happening
src/payment/onchain.rs
Outdated
There was a problem hiding this comment.
Do we need an optional fee-rate override parameter here, similar to what we have in send_to_address?
There was a problem hiding this comment.
Yes, I will go ahead and include it
| Error::InvalidFeeRate | ||
| })?; | ||
|
|
||
| builder.fee_rate(required_fee_rate); |
There was a problem hiding this comment.
Hmm, should we apply an upper-bound to the require_fee_rate, i.e., ensure it's still in some margin of what we expected? Maybe we even want to apply some upper-bound (relative to the original rate?) to our estimate to ensure we don't spend all our funds on fees if the chain source's fee estimation endpoint is lying to us?
There was a problem hiding this comment.
Yeah, I thought about this too, that's why I computed a min_required_fee_rate_sat_per_kwu above:
let min_required_fee_rate_sat_per_kwu =
old_fee_rate_sat_per_kwu + INCREMENTAL_RELAY_FEE_SAT_PER_1000_WEIGHT as u64;
Then take the max of that and the estimated_fee_rate. Based on the tests so far, the final_fee_rate is what ends up being used in most cases, the difference between it and required_fee_rate has been around 2-3 sat/kwu. I think it'd be pretty rare for the chain source estimate to actually win out, but an upper-bound sanity check on it is reasonable to add
There was a problem hiding this comment.
Based on the tests so far, the final_fee_rate is what ends up being used in most cases, the difference between it and required_fee_rate has been around 2-3 sat/kwu. I think it'd be pretty rare for the chain source estimate to actually win out, but an upper-bound sanity check on it is reasonable to add
Did you do tests on mainnet as well? Note that testing such things on regtest or even testnet/signet will lead to bogus values.
There was a problem hiding this comment.
Yes, this is correct. I will introduce an upper-bound sanity check.
There was a problem hiding this comment.
@tnull, I introduced a sanity check at 1.5x our target fee rate to avoid overpaying
There was a problem hiding this comment.
Hmm, thanks. So now I still wonder if should ever spend more than the explicit override fee rate set by the user. Generally, maybe this auto-retry logic is already to complicated, and it might be easier to just bubble-up the FeeRateTooLow error and have the user use an explicit override fee rate of their choice to adjust the target upwards if they'd ever hit it?
There was a problem hiding this comment.
How about we only do the auto-retry when the user didn't provide an explicit fee rate? If they passed a specific rate and BDK rejects it, we bubble up the error and let them adjust. If they passed None, we handle it internally with the 1.5x sanity cap.
00de553 to
b5b191b
Compare
|
🔔 1st Reminder Hey @tnull! This PR has been waiting for your review. |
tnull
left a comment
There was a problem hiding this comment.
Seems CI is unhappy/hanging due to some of the debug_asserts getting hit now.
|
Might also need a rebase now after #769 landed. |
Add automatic rebroadcasting of unconfirmed transactions triggered by the `ChainTipChanged` event from BDK. This ensures pending transactions remain in mempools.
b0b40b1 to
dc53ac5
Compare
src/payment/onchain.rs
Outdated
There was a problem hiding this comment.
I think we can drop The original transaction must be signaling RBF replaceability for this to succeed. as this is given for all of our payments, no?
There was a problem hiding this comment.
Yes, we can drop that. All our payments signal RBF, so it's not worth documenting.
src/payment/onchain.rs
Outdated
src/wallet/mod.rs
Outdated
There was a problem hiding this comment.
I this debug_assert shouldn't be necessary, if anything it might be race-y if our KVStore view is somehow temporarily inconsistent.
src/wallet/mod.rs
Outdated
There was a problem hiding this comment.
Ah, it seems we shouldn't fallback here to PaymentId(txid.to_byte_array()) but rather log and skip processing if we can't find the respective payment entry?
Then, not sure if we should use find_payment_by_txid at all if we now assume that the payment entry will be in our store?
There was a problem hiding this comment.
Agreed, we should not fall back to PaymentId(txid.to_byte_array()) here. However, we should still use find_payment_by_txid because the txid in TxReplaced is the old replaced txid, find_payment_by_txid looks up the pending_payment_store to map it back to the correct payment id (either directly or via conflicting_txids). If no payment id is found, we should log and skip processing.
| self.pending_payment_store.insert_or_update(pending_payment)?; | ||
| }, | ||
| WalletEvent::TxReplaced { txid, conflicts, tx, .. } => { | ||
| WalletEvent::TxReplaced { txid, conflicts, .. } => { |
There was a problem hiding this comment.
I do wonder if this also handles the case of inbound replaced transactions correctly?
There was a problem hiding this comment.
Yes, the logic handles inbound transactions correctly as well. I will also expand the integration test to verify the inbound side.
src/wallet/mod.rs
Outdated
There was a problem hiding this comment.
I'm confused by this comment: what logic exactly is this referring to - where does the replacement update the store, i.e., why can we be sure it has been updated here? It seems our processing order would ensure we always process this logic first, not TxUnconfirmed, if this is what this comment is referring to?
There was a problem hiding this comment.
The comment refers to bump_fee_rbf , It updates the payment store with the new replacement txId before the next wallet sync. So by the time the sync runs and emits TxReplaced, the payment is already updated in the store. I'll reword the comment to make this clearer.
There was a problem hiding this comment.
Ah, but this really only holds if the replacement was initiated by us then. How do we handle replacements for inbound payments for example?
There was a problem hiding this comment.
For inbound replacements, since we already have the unconfirmed transaction in the store, it relies on using find_payment_by_txid to look up the PaymentID by matching with the txid or conflicts and updates the payment.
src/wallet/mod.rs
Outdated
There was a problem hiding this comment.
This probably should use a different error variant.
src/wallet/mod.rs
Outdated
There was a problem hiding this comment.
Please try to always log when returning an error, also in this instance.
src/wallet/mod.rs
Outdated
There was a problem hiding this comment.
I think we can drop the intermediate variable here, as old_fee_rate.to_sat_per_kwu() is basically same length.
| Error::InvalidFeeRate | ||
| })?; | ||
|
|
||
| builder.fee_rate(required_fee_rate); |
There was a problem hiding this comment.
Hmm, thanks. So now I still wonder if should ever spend more than the explicit override fee rate set by the user. Generally, maybe this auto-retry logic is already to complicated, and it might be easier to just bubble-up the FeeRateTooLow error and have the user use an explicit override fee rate of their choice to adjust the target upwards if they'd ever hit it?
| ); | ||
|
|
||
| let pending_payment_store = | ||
| self.create_pending_payment_from_tx(new_payment.clone(), Vec::new()); |
There was a problem hiding this comment.
Wouldn't we want to track the original Txid as a conflict here?
There was a problem hiding this comment.
bump_fee_rbf just creates the initial pending entry, and the empty vec gets treated as None during update, so it won't overwrite existing conflicts. The conflict tracking is handled in the TxReplaced event. I've also updated conflict_txids to include the replaced txid, as this is not returned as part of the conflicts from bdk_wallet, so no intermediate txid gets lost across consecutive bumps.
Add `Replace-by-Fee` functionality to allow users to increase fees on pending outbound transactions, improving confirmation likelihood during network congestion. - Uses BDK's `build_fee_bump` for transaction replacement - Validates transaction eligibility: must be outbound and unconfirmed - Maintains payment history consistency across wallet updates - Includes integration tests for various RBF scenarios
dc53ac5 to
d9d061c
Compare
This PR enhances on-chain transaction management:
Rebroadcast/bumping of on-chain wallet transactions
Handle RBF'd Pending payments
Changes
Related Issues