Option WithMixedBatch instructs sweepbatcher to create mixed batches with regard
to cooperativeness. Such a batch can include both sweeps signed both
cooperatively and non-cooperatively. If cooperative signing fails for a sweep,
transaction is updated to sign that sweep non-cooperatively and another round of
cooperative signing runs on the remaining sweeps. The remaining sweeps are
signed in non-cooperative (more expensive) way. If the whole procedure fails for
whatever reason, the batch is signed non-cooperatively (the fallback).
Previously sweepbatcher used loop/labels.LoopOutBatchSweepSuccess to assign
transactions labels. The value is "BatchOutSweepSuccess -- $batch_id", which
does not fit use cases outside of loop-out. Now there is option WithTxLabeler
which sets a function used to generate the label.
Tested scenarios:
1. For a regular sweep newly added it waits for initialDelay + publishDelay
before publishing a transaction.
2. For a sweep recovered from DB it does not wait before publishing tx.
3. If a sweep is about to expire, initialDelay is skipped.
WithPublishDelay sets the delay of batch publishing that is applied in the
beginning, after the appearance of a new block in the network or after the
end of initial delay (see WithInitialDelay). It is needed to prevent
unnecessary transaction publishments when a spend is detected on that block.
Default value depends on the network: 5 seconds in mainnet, 0.5s in testnet.
For batches recovered from DB this value is always 0s.
WithInitialDelay instructs sweepbatcher to wait for the duration provided
after new batch creation before it is first published. This facilitates
better grouping. It only affects newly created batches, not batches loaded from
DB, so publishing does happen in case of a daemon restart (especially important
in case of a crashloop). Defaults to 0s. If a sweep is about to expire (time
until timeout is less that 2x initialDelay), then waiting is skipped.
Renamed defaultPublishDelay to defaultTestnetPublishDelay. Its godoc was already
for defaultTestnetPublishDelay, but the const was named defaultPublishDelay.
According to AddTapscriptInput's godoc, the first argument (leafWitnessSize)
must include not the whole size of witness, but only the size of data consumed
by the revealed script. Previously the value was too high, because it also
included the following extra elements: number_of_witness_elements (1 byte),
witness script and its size, control block and its size.
Tests for greedy_batch_selection were affected by this change.
Use customFeeRate if it is provided, otherwise use wallet's EstimateFeeRate.
Added flag SkipNextBump to rbfCache to avoid extra bumping upon updating
fee rate externally.
Fix test TestSweepBatcherCloseDuringAdding, it didn't have confTarget and
failed in wallet.EstimateFeeRate call.
Use SigHashDefault instead of SigHashAll in weight estimations.
Actual code uses SigHashDefault, not SigHashAll. SigHashAll is 1wu larger.
Use the actual destination address in weight estimator, not taproot always.
In the test the type of address is P2WPKH, not taproot.
Updated testSweepFetcher to calculate fee, fee rate and weight accurately and
to compare the data observed in the published transaction with the estimates.
If custom fee rates are used, the algorithm is tried. It selects a batch for the
sweep using the greedy algorithm, which minimizes costs, and adds the sweep to
the batch. If it fails, old algorithm is used, trying to add to any batch, or
starting a new batch.
The reason is because we want to know in advance if fee rates come from
an external source or are determined by sweepbatcher internally.
Also remove WithNoBumping option. It is now a part of WithCustomFeeRate:
if WithCustomFeeRate is passed, automatic fee bumping by sweepbatcher
is disabled.
Previously, when handling a sweep we assumed that if a sweep status
was completed, the parent batch was also finished. However, since the
batch confirmation status depends on three on-chain confirmations, it
is possible that a spend notifier was started for a sweep of an active
batch. The notifier would fetch the parent batch from the database, but
because we incorrectly assumed that the parent was confirmed (when it
was not), the DB call would fail with a 'no rows returned' error.
This failure would cause the sweep to fail and the sweep batcher to
stop, resulting in a permanent failure state.
MinFeeRate is minimum fee rate that must be used by a batch of
the sweep. If it is specified, confTarget is ignored.
This is useful for external source of fees.
This is needed to cover the code of SQLStore with tests.
To achieve compatibility with loopdb (SQLite), the following changes were done:
- DestAddr is filled to avoid crash in SQL layer
- Preimage is filled to avoid uniqueness checks by the DB
- the code working with batch IDs was changed to work correctly
when batch_id starts with 1 instead of 0
- SQL swap store has to have a swap with swap_hash from sweep
to satisfy foreign key constraint of the DB
Provide a wrapped store type, exposing ExecTx method with a subset
interface in the callback argument. BaseDB interfaces in instantout,
reservation and sweepbatcher use ExecTx with their subset Querier
instead of whole sqlc.Querier (*sqlc.Queries).
This is needed to make the packages more reusable, so they don't
depend on methods of *sqlc.Queries they don't use.
A Querier holds the methods of sqlc.Querier interface relevant for a package.
BaseDB has Querier + ExecTx method.
This change is needed to simplify further rework of ExecTx.
The race was detected in CI and locally when running with -race.
It happened between the following calls:
WARNING: DATA RACE
Write at 0x00c0003e6638 by goroutine 1374:
runtime.racewrite()
<autogenerated>:1 +0x1e
github.com/lightninglabs/loop/sweepbatcher.(*batch).Wait()
sweepbatcher/sweep_batch.go:463 +0x6e
github.com/lightninglabs/loop/sweepbatcher.(*Batcher).Run.func1()
sweepbatcher/sweep_batcher.go:272 +0x10e
Previous read at 0x00c0003e6638 by goroutine 1388:
runtime.raceread()
<autogenerated>:1 +0x1e
github.com/lightninglabs/loop/sweepbatcher.(*batch).monitorConfirmations()
sweepbatcher/sweep_batch.go:1144 +0x285
github.com/lightninglabs/loop/sweepbatcher.(*batch).handleSpend()
sweepbatcher/sweep_batch.go:1309 +0x10e4
github.com/lightninglabs/loop/sweepbatcher.(*batch).Run()
sweepbatcher/sweep_batch.go:526 +0xb04
github.com/lightninglabs/loop/sweepbatcher.(*Batcher).spinUpBatch.func1()
sweepbatcher/sweep_batcher.go:455 +0xbd
The race was caused because wg.Add(1) and wg.Wait() were running from different
goroutines (one goroutine was running batch.Run() and another - batcher.Run()).
To avoid this scenario, wg.Wait() call was moved into batch.Run() call, so it
waits itself for its children goroutines, after which the channel b.finished
is closed, and it serves a signal for external waiters (the batcher, calling
batch.Wait()).
Also the channel batch.stopped was renamed to batch.stopping to better reflect
its nature.
Added TestSweepBatcherCloseDuringAdding to make sure adding a sweep during
shutting down does not cause a crash. The test did not catch the original
race condition.
Changed argument of function NewBatcher from LoopOutFetcher to SweepFetcher
(returning new public type SweepInfo).
This change is backwards-incompatible on the package layer, but nobody seems
to use the package outside of Loop.
To use NewBatcher inside Loop, turn loopdb into SweepFetcher using
function NewSweepFetcherFromSwapStore.
This data is not used by Batcher since commit
"sweepbatcher: load swap from loopdb, not own store".
Now sweepbatcher.Store depends only on tables sweeps and sweep_batches
and does not depend on swaps, loopout_swaps and htlc_keys, making it
easier to reuse.
Method Store.GetBatchSweeps provides data from tables outside of sweepbatcher:
swaps, loopout_swaps, htlc_keys. It makes it harder to reuse. Batcher already
has a straightforward way to get swap data: LoopOutFetcher interface (loopdb).
In this commit I switch the source of data from the field returned by Store
(LoopOut) to loading independently by calling LoopOutFetcher.FetchLoopOutSwap.
It used to be set to default (defaultBatchConfTarget = 12) which
could in theory affect fee rate if updateRbfRate() and publish()
were not called before the batch was saved. (Unlikely scenario.)
Method sweepbatcher.Store.FetchBatchSweeps (implementation using real DB) runs
JOIN query to load LoopOut from swaps table. Now the mock does the same.
It is needed to test store and load scenarios in tests.
If the sweep was successfully updated in the batch, no need to
try to add it to all other batches.
Added a test reproducing adding a sweep to both batches without this change.
Boris Nagaev
yingshanghuangqiao
crystalstall
Andras Banki-Horvath
Slyghtning