This week saw incremental improvements and bugfixes. Here are some highlights.
Automatic wallet db backup when migrating schema
The wallet database migration process now saves a backup prior to migrating the database schema. This is a precaution in case anything goes wrong with the upgrade, or if the user decides to downgrade the wallet software. The backups are stored into <data-dir>/<network>/databases/migration_backups/. The backup filename includes the schema version of the backed up database and a counter to avoid overwriting an existing backup. So an example filename would be: wallet-schema-v0.bak.1.
MSMP Resync
Two users reported the observations described in Issue #568: the node crashes immediately after start. The context is that the nodes in question own UTXOs and spent some time offline.
After investigation a logic error was found in resync_membership_proofs_from_stored_blocks, which is responsible for syncing UTXOs’ membership proofs given access to historical blocks. The function was being tested, but with inadequately underpopulated blocks. PR #580 fixes both the function and the test.
In more detail: UTXOs owned by a wallet come with a membership proof which establishes membership in a cryptographically authenticated database called the mutator set. Wallets need to keep these mutator set membership proofs (MSMPs) up to date in order to spend them at a whim. So with every block, some update needs to be applied. And if there is a reorganization or a long string of blocks to catch up to, the function resync_membership_proofs_from_stored_blocks is called to perform those updates.
The update in question comes from inputs and outputs being confirmed by the block; these correspond to addition records and removal records, respectively, which are applied to the mutator set. The membership proofs must stay in sync with the mutator set, so they must undergo a parallel update. The error came from failing to also update the removal records, which must also be kept in sync with the mutator set, even in the course of applying a batch of them.
RemovalRecordList
The RemovalRecordList represents a list of RemovalRecords but without duplicate Merkle nodes, leading to a 83% reduction in space. Furthermore, it seems possible to encode a RemovalRecordList as a Vec of RemovalRecords, so that consensus programs that expect the latter need no change.
Mempool redesign begins
@jfs has begun work on a mempool redesign. The mempool, short for memory pool, is the “staging area” of transactions prior to their inclusion in the blockchain. The current mempool design and implementation have some shortcomings that impact Neptune Cash users and developers. For example, and crucial for users of Neptune Cash, the mempool is frequently cleared, which can lead to the entire network forgetting about a transaction, which means it will never be included in the blockchain. The transaction has to be re-initiated manually, which is understandably frustrating. For developers, the most prominent shortcoming is the tight integration of the mempool with the rest of the node software. This means, among other things, that third-party wallets with neptune-core integration are trickier to build than they would have to be. For these and additional reasons, we are redesigning the mempool. For more details, have a look at our issue tracker.
Contributor updates:
RemovalRecordList- PR comments
- participate in mempool / Upgrader discussion
- Issue 568 / MSMP resync
@jfs:
- CI workflow maintenance
- figure out mempool requirements, start designing
- #577 Backup databases prior to migration. PR: #578 merged
- opened PR 573: draft of document release candidate testnet policy
- normalize ‘mod tests’ naming PR #571 was merged as part of #574
- working on #579 “job cancelled by caller” error while composing causes panic