There is no “the” UTXO set.
The UTXO set model is way of thinking about the state of the Bitcoin blockchain, and something that guides developement. But it is generally not materialized as an actual data structure in its entirety, and there isn’t just a single one.
Internally in Bitcoin Core, UTXO sets are represented by the abstract CCoinsView class. There are multiple implementations for it:
- one that’s backed by an on-disk database.
- one that’s backed by a set of cached in-memory changes on top of another UTXO set (used to represent the UTXO set of the currently-active blockchain).
- one that’s backed by the mempool on top of another
When validating lose transactions that are being relayed and accepted into the mempool, the mempool’s UTXO set is used (which consists of all UTXOs in the currently active block, plus implicitly all outputs from mempool transactions added, plus implicitly all inputs from mempool transactions deleted).
When validating a block, the UTXO set corresponding to the currently active block is used, which does not include the implicit changes due to the mempool.
To answer your questions:
Will the 5btc output be deducted(locked to B’s address) from the utxo set while the txn is in the mempool(unconfirmed)?
From the mempool’s implicitly-defined UTXO set, yes. From the blockchain’s UTXO set, no.
If no, can A again use the 5BTC from the utxo set?
That’s unrelated to how the database is modeled, but governed by mempool replacement policies. Using RBF (replace-by-fee) it is in some cases possible to replace a mempool transaction with a higher-feerate one. This happens before validation against the mempool’s UTXO set happens.
