[raft][gate 0] Raftstore implementation

[MariemBaccari]

Context

A Raft-based alternative for replication is currently being developed. This issue aims to present the proposed architecture and implementation plan.

Architecture

Consensus layer (under pkg/raftstore/consensus)

This is the Raft engine which wraps the etcd Raft library and includes transport and Raft log persistence.
Consensus runs a handleReady() goroutine which processes the node.Ready() updates provided by etcd Raft. On every iteration, it persists entries and snapshots to the WAL, sends the required messages (Heartbeats, Votes etc. as decided by the underlying etcd Raft) to peers, applies the committed entries to the state machine then calls node.Advance() to communicate to the library that it is done processing the update. The general workflow of the goroutine is as follows:

Storage handles persisting hard state, entries and snapshots by wrapping etcd's WAL implementation.

Generic Store (pkg/raftstore/store.go)

The generic raftstore implements the store.Store[R] interface and connects the consensus layer with the specific DBs layers (currently only scd). The raftstore Transact() implementation builds a proposal from the serialized op data attached to the context, registers the action as a pending transaction then calls consensus.Propose and blocks. When the proposal comes back as a committed entry, the pending transaction is executed. Since the other nodes don't have access to the action, they use per-operation appliers to apply the entry. This means that the current design implies duplicated logic between the API handlers and these appliers.

Raft implementation plan

#1451: Adds the raftstore flag and store stubs
#1460: Adds Raft related dependencies
#1464: Adds Raft transport
#1470: Adds Raft persistence
#1473: Consensus Layer - Initializes and adds consensus configuration
#1474: Consensus Layer - Adds Raft updates consumption
#1525: Adds the generic raftstore
#1550: Use a consensus instance per store (rid, scd, aux)
#1577: Adapts the Transact signature to support raft.

Memstore

The memstore is the in-memory store used by the raftstore, it is implemented in the following PRs:
#1527: Adds the base memstore
#1529 and #1530: Add the aux memstore
#1534 and #1539: Add the rid memstore
#1542: Adds the scd memstore

Future PRs:

• [raft/aux] Add aux Raftstore #1528 Implements the aux Raftstore
• rid Raftstore
• scd Raftstore