Lectures 10. Longest-chain consensus.

Resources:
HW:

In this lecture we are working towards the two bottom results in this table:

Synchrony
Partial synchrony
Asynchrony
Permissioned
✅ [Dolev-Strong] PKI, any f<n ⇒ BB protocol ❌ [PSL, FLM] no PKI, f≥n/3 ⇒ no BB protocol
✅ ❌ [DLS] f<n/3     \iff SMR protocol, favoring safety over liveness in attack mode
❌ [FLP] f=1 ⇒ no BA protocol
Permissionless
✅ [Nakamoto] majority of hashrate is honest (f<n/2) ⇒ non-deterministic SMR protocol
✅ [Nakamoto] majority of hashrate is honest (f<n/2) ⇒ non-deterministic SMR protocol, favoring liveness over safety in attack mode

Contents of the lecture:

Two Protocol Paradigms

  1. BFT-type protocols (e.g. Tendermint)
    • canonical tool: multiple rounds of voting to ensure consistency
    • prefer consistency over liveness, and achieve consistency always (assuming f < n/3)
    • failure mode under attack: stall
    • very difficult to resolve forks (due to bugs, f ≥ n/3, etc) in-protocol
  2. Longest-chain protocols (e.g. Bitcoin, Ethereum)
    • no explicit voting, embrace forks as normal, resolve ambiguity in-protocol
    • favor liveness over consistency
    • failure mode under attack: big chain reorganizations (e.g. Ethereum Classic)
    • enables double-spend attacks

Bitcoin’s innovations

Really, there are two innovations in the bitcoin whitepaper, which together form Nakamoto consensus:

  • Longest-chain consensus - which blocks are confirmed. Can be studied in both permissioned and permissionless settings.
  • Proof-of-work - who can propose blocks. Enables permissionless setting, given we use the longest-chain consensus. [In more detail, it’s a sybil resistance mechanism, which ensures that too many Byzantine nodes joining network cannot break it. Viewed yet differently, proof-of-work is one-CPU-one-vote.]

To reiterate: primary reason for using longest-chain consensus is to be able to promote it to the permissionless setting.

Longest-chain consensus (abstractly)

  • This material is relevant for:
  • (PKI) permissioned + PKI setting (PoW) permissionless, proof-of-work [rmk: here “longest-chain” means chain with the highest total difficulty] (PoS) permissionless, proof-of-stake

  • Longest-chain consensus is primarily studied in the synchronous model (where ∆ bounds msg delays)
  • In the (i) PKI setting, this protocol is inferior to Dolev-Strong protocol. Still, it is very useful to study it, so that later transition to permissionless setting is smooth.

Longest-chain consensus protocol:

[for all three settings (PKI), (PoW), (PoS)]

(1) Hard-coded “genesis block” B0B_0

(2) In each round r=1,2,3,… [Definitions of round in each settting: (PKI): as before, rounds are governed by time; (PoW): rounds are event driven, by blocks getting mined. No shared clock needed! (PoS): same as in (PKI), governed by time]

(2a) One node is chosen as a leader [How chosen in each settting: (PKI): round-robin rotation OR randomly; (PoW): leader is the node that mined the block; (PoS): random selection, probability of being selected is proportional to the stake of the node]

(2b) Leader can create a set of round-r blocks, each with a predecessor block (could be B0B_0, forks allowed!) [In BFT-type protocols this doesn’t make sense, everyone knows which is the latest block, so there can only be one predecessor.]

Rmk. Blocks form an in-tree:

image

Assumptions:

(A1) (trusted setup) B0B_0 not chosen by, or known in advance to Byzantine nodes.

  • Really have to assume this

(A2) (to be enforced) Round-r leader can prove itself to other nodes + non-leaders cannot masquerade as a leader.

  • How to impose in each setting: (PKI): since leaders are governed by time, everybody knows who the leader is at every moment; (PoW): leader can prove themselves by presenting the mined block (hash with many zeroes); (PoS): being a leader there = having a proof that you are the leader, by some randomized algorithm.

(A3) (to be enforced) Nodes cannot manipulate their probability of being chosen as leader in (2a)

  • How to impose in each setting: (PKI): no problem; (PoW): “random oracle assumption” for hash functions guarantees this; (PoS): much harder here, since no source of randomness in-protocol. Fancy cryptography needed.

(A4) (to be enforced) Every round-r block’s predecessor was created in some previous round.

  • Consequences: (i) excludes Byzantine nodes creating a lot of blocks pointing to each other at step (2b) (⇒ no cycles!) (ii) excludes Byzantine nodes doing the following “delaying into the future”: not announce their block at r=17, and then at r=27 announce the block from r=17 and specify the predecessor to be some blocks created between r=18 and r=26. Note: simple delaying is allowed for Byz nodes, ie revealing in 27th round the r=17 block which points to r=13 block. This will be important in selfish mining. (iii) implies that leader can only add one block to any given chain in the tree.
  • How to impose in each setting: (PKI): You can have blocks “expire”, meaning honest nodes ignore delayed blocks from Byz nodes, to deal with (ii) above + honest nodes can detect the scenario (i) above; (PoS): Same ideas as in (PKI); (PoW): is automatically true, because node specifies the predecessor before it is notified that it is a leader. In other words, one can mine a block only with a previous block predecessor specified. In fact, PoW guarantees.

(A4’) The leader of a round produces exactly one block, and this block claims as its predecessor some block that belongs to a previous round. [Thiis stronger assumption is not needed, but will make some of the proofs easier]

(A5) (to be relaxed) At all times, all honest nodes know about the exact same set of blocks (and predecessors). [”instant communication mode” or “supersynchronous model”: ∆=0]

  • Rmk. This implies consistency trivially, but does not imply finality. Wasn’t the whole point is to sync honest nodes? With this assumption, each time an honest node creates block, that block automatically propogated to all other honest nodes, so consistency trivially holds. BUT Consistency with a future-yourself is not guaranteed (basically that blocks do not get rolled back). We call this property “finality
  • How to impose in each setting: (PKI): (A5) is not big of a deal, each leader rotation we can run Dolev-Strong BB protocol to sync honest nodes… In this model then, our protocol would be very similar to Dolev-Strong protocol, and only will differ from it in the way it treats Byzantine nodes… (PoW): we will be able relax (A5) to the synchronous model, as long as round-time >> ∆ (PoS): we will be able relax (A5) to the synchronous model, as long as round-time >> ∆

Rmk. (A2) and (A3) matter much more in PoW and PoS versions.

Honest vs Dishonest Behavior

Honest node behavior in block proposer step (2b):

(i) Form a block B := all known pending txs

(ii) Set predecessor := end of current longest chain (pick any if several)

(iii) Immediately broadcast B (+ specified predecessor) to all other nodes

Byzantine node: can deviate from any of (i-iii), but must commmit to B & predecessor at the time it’s selected as a leader. This is because “switching predecessors” can very quickly lead to rolling back txs.

image

Rolling blocks back: k Byzantine leaders in a row can “cancel” last k-1 blocks ⇒ blocks at the tip are “not confirmed”. How long do you have to wait to consider them confirmed? See below!

f ≥ n/2 ⇒ honest nodes are doomed: 51% Byz nodes ⇒ out of 1000 leaders, about 510 will be Byzantine and 490 honest, and due to 20 nodes excess those 510 nodes can eventually cancel the last 19 nodes on the chain (not as drawn on the right, there maybe a scenario when there are no 20 consecutive Byz leaders, but in total those 510 Byz nodes can outpace those 19 blocks…). Same thing for 10000 leaders, but the result is that 199 blocks can be canceled. Etc.

image

So here is the summary:

Longest-chain protocol:

(1) Hard-coded “genesis block” B0B_0

(2) In each round r=1,2,3,…

(2a) One node is chosen as a leader

(2b) Leader can create a set of round-r blocks, each with a predecessor block (could be B0B_0, forks allowed!). Honest blocks pick predecessors on the longest-chain. [PoW: longest according to total difficulty]

Assumptions:

(A1) B0B_0 not chosen by, or known in advance to Byzantine nodes.

(A2) Round-r leader can prove itself to other nodes + non-leaders cannot masquerade as a leader.

(A3) Nodes cannot manipulate their probability of being chosen as a leader

(A4) Every round-r block’s predecessor was created in some previous round.

(A4’) The leader of a round produces exactly one block, and this block claims as its predecessor some block that belongs to a previous round

(A5) At all times, all honest nodes know about the exact same set of blocks (and predecessors): ∆=0

So here is the summary of the protocol with assumptions embedded:

🔷
Longest-chain protocol:

(1) “Genesis block” B0B_0 hard-coded (A1) by an honest party.

(2) In each round r=1,2,3,…

(2a) One node is (A2+A3) randomly chosen as a leader.

(2b) Leader creates and broadcasts (A4’) one new round-r block, specifying a round-k predecessor block, where (A4) k<r and the predecessor is the tip of the current longest chain. [PoW setting: longest according to total difficulty]

Goals of the longest-chain consensus: finality & liveness

Assumption (A5) (“super-synchronous” model, all messages delivered instantaneously: ∆=0) ⇒ trivializes consistency between nodes, but not self-consistency (since blocks can be rolled back).

Rmk: this assumption reduces clutter that would otherwise block the key ideas. All results extend to the usual synchronous model assuming rate of block production is slow relative to ∆, since this implies that inadvertant honest forks (between honest nodes) rarely occur; see the last two papers in the resources section above]

Def. A block is considered finalized if it belongs to the longest chain and willl not be rolled back.

Ultimate goal for finality: there exists k > 0 such that [f < n/2 ⇒ all but last k blocks on the longest chain can be considered probabilistically finalized]

Notation: Bk(G)B_k(G) := all but last k blocks on longest chain, where GG is the current in-tree:

image

Our intention is to consider blocks in Bk(G)B_k(G) finalized.

Note 1: parameter k is not a part of the protocol description.

Note 2: not clear Bk(G)B_k(G) is well-defined. Turns out that if f < n/2 and k>>1, then it is! See below.

⚔️
Goals: given f < n/2 and k>>1, we have:

(1) Common prefix property: Bk(G)B_k(G) is well-defined

(2) Finality

(3) Liveness

Plan for the proofs: [f < n/2] ⇒ [(0) “sequence of leaders is ww-balanced for ww>>1”] ⇒ ⇒ (1) and (2) and (3)

Definition: (”A” stands for an adversarial node, “H” stands for honest node) A sequence 1,2,3,{H,A}\ell_1,\ell_2,\ell_3,\ldots\in\{H,A\} is ww-balanced (ww stands for “window”) if, in every window i,i+1,,j\ell_i,\ell_{i+1},\ldots,\ell_j of length ≥ww, strict majority are honest. Note 1. We want ww as small as possible (since this controls time of tx confirmation)] Note 2. Conceivably, sequence of leaders is ww-balanced for ww sufficiently large, if <51% honest nodes.

Main Theorem: If f < n/2, then there exists k > 0 such that the probability of [(0) the leader sequence is 2k2k-balanced] is close to 1.

Moreover, under the property (0), for every possible sequence of in-trees G0,G1,G2,G_0, G_1, G_2, \ldots we have: (1) common prefix property: for all ii, the prefix Bk(Gi)B_k(G_i) is well-defined; (2) finality: Bk(G0)Bk(G1)Bk(G2)Bk(G3)B_k(G_0)\subseteq B_k(G_1) \subseteq B_k(G_2) \subseteq B_k(G_3)\subseteq \cdots (once confirmed, always confirmed) (3) liveness: if a tx is known to all honest nodes, it will eventually be included in Bk(G)B_k(G)

Rmk. We wrote “for every possible” because the sequence is not fully determined because Byz nodes can do bad things, and honest nodes can break ties between longest chains arbitrarily.

Plan for the proofs
Plan for the proofs

(0) Balanced leaders property: f < n/2 ⇒ random leaders are balanced

From the viewpoint of ww-balanced sequences, round-robin leader selection is not as good as random — so we focus on random choice!

Good news: reasonabe to expect randomly chosen leaders to be balanced.

Bad news: stuck with non-zero (but hopefully astronomically small) failure probability.

⇒ forced to settle for probabilistic guarantees (i.e., have probabilistic finality). [Aside: this an extremely powerful idea in cryptography and computer science: things that are completely impossible become very much possible if we are okay with having 0.9999999 probability of them happening. And this is completely okay for reality, because the chance that an asteroid hits the planet is also non-trivial…]

Notation: α\alpha = fraction of nodes that are Byzantine. Assume α<12\alpha<\frac 1 2.

Note: for any window of consecutive leaders, expect 1α>121-\alpha>\frac 1 2 fraction to be honest (on average)

Intuition: bigger window length ww ⇒ bigger gap (12α)w(1-2\alpha)w between expected number of honest vs Byzantine nodes ⇒ bigger gap to absorb variation around expectation ⇒ less likely to see ≥50% Byzantine nodes.

Math: this probability is decreasing exponentially with ww!

Pr[a given length-ww window is ≥50% Byzantine] ≤ ecwe^{-cw} (c = some constant) [This is due to Chernoff bound, see Problem 3 in HW, or this link…]

⇒ Pr[any window of length ≥ww is ≥50% Byzantine] ≤ w\sum_wPr[…] (by a union bound)

⇒ Pr[any window of length ≥ww is ≥50% Byzantine] ≤ T2T^2ec(α)we^{-c(\alpha)w} (where T is the length of a sequence; T2T^2 because window starts and ends somewhere)

⇒ get failure probability ≤δ\delta as long as w1c(α)(2lnTlnδ)w\geq \frac {1} {c(\alpha)} (2\ln T - \ln \delta)

⇒ get failure probability ≤δ\delta as long as wc2(α)(lnT+ln1δ)w\geq c_2(\alpha)(\ln T + \ln \frac 1 \delta) where constant c2(α)c_2(\alpha) depends only on α\alpha

Since lnT\ln T and ln1δ\ln \frac 1 \delta grow very slowly, we can take ww that satisfies the above inequality for all conceivable TT

(0) Balanced leaders property ⇒ (1) Common prefix property

Theorem 1 (common prefix property) If a leader sequence is (2k+2)-balanced + assumptions (A1), (A5), (A4’) hold, then for any possible in-tree G known to all honest nodes (due to (A5)), the prefix Bk_k(G) is well-defined.

Rmk. We wrote “any possible in-tree” due to tie-breaking and Byzantine nodes.

Proof: assume the theorem is not true. Then there is an in-tree G that looks as follows:

image

Trace back to the first B0_0 which is honest (possible due to A1). We are doing this since B* could have been produce by Byzantine node (who delayed its production, for example).

image

Suppose r0_0 is the round of B0_0’s creation and announcement.

Now, the claim is that the bad unbalanced window (with ≥50% Byzantine leaders) is the window between r0_0 and the moment when the in-tree G is known to everyone.

(def: height of a block = the length of the chain from the block to the genesis block.)

Claims: (that finish the proof)

  1. ≥2k+2 leaders after r0_0 until G
  2. All the blocks pointing to B0_0 are created after r0_0, since they needed to know about B0_0 ⇒ 2k+2 purple blocks are created after r0_0 ⇒ 2k+2 leaders appeared, by the assumption (A4’): qed.

  3. ≤ h-h0_0 honest leaders after r0_0 until G
  4. Proposition: the heights of honest blocks strictly increase over time; follows obv from (A5). The statement b. now follows immediately from the proposition.

  5. ≥ h-h0_0 Byzantine leaders after r0_0 until G
  6. Well, half of the purple blocks need to be produced by Byzantine nodes + red blocks need to be produced by Byzantine nodes ⇒ we get precisely at least h-h0_0 Byzantine nodes.

qed.

(0) Balanced leaders property ⇒ (2) finality

Theorem 2 (finality) If a leader sequence is (2k+2)-balanced, then for any possible outcome sequence of in-trees G0,G1,G2,G_0, G_1, G_2, \ldots we have Bk(G0)Bk(G1)Bk(G2)Bk(G3)B_k(G_0)\subseteq B_k(G_1) \subseteq B_k(G_2) \subseteq B_k(G_3)\subseteq \cdots (once confirmed, always confirmed).

Rmk. We can write GiG_i as opposed to “GiG_i known to a node jj” because an in-tree known to one honest node is immediately propagated to the others, via assumption (A5).

Proof: suppose the opposite: a block b belongs to Bk(Gi)B_k(G_i) but not Bk(Gj)B_k(G_j) for some j>ij>i.

⇒ b belongs to every longest chain of GiG_i (we can even say that it is >k blocks deep there), but does not belong to any longest chains of GjG_j (if it is >k blocks deep, then it is in Bk(Gj)B_k(G_j) which is not true; if it is ≤k blocks deep, then this contradicts with b being in Bk(Gi)B_k(G_i))

Consider now the sequence of in-trees Gi,Gi+1,,GjG_i,G_{i+1}, \ldots, G_j, each time one block is added (if Byz node adds multiple blocks, we view it as multiple in-trees)

There needs to be the first (in the sequence Gi,Gi+1,,GjG_i,G_{i+1}, \ldots, G_j,) in-tree GhG_h when the block b is excluded from the longest chain:

image

(The content of Theorem 1 above was precisely the fact that such structures as above cannot arise…)

GhG_h has two longest chains that disagree on last ≥k+1 blocks ⇒ Bk(Gh)B_k(G_h) is not well-defined, contradicting Theorem 1.

qed.

(0) Balanced leaders property ⇒ (3) liveness

Theorem 3 (liveness) If a leader sequence is (2k)-balanced, whenever a tx is known to all honest nodes, it will eventually be included in Bk(G)B_k(G).

Proof: define an epoch as 2k consecutive rounds (+ their leaders).

By 2k-balancedness, ≥k+1 honest leaders ⇒ ≤k-1 Byzantine leaders in each epoch.

Note 1. Every honest leader adds 1 to the length of longest chain.

Note 2. Each Byzantine leader contributes ≤1 block to any given chain, by assumptions (A4).

⇒ after T epochs, longest chain contains ≥(k+1)T blocks, of which ≤(k-1)T are contributed by Byzantine nodes

⇒ ≥ 2T blocks on longest chain are produced by honest nodes (⇒ honest blocks are added to the longest chain infinitely often)

⇒ ≥ 2T-k of these blocks have been finalized

⇒ honest blocks are finalized inifinitely often

⇒ a tx* known to all honest nodes will eventually be finalized! [This is true since after each honest block is added the tx* will be in the longest chain (either in the past or in the new block)]

Chain Quality: the fraction of honest blocks

Possible stronger liveness guarantees:

(i) assume tx known to only one honest node; but then Theorem 3 becomes false!

(ii) quantitative bounds on time-till-finalization (research on this exists)

(iii) lower bound fraction on honestly produced bllocks on the longest chain

Revised analysis of randomly chosen leader, property (0)

As long as wc(ϵ)(lnT+ln1δ)w\geq c(\epsilon)(\ln T + \ln \frac 1 \delta), we have

Pr[any window of length ≥w has ≤(α+ϵ\alpha+\epsilon) fraction of Byzantine leaders] ≤ δ\delta

⇒ leader sequence is (w,α+ϵw,\alpha+\epsilon)-balanced with probability ≥1-δ\delta

Theorem 3’: if leader sequence is (2k, α+ϵ\alpha+\epsilon)-balanced, then ≥ 12α2ϵ1αϵ\frac{1-2\alpha-2\epsilon}{1-\alpha-\epsilon} fraction of blocks on longest chain is produced by honest nodes.

(Thm 3 is a special case of above, where α+ϵ\alpha+\epsilon is very close (and slightly less than) to 1/2, then 12α2ϵ1αϵ\frac{1-2\alpha-2\epsilon}{1-\alpha-\epsilon} tends to 0)

Proof: is very similar to Thm 3:

… ⇒ after T epochs, longest chain contains ≥2k(1 αϵ-\alpha-\epsilon)T blocks, of which ≤2k(α+ϵ\alpha+\epsilon)T are contributed by Byzantine nodes

⇒ ≥ 2k(1αϵ)T2k(α+ϵ)T2k(1αϵ)T\frac{2k(1-\alpha-\epsilon)T-2k(\alpha+\epsilon)T}{2k(1-\alpha-\epsilon)T}=12α2ϵ1αϵ\frac{1-2\alpha-2\epsilon}{1-\alpha-\epsilon} fraction of blocks is honestly produced.

qed.

What about partial synchrony?

We had synchronous model with ∆=0, but as wee said, this can be extended to ∆≠0 provided the average block time is >>∆, which guarantees that honest nodes don’t produce accidental forks too often.

If we want to impose partial synchrony and consider what happens to longest-chain consensus in an attack, some of the theorems above do not work:

Note: finality (Theorem 2) of longest-chain consensus breaks down in partial synchrony, even if all nodes are honest (f=0)! This is because the adversarial message delivery can model a network partition, where the sat of nodes A does not talk to the set of nodes C, and therefore they operate independently:

image

⇒ once the length of two forks exceeds k, we lose the common prefix property, and therefore it doesn’t even make sense to talk about finality. In particular, if one fork is shorter than the other (let’s say the one below, created by nodes C), when the network partition ends finality for nodes C truly breaks, since they have to throw out their whole chain.

Note: longest-chain consensus still makes progress with network partition!

In a nutshell, under attack we have lost consistency (finality, to be precise: the blocks can roll back, arbitrarily large number of them), but did not lose liveness (see below). In contrast to this, BFT-type protocol (e.g. Tendermint) in case of such a network partition, will just stall.

⇒ “longest-chain favors liveness over safety” (fail via reorgs) “BFT-type protocols favor safet over liveness” (fail by stalling)

Permissionless setting

BFT-type protocols wouldn’t quite work right away, since they are based on voting, and nodes wouldn’t know the number of nodes in advance, and therefore they wouldn’t know the threshold.

This problem can be circumvented by “wrapping” the BFT-type protocol via some sort of authentification of nodes (see ETH 2.0 design, for example, where validators do know about each other).

Longest-chain consensus, in contrast, extends to permissionless setting amazingly well: (Nakamoto probably invented longest-chain consensus presiccely to be able to extend it to permissionless setting.)

🔷
Longest-chain protocol:

(1) “Genesis block” B0B_0 hard-coded by an honest party.

(2) In each round r=1,2,3,…

(2a) One node is randomly selected as a leader.

(2b) Leader creates and broadcasts one new round-r block, specifying a round-k predecessor block, where k<r and the predecessor is the tip of current longest chain. [PoW setting: longest according to total difficulty]

Key points for transition:

  1. Balanced leadder sequence ⇒ consistency + liveness, and here we did not really care about being permissioned / permissionless.
  2. ⇒ we only need to take care of Theorem (0), where we proved that the leader sequence is w-balanced for some w>0 provided f<n/2.

  3. Proof-of-work (see Lecture3) allows to substitute one-node-one-vote by one-CPU-one-vote, and allows us to have f<n/2 in the premissionless setting, provided the majority of computational power is honest!
  4. More precisely, as long as the majority of hashrate is honest, we get that in each round r Pr[round-r leader is Byzantine]≤α\alpha for some α\alpha<1/2, and therefore the leader sequence will be w-balanced with high probability (parameter w increases as α\alpha tends to 1/2).

  5. [the only subtlety]
  6. The probability of being chosen as a leader now depends on the difficulty number of the previous block, and therefore, in order to preserve assumption

    (A3) Nodes cannot manipulate their probability of being chosen as a leader”

    we need “longest chain” to mean longest with respect to total difficulty.