Implementing Raft for Browsers with Rust and WebRTC

I've done some "modern" frontend development in the recent past, and it's usually involved horrifically complicated setups with a ton of glue code to get everything working together. Given that WebAssembly is pretty niche and bleeding-edge, I feared the worst. But to my surprise, the WebAssembly setup experience was impressively smooth. I was at "Hello World" using wasm-pack in less time than it's taken me to make an app with create-react-app in the past.

Even better, the wasm-bindgen developers have done a fantastic job creating Rust bindings for just about the entire web browser API space, including the relatively obscure APIs like WebRTC. You can even use Rust Futures with async / await syntax using the wasm_bindgen_futures crate (which was pretty much essential for implementing Raft). A surprising number of Rust crates work flawlessly in WebAssembly (with notable exceptions like Tokio). I haven't dug into it yet, but JS↔Wasm interop also looks pretty straightforward.

The one tricky issue was dealing with JS closures, which were pervasive in the APIs I was using. The wasm-bindgen approach is quite awkward and one of the few cases I ran across where you need to do a form of manual memory management. (Apparently the gloo crate makes it a little easier, but I didn't find out about it until I was done implementing all the hard JS-related parts.)

Implementing WebRTC communication, on the other hand, was much trickier than I anticipated. I didn't know much about WebRTC going into the project, but I had a vague idea that there had to be some sort of server involved for NAT traversal and whatnot.

What I didn't realize is that WebRTC sessions need a signaling server for browser windows to exchange connectivity information before they can communicate with each other. This might not be too bad, except for the fact that WebRTC signaling servers have no standard protocol or popular ready-to-use implementations.

Which means that if you want to use WebRTC, you're going to have to implement your own signaling server with some sort of ad-hoc protocol (or use a SaaS, but where's the fun in that?). WebSockets seems to be the most popular implementation choice; the best documentation resources I could find were an MDN tutorial and a fairly old YouTube talk. (Confusingly, signaling servers are unrelated to NAT traversal, which relies on standardized protocols like STUN and TURN.)

And implementing the signaling server is only half the battle. There's also the arcane WebRTC API to deal with, with its myriad of events and callbacks that arrive in hard-to-predict order and spout enormously unhelpful error messages if you do anything wrong. (To be fair a lot of the complexity comes from video/audio handling, but it's pretty hard to avoid even for data-only use-cases.)

Implementing the WebRTC-specific parts of WRaft probably ended up taking as much time as everything else combined. The end result looks something like this:

• Every cluster gets a 128-bit session key (I probably should have called this a "cluster ID", but oh well). The session key is randomly generated when the first "node" (i.e. browser window) starts a cluster.
• Every node has a 64-bit node id, which is basically Hash(session key, domain name) (so it will remain consistent if the user reloads the browser window but is unlikely to clash with other nodes).
• There's a simple WebSocket-based "introducer" signaling service (written in Rust, of course) that basically just forwards SDP messages between nodes and announces when new nodes join the cluster.
• Nodes with smaller node IDs are responsible for introducing themselves to nodes with bigger node IDs. (The "Smallest Node ID Goes First Protocol" (SNIGFP?) was the simplest way I could think of to make it work.)

Overall it's only about 250 lines of server-side code and 500 lines of browser-side code, but it's the area I'm least confident about (there are definitely bugs with Safari, and I'm pretty sure there are some lurking race conditions on other browsers as well).

To get from WebRTC data channels to something that's usable for implementing Raft, I hacked up a half-baked RPC framework that uses bincode for message serialization (I won't go into details to keep this post from getting too long, but it was fun to implement 😄).

Raft was originally billed as an "understandable" alternative to Paxos (which was previously considered the "go-to" distributed consensus algorithm but is notoriously difficult to use in practice). That may be true relatively speaking, but after implementing Raft I would definitely not say it's easy to understand! There are a whole lot of variables to keep track of across nodes and subtle ways things can go very wrong (I had some nasty bugs from not properly handling duplicate and out-of-order requests, for instance).

What Raft really has going for it is a fantastic descriptive paper and, even better, a formal TLA⁺ specification that's pretty readable (even for someone like me who doesn't know TLA⁺ very well). And once the algorithm has been implemented you end up with a usable real-world system (in contrast to Paxos, which is more of a basic building block that leaves out most of the practical details).

For WRaft, I stuck as close as possible to the TLA⁺ spec to avoid surprises. I didn't implement the trickier "optional" parts of Raft like cluster membership changes and log compaction, even though they would probably be essential to make WRaft useful for real-world apps. (Maybe some day if I have time…)

I don't really want this to turn into yet another "Thoughts on Rust" post, but I will nevertheless take this opportunity to rant a little about the Rust learning curve. Specifically: Why are there so many confusing names? Why Send and Sync instead of, I dunno, Threadsafe and Atomic? Does anyone really think Box, Rc, Arc, etc. are the most clear names for smart pointers? (How about Ptr and SharedPtr or something along those lines?) I was piling on horrifically ugly hacks for days until I found the indispensable Option.take() method (the docs are fine, but how are you supposed to find it in the first place?). FusedFutures confounded me until I realized it was "fuse" as in "fuse box", not "Asian Fusion". I lost count of how many times I misspelled mpsc. And don't even get me started on the inscrutable Pin / Unpin stuff.

(Oh and while we're at it: can we please have a moratorium on the use of static as a keyword in programming languages? It means like eleven different things across languages and at least three in Rust.)

But those are fairly minor gripes when it comes down to it, and overall I enjoy programming in Rust now that I've done my requisite time wrestling with the compiler. (My breakthrough moment came when I realized that "moving" a value kind of means that the value has been "eaten"; I've since started mentally reading compiler errors as "use of devoured value" and so forth.) There's something oddly addictive about doing a big refactor and having all the types click into place like weirdly-shaped puzzle pieces.

I've heard it said that the Rust language guides programmers to make "better" design choices; I'm not entirely sure I buy that, but certain approaches definitely end up causing fewer headaches than others. I thought I'd go through an example for interest's sake.

loop {
    // ...
    select! {
        // Handle RPC requests
        res = self.inner.rpc_rx.next() => {
            let (req, resp_tx) = res.expect("RPC channel closed");
            self.handle_rpc(req, resp_tx, &mut timeout);
        }
        // Handle client requests (they get forwarded to the leader)
        res = self.inner.client_rx.next() => {
            let (req, resp_tx) = res.expect("Client channel closed");
            match req {
                // ...
                ClientRequest::Apply(_) => self.forward_client_request(req, resp_tx),
            }
        }
        // Timeout for when there aren't heartbeats from the leader
        _ = timeout => {
            console_log!("Calling election!");
            return RaftWorkerState::Candidate(self.into());
        }
    }
}

async fn handle(&self, req: RpcRequest<Cmd>) -> Result<RpcResponse<Cmd>, transport::Error> {
    // We make a oneshot response channel for each request
    let (resp_tx, resp_rx) = oneshot::channel();
    // self.tx is the "sender" half of the RPC channel; the "receiver" half is
    // in the Raft worker event loop
    self.tx
        .clone()
        .send((req, resp_tx)) // send both the request and the response channel as a tuple
        .await
        .expect("request channel closed");

    // Now we wait for the response, which the Raft worker should (hopefully)
    // send us
    match resp_rx.await {
        Ok(resp) => resp,
        // ...
    }
}