Two days.
It took two days to get the basic functionality for participant discovery, publisher and subscriber initialization, as well as a considerable part of the reliable UDP transmission to work.
The overall objective is to be able to start a publisher or subscriber on any device in the network, and have it distribute data automatically without any further configuration.
Participants are background processes that host multiple local publishers and subscribers. Whenever a new participant appears on the network (check by sending/listening to beacon packets), the Participants set up a TCP stream amongs themselves. This way, at all times, each participant pair is connected by one TCP stream. This stream is used to communicated administrative stuff, and the first message that two newly meeting participants are exchanging, is a list of all publishers and subscribers that they manage.
Whenever a new subscriber connects to the local participant, this participant informs every other participant that it knows, about the new subscriber. knowledge of new subscribers in the network is passed to the publishers, so they can start transmitting the data. When a subscriber disappears, all participants are also notified, so in turn, the publishers can stop transmitting.
And all this should run as efficiently as possible, using all available CPU threads as tightly as possible, never blocking on anything, yadda, yadda.
But instead of this being the monumental task that it used to be with C or C++, it is a pretty straightforward bunch of code in asynchronous Rust.
Each participant starts running 4 tasks:
Beacon broadcaster – A beacon message is broadcasted periodically.
Beacon listener – Picks up beacons from other participants, and opens a TCP stream with them.
Peer listener – Responds to incoming TCP connection requests from participant peers. This is the other side of the participant peer connection.
Local listener – Responds to incoming TCP connection requests from publishers and subscribers.
For each TCP connection end (whether to other participant or local subscriber or publisher), a new task is spawned. This task contains all the local state relevant to the connection. Having everything local like that makes the code extremely readable and easy to maintain. There is still shared state needed across all tasks. Each participant maintains mutex-protected lists of peers as well as lists of local publishers and subscribers.
The actual messages sent are Rust enums, for which serialization and deserialization is automatically generated via derive macros. In practice, this means you can define an enum with all possible messages, and “just send it across the TCP stream”.
I’m pretty excited!