This presentation covers HyParView and Plumtree - protocols used to build highly scalable clusters of data capable of gossiping messages between thousands of clients.

1. SCALING CONNECTIONS IN
P2P APPS

2. INTRODUCTION
Bartosz Sypytkowski
▪ @Horusiath
▪ b.sypytkowski@gmail.com
▪ bartoszsypytkowski.com

3.  P2P vs. Client/Server
 Networking 101
 Membership & peer discovery
 Gossiping data efficiently
AGENDA

4. CLIENT / SERVER PEER TO PEER
Local network
Local network
“The cloud”
Local network
Local neatwork

5. CLIENT / SERVER PEER TO PEER
• Roles: initiator or acceptor
• Cluster of predictable size
• Homogeneous, fast, stable network
• Servers: powerful hardware, always
on
• Dedicated roles (DB, app server,
cache)
• Roles: both initiator and acceptor
• Unbounded number of members
• Variadic, unpredictable network
• Peers: weak hardware, powered by
batteries
• No preconfigured roles

6. CLIENT / SERVER PEER TO PEER
Local network
Local network
“The cloud”
Local network
Local neatwork
How two devices
can discover each
other?

7. NETWORK ADDRESS TRANSLATION
Internet
Router Server
172.23.208.1
200.100.10.1
Client

8. NETWORK ADDRESS TRANSLATION
Internet
Router Server
172.23.208.1
200.100.10.1
Source: 172.23.208.1
Destination: 200.100.10.1
Client

9. NETWORK ADDRESS TRANSLATION
Internet
Router Server
172.23.208.1
200.100.10.1
Source: 172.23.208.1
Destination: 200.100.10.1
Client

10. NETWORK ADDRESS TRANSLATION
Internet
Router Server
172.23.208.1
200.100.10.1
Source: 84.11.65.1
Destination: 200.100.10.1
NAT table
172.23.208.1 84.11.65.1
Client

11. NETWORK ADDRESS TRANSLATION
Internet
Router Server
172.23.208.1
200.100.10.1
Source: 84.11.65.1
Destination: 200.100.10.1
NAT table
172.23.208.1 84.11.65.1
Possible further
source IP
changes
Client

12. HOW DO WE KNOW THE IP ADDRESS OF ACCEPTOR?

13. NETWORK 101
DNS
172.23.208.1
Server
200.100.10.1
DNS Server
DNS records
example.com 200.100.10.1
Client

14. NETWORK 101
DNS
172.23.208.1
Server
200.100.10.1
DNS Server
DNS records
example.com 200.100.10.1
Client

15. WHY CAN’T WE SETUP A DNS RECORD FOR EVERY PEER?

16. STUN
Router STUN Server
172.23.208.1
stun.l.google.com
Client
DISCOVERING PUBLIC IP

17. STUN
Router STUN Server
172.23.208.1
stun.l.google.com
Client
DISCOVERING PUBLIC IP
What is my public IP?
Source: 172.23.208.1
Destination: 200.100.10.1

18. STUN
Router STUN Server
172.23.208.1
stun.l.google.com
Client
DISCOVERING PUBLIC IP
What is my public IP?
Source: 84.11.65.1
Destination: 200.100.10.1
NAT table
172.23.208.1 84.11.65.1

19. STUN
Router STUN Server
172.23.208.1
stun.l.google.com
Client
DISCOVERING PUBLIC IP
Your public IP is:
84.11.65.1
NAT table
172.23.208.1 84.11.65.1

20. STUN
Router STUN Server
172.23.208.1
stun.l.google.com
Client
DISCOVERING PUBLIC IP
Your public IP is:
84.11.65.1
NAT table
172.23.208.1 84.11.65.1
Periodically send ping to
keep the NAT table
mapping unchanged.

21. NOT ALL FIREWALLS ENABLE DIRECT DEVICE-DEVICE
CONNECTION

22. TURN
Firewall
84.16.55.1
Client
RELAYING MESSAGES OVER NATS/FIREWALLS
TURN Server
54.23.201.1
Client

23. ICE
NEGOTIATE THE CONNECTION CAPABILITIES

24. ICE
NEGOTIATE
WHICH
CONNECTION TO
USE
const conn = new RTCPeerConnection({
iceServers: [
{
urls: "stun:stunserver.example.com:3478",
},
{
urls: 'turn:turnserver.com:3478',
credential: 'password',
username: 'username'
}
]
})

25. CLIENT / SERVER PEER TO PEER
Local network
Local network
“The cloud”
Local network
Local neatwork
What about that
part?

26. mDNS svc-1.local 192.168.0.1 svc-3.local 192.168.0.3 svc-5.local 192.168.0.5
svc-2.local 192.168.0.2 svc-4.local 192.168.0.4

27. mDNS svc-1.local 192.168.0.1 svc-3.local 192.168.0.3 svc-5.local 192.168.0.5
svc-2.local 192.168.0.2 svc-4.local 192.168.0.4
Where is svc-4.local?

28. mDNS svc-1.local 192.168.0.1 svc-3.local 192.168.0.3 svc-5.local 192.168.0.5
svc-2.local 192.168.0.2 svc-4.local 192.168.0.4
Where is svc-4.local?
Where is svc-4.local? Where is svc-4.local?
Where is svc-4.local?
multicast

29. mDNS svc-1.local 192.168.0.1 svc-3.local 192.168.0.3 svc-5.local 192.168.0.5
svc-2.local 192.168.0.2 svc-4.local 192.168.0.4
svc-4.local = 192.168.0.4

30. mDNS svc-1.local 192.168.0.1 svc-3.local 192.168.0.3 svc-5.local 192.168.0.5
svc-2.local 192.168.0.2 svc-4.local 192.168.0.4
svc-4.local = 192.168.04
svc-4.local = 192.168.04
svc-4.local = 192.168.04 svc-4.local = 192.168.04
multicast

31. mDNS
svc-1.local 192.168.0.1
svc-4.local 192.168.0.4
svc-3.local 192.168.0.3
svc-4.local 192.168.0.4
svc-5.local 192.168.0.5
svc-4.local 192.168.0.4
svc-2.local 192.168.0.2
svc-4.local 192.168.0.4
svc-4.local 192.168.0.4
svc-4.local = 192.168.0.4

32. mDNS
const mdns = require('mdns')
// advertise service svc-1 at port 9999 via TCP
const service = mdns.createAdvertisement(mdns.tcp(), 9999, {
name: 'svc-1'
})
service.start()
// discover services
const browser = mdns.createBrowser(mdns.tcp())
browser.on('ready', () => browser.discover())
browser.on('update', (data) => {
console.log(data);
// {
// interfaceIndex: 4,
// name: svc-1',
// networkInterface: 'en0',
// type: {name: '', protocol: 'tcp', subtypes: []},
// replyDomain: 'local.',
// fullname: 'svc-1._tcp.local.',
// host: 'svc-1.local.',
// port: 9999,
// addresses: [ '10.1.1.50', 'fe80::21f:5bff:fecd:ce64' ]
// }
})

33. CLUSTERING

34. HOW TO BUILD A CLUSTER THAT COULD SPAN OVER >1K
NODES USING DIFFERENT NETWORKS?

35. HYPARVIEW
HYBRID PARTIAL VIEW FOR CLUSTER MEMBERSHIP

36. CONNECTING EVERYONE TO EACH OTHER DOESN’T SCALE

37. CONNECTING EVERYONE TO EACH OTHER DOESN’T SCALE
BUT…

38. NAÏVE
CONNECTIVITY
ISSUES
F
A
E B
C
D
Connections limit: 4

39. NAÏVE
CONNECTIVITY
ISSUES
F
A
E B
C
D
Connections limit: 4
Can I join?

40. NAÏVE
CONNECTIVITY
ISSUES
F
A
E B
C
D
Connections limit: 4
Sorry, I’m at
my limit.

41. SOLUTION: INTRODUCE PRIORITY CONNECTIONS

42. PRIORITY
CONNECTIONS
F
A
E B
C
D
Connections limit: 4

43. PRIORITY
CONNECTIONS
F
A
E B
C
D
Connections limit: 4
SUDO: let
me join

44. PRIORITY
CONNECTIONS
F
A
E B
C
D
Connections limit: 4
drop existing connection at
random to free the pool

45. PRIORITY
CONNECTIONS
F
A
E B
C
D
Connections limit: 4
establish new connection

46. F
A
E B
C
D
Connections limit: 4
FORWARD NEW
PEER INFOR TO
OTHERS

47. F
A
E B
C
D
Connections limit: 4
FWD(1)
FORWARD NEW
PEER INFOR TO
OTHERS

48. HYPARVIEW 101
A
C
B
D
passive view
active view
K
J
I
H
G
F
E
L
M
N
Active peers
Passive peers

49. HOW TO GOSSIP MESSAGES EFFICIENTLY?

50. PLUMTREE
EPIDEMIC BROADCAST TREES

51. PLUMTREE 101
A
C
B
D
K
J
I
H
G
F
E
L
M
N

52. BUILDING A TREE
A
C
B
D
K
J
I
H
G
F
E
L
M
N

53. BUILDING A TREE
A
C
B
D
K
J
I
H
G
F
E
L
M
N

54. BUILDING A TREE
A
C
B
D
K
J
I
H
G
F
E
L
M
N

55. BUILDING A TREE
A
C
B
D
K
J
I
H
G
F
E
L
M
N

56. BUILDING A TREE
A
C
B
D
K
J
I
H
G
F
E
L
M
N

57. BUILDING A TREE
A
C
B
D
K
J
I
H
G
F
E
L
M
N

58. BUILDING A TREE
A
C
B
D
K
J
I
H
G
F
E
L
M
N

59. BUILDING A TREE
A
C
B
D
K
J
I
H
G
F
E
L
M
N

60. BUILDING A TREE
A
C
B
D
K
J
I
H
G
F
E
L
M
N

61. BUILDING A TREE
A
C
B
D
K
J
I
H
G
F
E
L
M
N

62. BUILDING A TREE
A
C
B
D
K
J
I
H
G
F
E
L
M
N

63. BUILDING A TREE
A
C
B
D
K
J
I
H
G
F
E
L
M
N
Eager peers
Lazy peers

64. WHAT IF CONNECTION FAILS?

65. TREE REPAIR
A
C
B
D
K
J
I
H
G
F
E
L
M
N
connection failure

66. TREE REPAIR
A
C
B
D
K
J
I
H
G
F
E
L
M
N
Part of the gossip tree is
disconnected from the rest

67. TREE REPAIR
A
C
B
D
K
J
I
H
G
F
E
L
M
N
Once in a while send message
to lazy peers about latest
gossips ids (m1)

68. TREE REPAIR
A
C
B
D
K
J
I
H
G
F
E
L
M
N
Message receiver awaits for m1

69. TREE REPAIR
A
C
B
D
K
J
I
H
G
F
E
L
M
N
If m1 didn’t arrive before
timeout, send graft back

70. TREE REPAIR
A
C
B
D
K
J
I
H
G
F
E
L
M
N
Promote lazy peer to eager one

71. TREE REPAIR
A
C
B
D
K
J
I
H
G
F
E
L
M
N

72. SUMMARY

73.  Rainbow connections: https://www.youtube.com/watch?v=8_A1CkYfzoM
 Partisan: https://github.com/lasp-lang/partisan
 DotNext:
https://github.com/dotnet/dotNext/tree/1d551b091db81e55d93fb4d45856b3edb11602d3/src/cluster/Do
tNext.AspNetCore.Cluster/Net/Cluster/Discovery/HyParView
 Hyparview: https://bartoszsypytkowski.com/hyparview/
REFERENCES

74. THANK YOU

76. PLUMTREE 101
A
C
B
D
K
J
I
H
G
F
E
L
M
N

77. PLUMTREE 101
A
C
B
D
K
J
I
H
G
F
E
L
M
N
gossip

78. PLUMTREE 101
A
C
B
D
K
J
I
H
G
F
E
L
M
N

79. PLUMTREE 101
A
C
B
D
K
J
I
H
G
F
E
L
M
N

80. PLUMTREE 101
A
C
B
D
K
J
I
H
G
F
E
L
M
N
gossip

81. PLUMTREE 101
A
C
B
D
K
J
I
H
G
F
E
L
M
N

82. PLUMTREE 101
A
C
B
D
K
J
I
H
G
F
E
L
M
N

83. PLUMTREE 101
A
C
B
D
K
J
I
H
G
F
E
L
M
N