Big datadc skyfall_preso_v2

Project Skyfall

Matt Abrams (@abramsm)

Agenda

A bit about AddThis!
!
Why did we need Skyfall?!
!
Architecture!
!
Operations/Performance!

Fun with Numbers
AddThis JavaScript loads > 3 Billion times per day

Edge Network (Skyfall) receives around 4B hits per
day

Either datacenter can handle 100% load (we test this
often)

Currently using around 1K servers (will double next
year)

Why did we need Skyfall?
We couldn’t ﬁnd anyone else to do it for us
•  Pervious vendors log aggregation was delayed by a
minimum of 3 hours and could take up to 5 days

Minimize impact on our publishers
•  Combining log collection with remote services means we only
need 1 event instead of n

Support near real time applications

Skyfall Goals and Architecture!

Skyfall Goals (Technical)
High Availability
Handle Server and DC failure

gracefully
Low latency

Zero downtime deployment and
conﬁguration
Use for internal and external Logging
needs

In session RPC
O(1) reads and writes

Support data ﬁltering at the
edge
Smart Clients

Why speed and robustness matters

Architecture
Web Event
Web Event
Web Event

Global Traffic
Management

DC1 DC2

Skyfall Skyfall Skyfall Skyfall Skyfall Skyfall

Repeater

Consumer Service Consumer
Consumer
Consumer Service Service
Consumer Service Consumer
Consumer Service
Consumer Service

1.  Messages are placed on concurrent non-blocking queue
(CNBQ) to minimize latency impact on producer

2.  Messages are then popped from CNBQ and placed on a
Disk-Backed queue (DBQ)

3.  DBQ is used to provide temporary storage in case Kafka is
down or backed up

4.  Messages from DBQ are popped and sent to Kafka where
they are persisted to ﬁle system

Kafka
Kafka is treats persistence as a ﬁrst class citizen

Focus is on high throughput vs lots of bells and whistles

State about what has been consumed is maintained in the
client rather than the server

Kafka is explicitly distributed

Supports O(1) reads and writes

Pull rather than push

http://incubator.apache.org/kafka/design.html

Circuit Breaker for remote Services
Pattern is used to detect failures and encapsulates logic of
preventing a failure to reoccur constantly[1]

If a service instance throws an error, times out, or responds
with a failure message an error event is marked

If the error rate threshold is exceeded that service instance is
removed from the pool of available services

Before re-adding a service to the pool a test request is made
and validated

Internal service failures should not be reﬂected in response to
message originator

[1] - http://en.wikipedia.org/wiki/Circuit_breaker_design_pattern

What does a call to our endpoint look like?

Topic

•  "GET /live/t00/250lo.gif&foo=bar" 200 37 "http://
s7.addthis.com/static/r07/sh103.html" "Mozilla/5.0
(compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/
5.0)"!


Version
Topic

5.0)"!


Version Resource
Topic

5.0)"!


Version Resource URL Params
Topic

5.0)"!


Version Resource URL Params Status Code
Topic

5.0)"!


Topic
Bytes Transferred

5.0)"!


Topic
Bytes Transferred

5.0)"!

CDN Resource User Agent


Version Resource URL Parameters Status Code
Topic
Bytes Transferred

"GET /live/t00/250lo.gif&foo=bar" 200 37 "http://
5.0)"

CDN Resource User Agent

The endpoint also receives header and cookie information not
Shown here.

Zero Downtime Deployment and
Conﬁguration

Group 1
4 8 16
S1 S2 S2 S3 S3 S4 S4 S5 S5

Group 2
4 8 16
S1 S2 S2 S3 S3 S4 S4 S5 S5

Endpoint Conﬁguration

Each endpoint maps to a ‘topic’

Header elements may be extracted from the HTTP request

Parameters may be mapped to new key names

Variables may be extracted from the URL path

Data Center Repeater

DC Repeater nodes
automatically negotiate N1
peering relationships with
nodes in the other data N1
center
N2
If a peer node becomes
unreachable the local node N2
will select a new peer
N3
These are special consumers
of the Kafka log data created
by the local node

Requests per/second (VA Data Center)

TCP - When do you say goodbye?

http://upload.wikimedia.org/wikipedia/commons/a/a2/Tcp_state_diagram_fixed.svg

Connection Tracking – what you need to
know
Connection information is maintained in memory

The message: “ip_conntrack: table full, dropping packet” is
BAD

Chrome – doesn’t close connection on FIN

This means that the connection info remains open until it
times out, drastically increasing the number of connection
your server needs to track

You need some mechanism for timing out the connection in a
reasonable time period

HA Proxy
We use a simple round-robin load balancing algorithm with a
liveness check

Default connection timeouts are way to high. Reasonable
values are used to prevent excessive connection tracking

“http-close” and “http-server-close” are enabled to ensure low
latency for clients and fast session reuse for the server

HA Proxy is our solution of choice our LB needs. We prefer
software solutions on commodity hardware vs expensive
custom LB appliances

They could use a new logo

Big datadc skyfall_preso_v2

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Big datadc skyfall_preso_v2

Semelhante a Big datadc skyfall_preso_v2 (20)

Big datadc skyfall_preso_v2