This document discusses Debian cloud images for Amazon EC2. It provides an overview of Debian, AWS EC2, different types of EC2 machine images including EBS-backed and S3-backed AMIs. It describes how Debian generates and distributes its official Debian AMIs across different AWS regions and architectures while managing the image lifecycle and security updates. It also briefly mentions using Cloudfront CDN for Debian archive access from EC2 instances.
2. Agenda
• What is Debian
• What is AWS EC2
• A meander through block storage for EC2
instances
• Types of images
• Generating & distributing Debian’s AMIs
• Debuab Image lifecycle and security
• If there is time: Debian via Cloudfront CDN
8. What is AWS and EC2
• AWS = Amazon Web Services
• EC2 = Elastic Compute Cloud
– Virtual servers running Linux, Windows, BSD
• Started 2006
• Now with 11 Regions and 52 Edge Locations
• Compute, storage, platform, infrastructure – as-a-service
– typically billed by the hour or by the month
Amazon EC2
10. What is EC2
• Amount of CPU & Memory is combined into
“instance type”:
– Small
– Medium
– Large
– ...
instance
instance
instance
11. What is EC2
• Several instance types are grouped into an
“instance family”:
– General Purpose (balanced memory:cpu)
– Memory Optimised (more memory:cpu)
– CPU Optimised (more cpu:memory)
– Storage Optimised (more ‘ephemerial’ storage)
– GPU (CUDA, OpenCL)
– Cluster Nodes (10 GB/sec networking and more)
12. What is EC2
• EC2 instance run on real servers!
instance instance instance instance
Total number of
(hyperthread)
CPU cores, each
dedicated* to an
instance
Disk inside the
physical server is
deemed
‘ephemeral’. Not
raid, but is local to
CPU and Memory.
Different amounts
of storage
depending on
instance type
RAM is dedicated
to each instance
Each instance can
send a certain
number of packets
per second
17. Persistent (EBS) Storage
Amazon EBS
Mechanical disk
General Purpose SSD (GP2)
Provisioned IOPS (SSD)
Amazon S3
18. Persistent (EBS) Storage
Amazon EBS
Mechanical disk
General Purpose SSD (GP2)
Provisioned IOPS (SSD)
Amazon S3
AFR of a typical standard HDD
Designed for 99.999% availability
(5.26 min/yr)
Single instance attach only
(currently)
1GB..1TB (currently)
Your choice of file-system
Optional transparent encryption
by AWS
Network attached to your
instance back in the EC2
environment
99.999999999% durability
Replicated multiple times
within the same Region
Check-summed and re-check-
summed periodically
Designed for 99.99%
availability (SLA at 99.9%)
Can be shared with other
customers (specific, or all)
unless AWS-encrypted
Can be used to create a
new EBS volume
EBS snapshots cannot be
seen in your S3 buckets
19. Persistent (EBS) Storage
Amazon EBS
Mechanical disk
General Purpose SSD (GP2)
Provisioned IOPS (SSD)
Amazon S3
AFR of a typical standard HDD
Designed for 99.999% availability
(5.26 min/yr)
Single instance attach only
(currently)
1GB..1TB (currently)
Your choice of file-system
Optional transparent encryption
by AWS
Network attached to your
instance back in the EC2
environment
99.999999999% durability
Replicated multiple times
within the same Region
Check-summed and re-check-
summed periodically
Designed for 99.99%
availability (SLA at 99.9%)
Can be shared with other
customers (specific, or all)
unless AWS-encrypted
Can be used to create a
new EBS volume
EBS snapshots cannot be
seen in your S3 buckets
24. Amazon Machine Images
• AMI is “golden master”
• Start as many instances as you like*
AMI
instance
instance
instance
instance instance instance
25. Ephemeral and EBS
• Why is the Ephemeral and EBS storage options
important in AMIs?
Your root volume
/ -> persistent (EBS)
/ -> transitory (Ephemeral)
26. Ephemeral and EBS
• Why is the Ephemeral and EBS storage options
important in AMIs?
Your root volume
1,000 systems for 24 hours,
8 GB EBS each in SYD: ~$30.85
27. Ephemeral and EBS
• Why is the Ephemeral and EBS storage options
important in AMIs?
Your root volume
1,000 systems for 24 hours,
Ephemeral in SYD: $0
28. Ephemeral and EBS
• Why is the Ephemeral and EBS storage options
important in AMIs?
Amazon S3 S3 backed AMI
snapshot
EBS backed AMI
29. CPU Architectures
• EC2 currently supports 2 architectures:
EBS backed AMI S3 backed AMI EBS backed AMI S3 backed AMI
30. Virtualisation Types
• EC2 uses (highly customised) Xen, and
supports two virtualisation types:
Para-
Virtualization
(threads)
Hardware
Virtualization
(emulation)
EBS backed AMI S3 backed AMIEBS backed AMI S3 backed AMI EBS backed AMI S3 backed AMIEBS backed AMI S3 backed AMI
31. Each Region is independent
Para-
Virtualization
(threads)
Hardware
Virtualization
(emulation)
EBS backed AMI S3 backed AMIEBS backed AMI S3 backed AMI EBS backed AMI S3 backed AMIEBS backed AMI S3 backed AMI
EBS backed AMI S3 backed AMIEBS backed AMI S3 backed AMI EBS backed AMI S3 backed AMIEBS backed AMI S3 backed AMI
EBS backed AMI S3 backed AMIEBS backed AMI S3 backed AMI EBS backed AMI S3 backed AMIEBS backed AMI S3 backed AMI
AP... US West 1 US East 1
33. • 2 architectures
• 2 virtualisation types
• 2 root volume types
• 11 Regions
• 3 Debian releases
= 198 images
(Plus images currently being end-of-lifed,
experimented with, and used for other purposes)
34. Current Debian AMIs: Squeeze (6)
Architecture EBS Backed S3 Backed
32 bit PVM Yes
64 bit PVM Yes
32 bit HVM
64 bit HVM
35. Current Debian AMIs: Wheezy (7)
Architecture EBS Backed S3 Backed
32 bit PVM Yes
64 bit PVM Yes Yes
32 bit HVM
64 bit HVM Yes (experimental)
36. Future Debian AMIs: Jessie (8)
Architecture EBS Backed S3 Backed
32 bit PVM
64 bit PVM Yes
32 bit HVM
64 bit HVM Yes Yes*
37. Two ways of creating AMIs
Start from scratch
• Uses a fresh, blank volume,
install as a debootstrap
Update existing
• Start existing instance,
customise, create new
image
38. EBS Backed AMI overview
instance
volume
/
volume
/target
snapshot
EC2 API
Endpoint
AMI
39. Let’s create a Jessie image
• Fire up an existing instance (easiest is to use
an existing Debian AMI)
• Install git, debootstrap, python-boto, python-jsonschema,
and some other python bits
– Configure your AWS IAM credentials for boto
• Grab bootstrap-vz from Github
43. Debian AWS Accounts
Region AWS Account ID
Beijing 673060587306*
Gov Cloud 256493402735**
Standard Regions 379101102735
45. Community Shared AMIs
• Un-vetted by AWS
– Trojan horses
– Left over SSH keys in other accounts
– Cron jobs that go bump in the night
• Anyone can share any AMI under their control
(provided they have access within their AWS account to do so – IAM Policy)
– Caveat emptor
48. Pushing images to Marketplace
Vendor
AWS
Account ID
Vendor
Display
Name
Product ID Version ID ASIN SKU Software
by
Title Version
Title
Release
Notes
Short
Description
Description Highlight1
51. AMI Lifecycle
Our aim is to keep the final point release AMI
available for each Debian major release,
starting from Squeeze:
• 6.0.10
• 7.7
52. AMI Lifecycle
Wheezy 7.4
Wheezy 7.5
Try to keep a 2 – 5
week overlap for point
releases, then un-share
Wheezy 7.6
for a period,
Wheezy
7.6.aws.
1
Wheezy
7.6.aws.2
Wheezy 7.7
then delete
Time
Occasionally security
releases that are urgent
in BASE images (AMIs)
force additional version
numbers out of step with
Debian. This was
shellshock,
53. Security in base images
• EC2 instances may be deployed such that they
don’t have direct access to fetch updates
• Administrators may chose not to install
updates unattended
55. Workflow overview
1. Generate AMIs in US East 1
2. Tag AMIs and Snapshot
3. Test image in US East 1
4. Copy to all Standard Regions (python script)
5. Mark AMI and Snapshot as Public (python script)
6. Generate in Beijing and Gov Cloud, tag, mark public
7. Generate signed message to the Debian-cloud mailing list, update wiki
8. Wait a few days (for bugs to surface), then push to AWS Marketplace
9. Announce deprecation of previous versions (typically 3 – 5 weeks notice)
in signed email to Debian-cloud ML
10. After elapsed period, remove public sharing from AMI and Snapshots
(python script)
11. A day or so later, deregister the AMI and delete the snapshot (python
script)
56. What’s new in Jessie EC2 images
• Single Root IO Virtualisation (Enhanced
Networking)
• Multiple Network Interfaces (ENI)
• Multiple sub-interfaces
• AWS CLI and python-boto installed in base
image
• Cloud-init (since Wheezy 7.4)
57. Cloud-init
• Insert this as “User
Data”
• Can be embedded into
CloudFormation
templates
#cloud-config
package_update: true
package_upgrade: true
package_reboot_if_required: true
packages:
- pwgen
- less
locale: fr_FR.UTF-8
ssh_authorized_keys:
- ssh-rsa AAAAB3Nz....89dGp5 me@mykey1
- ssh-rsa AAAAB3Nz....89dGp5 me@mykey2
final_message: "The system is finally up,
after $UPTIME seconds"
59. Debian Archive via CDN
• Default apt sources.list
for EC2 images uses
cloudfront.debian.net
• Primarily for EC2
instances, but is active
in all 52 Cloudfront
locations world-wide
CloudFront
60. Cloudfront.debian.net
• Each edge location is
independent of all
others
edge location
edge location
edge location
traditional server
61. Cloudfront.debian.net
• However, Debian HTTP
servers don’t put any
cache advisory headers
on how long objects
(files) may be cached
for; some of these are
quite volatile, and some
are very stable
edge location
edge location
edge location
traditional server
62. Cloudfront.debian.net
• Luickly, Cloudfront
supports “Cache
behaviours”, mapping
different URL paths to
alternate origin servers
edge location
edge location
edge location
traditional server