AWS Concepts – S3

First of all, what does S3 stand for? The answer is Simple Storage Service which was launched by AWS in March 2006, so it’s over 13 years old at the time of writing this blog post!

S3 provides developers and IT teams with a secure, durable and highly scalable object storage. Examples of object storage are flat files such as word documents, PDFs, text documents etc.

S3 allows you to securely upload and control access to files from 0 bytes to 5TB into a ‘Bucket’ which is essentially a folder. The ‘Bucket’ requires a globally unique namespace across S3. AWS claim that S3 provides unlimited storage (not sure if anyone would or could test this claim!).

Data Consistency

S3 uses the concept of read after write consistency for new objects. In a nutshell this means when the object is committed to S3, you can read it.

However when you are updating or deleting the object you receive eventual consistency. So if you have a file which you have just updated and you access it immediately, you may get the old version, wait a few seconds and you will receive the new version.

S3 The Basics

S3 is built for 99.99% availability, however AWS only provides a 99.9% availability SLA see here. Which means you could have just over 8 hours 40 minutes of acceptable downtime per annum.

Think about the impact of AWS S3 availability SLA on any designs

Amazon provides a 99.999999999% (11x9s) durability SLA which means data won’t be lost. However does this same guarantee apply to data integrity?

S3 provides the option for tiered storage with lifecycle management, for example if an object is older than 30 days move it to a lower cost/class tier.

Outside of this, S3 allows versioning of objects, encryption and policies such as to delete an object you have to use MFA.

S3 Storage Tiers/Classes

At the time of writing this blog post, S3 offers six different classes of storage. Each of these has it’s own availability SLA, differing levels of costs and latency.

It’s easiest to explain this in the table below which is taken from AWS.

If we look at all of the different storage classes except for S3 One-Zone IA they can suffer the loss of two availability zones. Whereas only S3 Standard and S3 Intelligent-Tiering includes data retrieval fees.

Lastly if we examine ‘first byte latency’ this determines how quickly you will be able to access any objects.

Replication

When you create an S3 ‘Bucket’ you are able to replicate this from one AWS region to another, providing either high availability or disaster recovery (dependent on your configuration). This can also be coupled with S3 Transfer Acceleration which uses CloudFront edge locations. The benefits this provides are that users upload files to the closest of AWS’s 176 edge location which are then transferred securely to the S3 Bucket.

The diagram below provides a logical overview of this process.

A few things to note about S3 Cross Region Replication:

  • Only new objects and changed objects are replication, not existing objects
  • Cross Region Replication requires versioning to be enabled
  • The target bucket can use a different storage class
  • The target bucket receives the same permissions as the source bucket
  • Deletions and deletion markers are not replicated from the source to target region

Security

S3 security is achieved by a number of elements, which include encryption in transit using TLS and at rest when the data is committed to the S3 bucket.

Dependent on how a business operates it may choose to either handle encryption at rest locally and then upload the encrypted object to the S3 bucket or trust AWS to encrypt the object within the bucket. This process is managed by keys to encrypt and decrypt the object.

  • S3 Managed Keys – Built Into S3, managed by AWS
  • AS Key Management Service – Managed by both AWS and the customer
  • Server Side Encryption with Customer Keys – Customer provides the keys

By default a bucket is private e.g. the objects within this are not accessible over the internet. You can apply access policies to a bucket either on the resource directly or via a user policy.

AWS Concepts – Billing Alarms

So you have set up your first AWS Account and want to make sure that the costs don’t spiral out of control? This quick guide will help you put things in place to ensure that you are notified when costs exceed a predefined limit.

First of all select your Organisation > My Billing Dashboard and then Billing Preferences

Next tick ‘Receive Free Tier Usage Alerts’ and enter an email address along with ticking ‘Receive Billing Alerts’. Finally click on Save Preferences.

This is the initial configuration at account level on how we want our billing preferences to be set. Now we need to use CloudWatch to send an alert when a metric his hit.

Select Services > CloudWatch. Then Select Alarms > Create Alarm.

Select Metric > Billing and then Total Estimated Charge > Currency USD > Select Metric, as shown below.

We now need to specify our metric conditions. Using a ‘Maximum’ value over a period of ‘6 Hours’ when it is Greater than 10 USD….do something!

In the notification screen select ‘in Alarm’ which essentially means when the alert is triggered, do something. We want that something to be a Simple Notification Service (SNS). Select NotifyMe. Finally we now need to click on the SNS Console to setup and verify the email address.

Within the SNS Dashboard, we should have a Topic named ‘NotifyMe’ without a Subscription assigned to it. Click ‘Create Subscription’

Select Protocol > Email and the Endpoint, in this example it’s craig@vmfocus.com. Finally select Create Subscription.

Before the alert subscription goes live, we need to confirm that we have access to the email endpoint. Check you inbox (or spam). Once confirmed you should receive a verification such as this.

Double check your Subscription in SNS to verify the status.

Back into the CloudWatch dashboard and if we remove and re-add NotifyMe, we should see the email (endpoint) change to our verified address.

Lets give the alarm a unique name. I have chosen BillingAlarm with the description Greater than 10 USD.

Finally click ‘Create Alarm’ and verify the details. Voila we will receive an email alert when our expenditure is over 10USD within a six hour window.

AWS Concepts – Identity & Access Management

When we talk about Identity and Access (IAM) what do we really mean? For me, it boils down to who you are and what you are entitled to access.

Simple statement, but it does start to get rather complicated when you think about Identity Management.

If you think about your own organisation what directory service does it use? Probably Active Directory Domain Services (AD DS). Think about how many years it has it been fine tuned with integration with third party solutions such as MFA, SSO and VPNs.

The list does truly go on and on. Most organisations will treat their on-premises Active Directory Domain Services (AD DS) as the one source of all truth for users, groups, permissions and passwords.

So the question is how does AWS deal with IAM?

What Is AWS IAM?

It is AWS hyperscale web service that allows users and services shared access to your AWS account. It uses an eventually consistent model, which in a nutshell means that changes are not immediately available.

Users are authenticated and then authorised to use AWS services. To ease the management of individual users, groups are used. Policies are applied to groups which then dictate what the user or service can do.

Policies are JSON documents, used to define actions, effect, resources and conditions on what can be evoked for example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": "*",
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": "192.168.1.0/24"
                }
            }
        }
    ]
}

When you create an IAM user, they can’t access anything until you give them permission.

It should be noted that actions or resources which are not explicitly allowed are denied by default.

We also have IAM Roles, which are similar to users but are AWS identities with permissions (JSON policy) which determines what can or can’t do. It is important to note that IAM Roles don’t have any passwords or long terms access credentials. Access keys are created dynamically and provided on a temporary basis. Typically they are used to delegate access to applications or services.

IAM Roles can also be used to provide users with enhanced privileges on a temporary basis for example a user requires occasional admin access to an S3 bucket.

To enable policies to be tested before you apply them into production, AWS have a handy policy simulator which can be found here.

Identity Federation

AWS IAM supports identity federation for delegated access to either the AWS Management Console or APIs. Federated users are created within your corporate directory outside of the AWS account.

These can be web identity providers such as Amazon, FaceBook, Google or an OpenID Connect provider.

Within the enterprise world, we tend to see Active Directory Domain Services used in conjunction with Active Directory Federation Services. AWS have integration using Security Assertion Markup Language 2.0 (SAML 2.0) using STS AssumeRoleWith SAML.

A high level overview of this is shown below in the diagram below.

  1. User browsers to URL and is redirected to AD FS sign in page
  2. User enters Active Directory credentials
  3. User authenticated by Active Directory Domain Services
  4. Users browser receives a SAML 2.0 assertion
  5. User browser posts the SAML 2.0 assertion to AWS STS
  6. AssumeRoleWithSAML requests temporary security credentials and constructs a sign in URL for the AWS Management Console
  7. User browser receives the sign in URL and is redirected to the AWS Management Console

Single Sign On Cloud Applications

To provide easier integration with popular cloud applications
such as Dropbox, Office365 and SalesForce. AWS provide single sign on (SSO) using SAML 2.0via a configuration wizard.

Further information can be found here.

MultiFactor Authentication

AWS MFA provides an extra later of security to reduce the overall risk of compromised credentials. Providing a secondary authentication step for Management Console and API users.

For the MFA device, you have a choice of three items:

  1. Virtual MFA Device
  2. Hardware Key
  3. Hardware Device

This link shows which form factors can be used across devices.

Final Thoughts

AWS IAM is a web scale directory that can provides integration with on-premises directory services and cloud applications. Interestingly this is an added value service with no extra cost, which is a different approach from traditional licensing vendors.

Azure CDN: Custom Cache Rules

It was just over a couple of years ago when I wrote the Azure CDN Concept blog post.

I was recently asked by a customer to apply caching rules to only a specific set of file extensions using a custom domain name.  So with this in mind, I thought I would share the process with you.

Step 1 – Which CDN?

Microsoft Azure provides a number of CDN, so we need to find the correct CDN to meet requirements which are custom caching rules and custom domain HTTPS.

Looking at the Compare Azure CDN Product Features page it shows that only Standard Verizon and Premium Verizon will meet the requirements.

In this case, I will start by using Standard Verizon, we can migrate to Premium Verizon if needed.

Step 2 – Caching Rules

Azure CDN uses the HTTP caching specialisation RFC 7234.  It should be noted that not all resources can be cached in particular Standard Verizon only deals with:

  • HTTP Status Codes 200
  • HTTP Methods GET
  • File Size Limits 300GB

By default Standard Verizon caches any HTTP Status 200 Codes for 7 days.  To override this, we need to enable Global Caching Rules which affect the caching behaviour for all requests.

In this case we want to set the caching behaviour to ‘Bypass Cache’ meaning that no content which will be cached.

Next we then set our specific Custom Caching Rules which supersede the Global Caching Rules using File Extension types for example:

We are now utilising the Standard Verizon CDN to only cache jpg, jpeg, png and gif file extensions.

Final Thought

In a nutshell Custom Caching Rules override, Global Caching Rules which override Default Caching Rules.

Think of it like a game of top trumps, for those of you who don’t know what this is, I would suggest adding a pack to your Christmas list!

VMware on AWS My Thoughts

vmware-and-amazon-web-services-extending-vmware-into-aws-1As VMworld 2017 has just finished I have been giving VMware on AWS some thought.  Lot’s of questions have been running through my head, so I thought I would try and transcribe some here.

What Is It?

It’s a minimum of 4 x of vSphere Hosts running VMware’s SDDC (ESXi, NSX and vSAN) which is dedicated to a customer.  VMware manage the availability, patching and maintenance whilst the customer consumes the resources.

Each ESXi Host provides 36 x CPU Cores, 512GB RAM and 8 NVMe drives.  Some of this space is dedicated to management items such as vCenter and NSX VM’s so overall usable resources will be less.

Why Would I Use It?

This is a question I have been pondering on, my initial thoughts are:

  • A customers infrastructure lifecycle is at the point of refresh and they are moving to an ‘opex model’
  • A customer needs to exit a datacentre quickly and this could be one of a number of options
  • A customer is deploying a remote office and doesn’t want to invest in on-premises infrastructure for their VM estate
  • Target for disaster recovery to reduce on-premises secondary datacentre footprint (not sure if SRM is supported yet)

Even though I’m not convinced by this one, a potential candidate for a use case is to extend your on-premises operational model to AWS.

Another one which I’m not convinced by is reducing your on-premises operational costs by having someone else manage maintenance by patching your storage, ESXi Hosts and vCenter.  Are companies really going to make Dave redundant? Nope they are just going to get Dave doing something different for that one day a month (or Dave gets to chill out).

Would I Recommend It?

The concise answer is potentially.  The customers that I work with are reviewing their application estate and looking to either keep, kill, consolidate or transform them.

  • The keep category often fall into ‘that’s too difficult to tackle basket’ or we have only just invested in a new application or release
  • Kill generally means that the application will be ‘withered on the vine’
  • Consolidate generally means a number of applications will be collapsed into a single master
  • Transform usually means from on-premises to a SaaS type offering for example Exchange On-Premises to Office 365 Exchange Online

Out of these, which are the use cases for VMware on AWS?  The answer is simple anything heritage AKA Virtual Machine, as PaaS and SaaS will go somewhere else.

Infrastructure Applications such as Active Directory Domain Services, Certificate Services, File, Print and SQL are either highly available natively or can be designed and deployed on IaaS in a highly available fashion and as such aren’t great candidates for VMware on AWS.

Whats The Cost?

The monthly cost of an one year reserved ESXi Host (30% discount) is $4,332.00 of which we need four which makes the monthly cost roughly $17,328.00 which is circa £13,500 per month or £162,000 per year for compute and storage.  Note network charges and Operating System licenses are not included.

Using the same 30% discount level on Microsoft Azure you could run:

  • 268 x A2 v2 VM continuously for 12 months
  • 143 x D2 v2 VM continuously for 12 months

Taking into account that a single ESXi Host is used for tolerate failures.  We have 1,536GB of RAM minus circa 10% of management cluster and general overhead gives circa 1,382GB of useable RAM.

Using the same RAM metrics as the above Azure VM’s you could run the equivalent of:

  • 346 x A2 VM’s using VMware on AWS
  • 197 x D2 VM’s using VMware on AWS

Final Thought

Generally I’m seeing customers moving to a PaaS or SaaS offering for low hanging fruit and then dealing with the more complex applications on a case by case basis with a view to transforming these into a PaaS or SaaS model.

If customers are migrating 100 plus heritage VM’s to a cloud platform and they cannot be re-architected to be natively highly or have an SLA that simple backup and restore routines will not cater for then VMware on AWS is a viable option.

I do see that VMware on AWS has a place in the market, however the place is for heritage systems and I wonder how long it will be until the earnings from VMware on AWS start to dwindle?