Storage Spaces Direct Overview

Storage Spaces Direct is an area which I have been meaning to look into, but for one reason or another it has slipped through the gaps until now.

What Is Storage Spaces Direct

Storage Spaces Direct is a shared nothing software defined storage which is part of the Windows Server 2016 operating system.  It creates a pool of storage by using local hard drives from a collection (two or more) individual servers.

The storage pool is used to create volumes which have in built resilience, so if a server or hard drive fails, data remains online and accessible.

What Is The Secret Sauce?

The secret sauce is within the ‘storage bus’ which is essentially the transport layer that provides the interaction between the physical disks across the network using SMB3. It allows each of the Hosts to see all disks as if they where it’s own local disk using Cluster Ports and Cluster Block Filter.

The Cluster Ports is like an initiator in iSCSI terms and Cluster Block Filter is the target, this allows each disk to presented to each Host as if it was it’s own.

Storage Bus v0.1

For a Microsoft supported platform you will need a 10GbE network with RDMA compliant HBA’s with either iWARP or RoCE for the Storage Bus.

Disks

When it comes to Storage Spaces Direct, all disks are not equal and you have a number of disk configurations which can be used.   Drive choices are as follows:

  • All Flash NVMe
  • All Flash SSD
  • NVMe for Cache and SSD for Capacity (Writes are cached and Reads are not Cached)
  • NVMe for Cache and HDD for Capacity
  • SSD for Cache and HDD for Capacity (could look at using more expensive SSD for cache and cheaper SSD for capacity)
  • NVMe for Cache and SSD and HDD for Capacity

In a SSD and HDD configuration the Storage Bus Layer Cache binds SSD to HDD to create a read/write cache.

Using NVMe based drives will provide circa 3 x times performance at typically 50% lower CPU cycles versus SSD, but come at a far greater cost point.

It should be notes that as a minimum 2 x SSD and 4 x HDD are needed for a supported Microsoft configuration.

Hardware

In relation to the hardware it must be on Windows Server Catalog and Certified for Windows Server 2016.  Both HPE DL380 Gen10 and Gen9 are supported along with HPE DL360 Gen10 and Gen9.  When deploying Storage Spaces Direct you need to ensure that the Cluster creation passes all validate tests to be supported by Microsoft.

  • All servers need to be the same make and model
  • Minimum of Intel Nehalem process
  • 4GB of RAM per TB of cache drive capacity on each server to store metadata e.g. 2 x 1TB SSD per Server then 8GB of RAM dedicated to Storage Spaces Direct
  • 2 x NICS that are RDMA capable with either iWARP or RoCE dedicated to the Storage Bus.
  • All servers must have the same drive configuration (type, size and firmware)
  • SSDs must have power loss protection (enterprise grade)
  • Simple pass through SAS HBA for SAS and SATA drives

Things to Note

  • The cache layer is completely consumed by Cluster Shared Volume and is not available to store data on
  • Microsoft recommendation is to make the cache drives a multiplier of capacity drives e.g. 2 x SSD per server then either 4 x HDD or 6 x HDD PER SERVER
  • Microsoft recommends a single Storage Pool per cluster e.g. all the disks across A 4 x Hyper-V Hosts contribute to a single Storage Pool
  • For a 2 x Server deployment the only resilience choice is a two way mirror.  Essentially data is written to two different HDD in two different servers, meaning your capacity layer is reduced by 50%.
  • For a 3 + Server deployment Microsoft recommends a three way mirror.  Essentially three copies of data across 3 x HDD on 3 x Servers reducing capacity to 33%.  You can undertake single parity (ALA RAID5) but Microsoft do not recommend this.
  • Typically a 10% cache to capacity scenario is recommended e.g. 4 x 4TB SSD is 16TB capacity then 2 x 800GB SSD should be used.
  • When the Storage Pool is configured Microsoft recommend leaving 1 x HDD worth of capacity for immediate in-place rebuilds of failed drives.  So with 4 x 4TB you would leave 4TB un allocated in reserve
  • Recommendation is to limit storage capacity per server to 100TB, to reduce resync of data after downtime, reboots or updates
  • Microsoft recommends using ReFS for Storage Spaces Direct for performance accelerations and built in protection against data corruption, however it does not support de-duplication yet.  See more details here https://docs.microsoft.com/en-us/windows-server/storage/refs/refs-overview

Azure Announcements September 2017

We are only two days into Microsoft Ignite and I thought I would share the announcements which I believe will become ‘heavy hitters’  in the near future.

Planned Maintenance (Preview)

One of the biggest deal breakers when migrating to public cloud, is the shear amount of single instance VMs in a customer estate which relay upon infrastructure availability to meet business SLA’s.   The cost to translate this into cloud native applications to place them into an Availability Group to receive an SLA from Microsoft and to minimise the impact of planned maintenance is often too burdensome, so they are left to wither on the vine on-premises.

Microsoft have recognised this issue and have announced ‘Planned Maintenance’ which means that you will be notified of when maintenance is going to occur and you will have up to four weeks to schedule in a reboot of your virtual machine.

This is a game changer for customers, and I would encourage you to read more here.

Azure Migrate (Preview)

To start the journey to public cloud services, you need to understand your application estate.  This is a process which should not be under estimated as many customer environments are poorly documented, application owners have left the business, operations and IT don’t really understand how an application is coupled together so trying to migrate anything but low hanging fruit often gets placed into the ‘too hard to deal with bucket’.

To counter act this, Microsoft have announced Azure Migrate which uses an application based approach for the following:

  • Discovery and assessment for on-premises virtual machines
  • Inbuilt dependency mapping for high-confidence discovery of multi-tier applications
  • Intelligent rightsizing to Azure virtual machines
  • Compatibility reporting with guidelines for remediating potential issues
  • Integration with Azure Database Management Service for database discovery and migration

I wondering if this will be a PaaS offering of Microsoft Assessment and Planning Toolkit? Any how read more here.

Azure File Sync (Preview)

You would have thought with the advent of SharePoint and OneDrive for Business that the traditional file server would be on the way out,  however file storage still continues to be an issue for many companies.  Microsoft have announced Azure File Sync which enables you to replicate file data across the globe and tier data from on-premises to Microsoft Azure without a StoreSimple device.

When more details are announced, I will be interested to understand how Microsoft deal with file locking and if this will be dealt with using Optimistic Concurrency, Pessimistic Concurrent or Last Writer Wins.  Also backup of data needs to be addressed as well.

For more information see here.

Azure DDoS Protection Service (Preview)

Security is always a hot topic when discussing public cloud services, figuring how you protect the ‘crown jewels’ is difficult and can be difficult to get Information Security Risk officers to agree on your approach.

To counter act this Microsoft have announced Azure DDoS Protection Service which in a nutshell protects a virtual network and everything behind it.  The service understands your normal application traffic profiles using machine learning and detects malicious traffic attacks.  Azure DDoS Protection can also be combined with Web Application Firewalls to provide protection from:

  • Request rate-limiting
  • HTTP Protocol Violations
  • HTTP Protocol Anomalies
  • SQL Injection
  • Cross site scripting

For more information see here.

 

End Of A VCP Era

No VCPIt was back in November 2011 when I first obtained the VCP qualification, I remember how excited I was when I passed this exam.  I was designing and installing vSphere 4.x on HPE hardware and this was a string in my bow that validated my expertise.

Move forward to September 2017 and the IT industry has evolved.  The skills required to design and deploy vSphere environments are common place and the demand that I see from customers isn’t around architecting a vSphere platform.

A customer might require some assistance to migrate to the target platform, but when they are consuming it, where does the next the next requirement come from?

Throughout my IT career, I have always built my knowledge based on what I believe is credible within the market place.  This hasn’t ever been from a technical perspective, rather a business point of view.

When studying for certifications, you are taking time away from family, friends and generally doing the things you enjoy outside of work.  When I make this commitment, I want to use my time efficiently on what I believe will yield the highest reward for the least investment.

With the above in mind, it is with a heavy heart that I will let me VCP expire this month.  The reason I say heavy heart is that I have so many found memories of not only of using the cool technology that VMware have provided, but the people I have worked with designing and deploying vSphere solutions not only on a professional level but also across the community.

As they say when one chapter closes another begins, watch this space!

What’s This GDPR Thing All About?

GDPRThis blog post is certainly a bit different from my usual repertoire, however I strongly encourage any ‘Architect’ level person to have a good grasp of the implications to their customers.

Overview

In it’s essence General Data Protection Regulation (GDPR) strengthens the rights of individuals in the European Union (EU) to control their personal data.  This places emphasis on businesses ensure they have adequate privacy and data protection measures whilst protecting personal data by design and default.

Individuals will have significantly enhanced rights, such as to access and receive a copy of their personal data, as well as to have it erased.

It should be noted that companies who violate the GDPR could face fines up to the greater of €20 million or 4% of annual global turnover (revenue), whichever is greater. GDPR actually become law in the EU on 27 April 2016 but given the significant changes faced, a two year transition was granted, which means that enforcement begins on 25 May 2018.

For more GDPR information visit the Information Commissioners Office.

Key Concepts

GDRP is structured around six principles which are:

  • Requiring transparency on the handling and use of personal data.
  • Limiting personal data processing to specified, legitimate purposes.
  • Limiting personal data collection and storage to intended purposes.
  • Enabling individuals to correct or request deletion of their personal data.
  • Limiting the storage of personally identifiable data for only as long as necessary for its intended purpose.
  • Ensuring personal data is protected using appropriate security practices.

Key Changes

Microsoft have broken down the key changes to GDPR under four key verticals which are:

  1. Personal Privacy – Individuals have the right to :
    • Access their personal data
    • Correct errors in their personal data
    • Erase their personal data
    • Object to processing of their personal data
    • Export personal data
  2. Controls & Notifications – Companies will need to:
    • Protect personal data using appropriate security
    • Notify authorities of personal data breaches within 72 hours
    • Obtain appropriate consents for processing data
    • Keep records detailing data processing
  3. Transparent Policies – Companies are required to:
    • Provide clear notice of data collection
    • Outline processing purposes and use cases
    • Define data retention and deletion policies
  4. IT & Training – Companies will need to:
    • Train privacy personnel and employees
    • Audit and update data policies
    • Create and manage compliant vendor contracts

Challenges

Let’s have a look at a scenario around the starters and leavers process.  Bob leaves the employment of VMFocus after 10 years service and is moving onto pastures new.  Bob requests that VMFocus erases his personal data, what could this effect?

  • Active Directory credentials as most are personally identifiable including device logins
  • Line of business applications which hold user specific credentials
  • Database records
  • Email records
  • ERP systems
  • File systems
  • HR records

All of the above items are fairly straight forward to erase data, however thought needs to be given to the order of data removal as a significant number of applications are Active Directory integrated to provide either Same Sign-On or Single Sign-On.

Outside of this, real consideration needs to be given to the 10 years worth of backups that hold Bob’s details, how does VMFocus go about restoring information of tapes and then deleting this?  What are the implications in man hours to undertake such as task?

Three months after Bob requesting his personal data is removed, VMFocus is scrutinised and a specific client file and email that Bob was working on needs to be accessed.  How does VMFocus go about providing these details?

From a technical perspective, these are some of the questions we need to start thinking about answering.  The next few months are going to be very interesting and I expect we are going to see a few companies on the news with GDPR breaches.

Microsoft Azure Enterprise Cost Management

azureMicrosoft have announced the preview of Enterprise Cost Management for Azure, which is great news for Enterprise Agreement customers.

Until now gaining visibility of spend on an Azure Enterprise Agreement has been difficult to manage even when combined with Tags and Resource Groups.

It should also be noted that an Enterprise Agreement doesn’t provide spending limits (see offer details), quotas or even billing alerts (see prevent unexpected costs) so customers are often wary of migrating services to Microsoft Azure and/or providing access to their Azure Portals due to fear of being stung by large bills.

It is understandable that Microsoft do not want to ‘turn off’ customers workloads, however their could be a case for this in a development environment where a person leaves a ‘monster VM’ up and running of a month by mistake.

This is a step in the right direction, hopefully we will see billing alerts added in the not to distant future.