Azure

I was set an interesting challenge by a customer to copy the data in their Production Subscription Azure File Shares into their Development Subscription Azure File Shares. The reason behind this was to ensure that any uploads to their Production environment are kept inline with the Development environment, enabling testing to be performed on ‘live’ data.

The customer wanted something which was easy to manage, which provided visibility of data movement tasks within the Azure Portal without needing to manage and maintain PowerShell scripts.

The answer to this was Azure Data Factory.

What Is Azure Data Factory?

Azure Data Factory is a managed data integration service that enables data driven workflows between either on-premises to public cloud or within public clouds.

Pipelines

A pipeline is a logical grouping of activities that together perform a task.  The activities within the pipeline define actions to perform on data. 

Data Factory supports three types of activities data movement activities, data transformation activities and control activities. In this use case, data movement activities will be used to copy data from the source data store to the destination data sink.

Linked Service

Linked Services are used to link data stores to the Azure Data Factory.   With the ‘data set’ representing the structure of the data and the linked service defining the connection to the external data source. The diagram below provides a logical overview of this.

Integration Runtime

For copy activities an integration runtime is required to determine the source and sink linked services to define the direction of data flow.  To ensure data locality a custom integration runtime will be used within West Europe.

Source Datastore

Each file share within the vmfwepsts001 Storage Account is an individual linked service.  Therefore, four source linked services will be defined for data, documents, images and videos.

Sink Datastore

Each destination file share within the vmfwedsts001 Storage Account is an individual linked service.  Therefore, four source linked services will be defined for data, documents, images and videos.

Copy behaviour to the sink datastore can be undertaken using three methods:

• Preserve Hierarchy the relative path of source file to source folder is identical to the relative path of the target file and folder
• Flatten Hierarchy all files from the source folder are in the first level of target folder.  The target files have auto generated names
• Merge Files merges all files from the source folder to one file, using an auto generated name

To maintain the file and folder structure, preserve hierarchy copy behaviour will be used.

Tune in for the next blog post when we will cover the configuration settings.

As you would expect quite a few changes within the world of Microsoft Azure since my last update in March 2018.

So without further delay, below is my take on the heavy hitters which have been announced.

Azure Database for MySQL and PostgreSQL

Generally availability was a announced in April 2018, which essentially means that for customers using either database service are no longer running at risk and are now protected by an SLA of 99.99%

Read more here.

Integration of Azure Backup into VM Create Experience

OK, this is quite a small thing but when you have deployed oodles of VMs and you have to make sure you undertake backup configuration separately, can lead to a bit of frustrating.

Quite pleased Microsoft have introduced this as it makes for an overall better VM creation workflow.

Azure Service Health

Put simply being able to see the affect of any Azure outages on your resources rather than a generic status update is a great step forward.

Read more here.

Azure Virtual Machine Serial Console

Wondering why your VM won’t power on, or is hung without knowing the status.  Well we finally have an answer thanks to Serial Console access.

Makes the diagnosis of issues far easier and transparent.  Read more here.

Application Security Groups

This makes managing Network Security Groups at scale far easier, no longer are we reliant on IP Addresses, we can group network interfaces into Application Security Groups to govern traffic flow.

Read more here.

Security Centre Overhaul

I have to admit it, Security Centre wasn’t the most pleasant place to be.  Microsoft have taken this on-board and given the UI an overhaul.  Now it’s easier to understand, find and remediate issues within your Azure environment.

Azure Firewall

When I read into Azure Firewall, it wasn’t the big news as I was expecting, more so a v0.1 release from Microsoft.

In a nutshell, it monitors interesting outbound traffic.  So a step in the right direction, but more to come soon hopefully.

Read more here.

Azure Management Groups

For those of us that work across customers with multiple subscriptions this makes life so much easier!

Quite a simple thing, but again a great time sizer.  Read more here.

Encrypted ExpressRoute Microsoft Peering

This was announced under the radar so to speak.  We now have the ability to natively encrypt ExpressRoute Microsoft Peering traffic to access PaaS services within Azure.

So if you are a department with a requirement to use Azure PaaS securely, this will likely be the solution you are looking for.