Using Azure Data Factory to Copy Data Between Azure File Shares – Part 1

I was set an interesting challenge by a customer to copy the data in their Production Subscription Azure File Shares into their Development Subscription Azure File Shares. The reason behind this was to ensure that any uploads to their Production environment are kept inline with the Development environment, enabling testing to be performed on ‘live’ data.

The customer wanted something which was easy to manage, which provided visibility of data movement tasks within the Azure Portal without needing to manage and maintain PowerShell scripts.

The answer to this was Azure Data Factory.

What Is Azure Data Factory?

Azure Data Factory is a managed data integration service that enables data driven workflows between either on-premises to public cloud or within public clouds.

Pipelines

A pipeline is a logical grouping of activities that together perform a task.  The activities within the pipeline define actions to perform on data. 

Data Factory supports three types of activities data movement activities, data transformation activities and control activities. In this use case, data movement activities will be used to copy data from the source data store to the destination data sink.

Linked Service

Linked Services are used to link data stores to the Azure Data Factory.   With the ‘data set’ representing the structure of the data and the linked service defining the connection to the external data source. The diagram below provides a logical overview of this.

Integration Runtime

For copy activities an integration runtime is required to determine the source and sink linked services to define the direction of data flow.  To ensure data locality a custom integration runtime will be used within West Europe.

Source Datastore

Each file share within the vmfwepsts001 Storage Account is an individual linked service.  Therefore, four source linked services will be defined for data, documents, images and videos.

Sink Datastore

Each destination file share within the vmfwedsts001 Storage Account is an individual linked service.  Therefore, four source linked services will be defined for data, documents, images and videos.

Copy behaviour to the sink datastore can be undertaken using three methods:

  • Preserve Hierarchy the relative path of source file to source folder is identical to the relative path of the target file and folder
  • Flatten Hierarchy all files from the source folder are in the first level of target folder.  The target files have auto generated names
  • Merge Files merges all files from the source folder to one file, using an auto generated name

To maintain the file and folder structure, preserve hierarchy copy behaviour will be used.

Tune in for the next blog post when we will cover the configuration settings.

Standard SSD: Azure Backup Failure

imagesI have been undertaking a customer deployment and thought I would share this nugget of information which may save you some time.

Standard SSD

Even though Standard SSD are now GA as per this article.  We are unable to backup VMs with Standard SSD, receiving in total two error messages.

The first error message is the initial job to configure the backup fails with the message ‘Deployment to resource group ‘name’ failed.  Additional details from the underlying API that might be helpful: At least one resource deployment operation failed.  Please list deployment operations for details.  Please see https://aka.ms/arm-debug for usage details.

Azure Backup 01

Digging a bit deeper we receive the Error Code ‘UserErrorGuestAgentStatusUnavailble’ with a recommended action of ‘Ensure the VM has network connectivity and the VM agent is update and running.  For more information, please refer to https://aka.ms/guestagent-status-unavailable’.

A quick reboot of the VM and this resolves the initial ‘configure backup error’ with Standard SSDs.

We then go to protect the VM and undertake the initial backup and this is where the problem occurs.  After two plus hours, you will receive an error notification which states ‘The storage type is not supported by Azure Backup’.

Azure Backup 02

This is a known issue and is documented in the ‘Prepare Resource Manager Deployed VMs‘ article under the section ‘Limitations when backing up and restoring a VM’.

So for now, you can deploy VMs with Standard SSD but you can’t back up the entire VM using Azure IaaS VM Backup!

Update

Azure Backup now supports Standard SSD see blog post here.

 

Application Gateway WAF, does it Load Balance?

images

I was recently working on a project in which we where using an Application Gateway with WAF to send traffic to certain destinations based on URL path.

During a conference call with the application developer and a Microsoft Cloud Solution Architect I was asked the question, what are you going to use to load balance the backend pools?

I initially responded the WAF as this is polling the backend pool to determine which VMs to send traffic to, so logically should include a Load Balancer, but hold on a minute I have never seen any settings for Load Balance rules.  In comes that moment of doubt when someone from Microsoft questions you.

After trawling over the documents, I was able to find reference to load balancing on the main product overview along with internal load balance configuration, but what about external connections?

I was able to find this golden nugget of information, written by David Sanchez entitled Azure Application Gateway uses the Load Balancer.  This confirms that it is a built in to the Application Gateway by default using an algorithm to provide load balance services.

So in short, yes the Application Gateway WAF, does include a Load Balancer, it is just inbuilt and therefore shielded from configuration choices.

 

Azure Announcements August 2018

azure

As you would expect quite a few changes within the world of Microsoft Azure since my last update in March 2018.

So without further delay, below is my take on the heavy hitters which have been announced.

Azure Database for MySQL and PostgreSQL

Generally availability was a announced in April 2018, which essentially means that for customers using either database service are no longer running at risk and are now protected by an SLA of 99.99%

Read more here.

Integration of Azure Backup into VM Create Experience

OK, this is quite a small thing but when you have deployed oodles of VMs and you have to make sure you undertake backup configuration separately, can lead to a bit of frustrating.

Quite pleased Microsoft have introduced this as it makes for an overall better VM creation workflow.

Azure Service Health

Put simply being able to see the affect of any Azure outages on your resources rather than a generic status update is a great step forward.

Read more here.

Azure Virtual Machine Serial Console

Wondering why your VM won’t power on, or is hung without knowing the status.  Well we finally have an answer thanks to Serial Console access.

Makes the diagnosis of issues far easier and transparent.  Read more here.

Application Security Groups

This makes managing Network Security Groups at scale far easier, no longer are we reliant on IP Addresses, we can group network interfaces into Application Security Groups to govern traffic flow.

Read more here.

Security Centre Overhaul

I have to admit it, Security Centre wasn’t the most pleasant place to be.  Microsoft have taken this on-board and given the UI an overhaul.  Now it’s easier to understand, find and remediate issues within your Azure environment.

Azure Firewall

When I read into Azure Firewall, it wasn’t the big news as I was expecting, more so a v0.1 release from Microsoft.

In a nutshell, it monitors interesting outbound traffic.  So a step in the right direction, but more to come soon hopefully.

Read more here.

Azure Management Groups

For those of us that work across customers with multiple subscriptions this makes life so much easier!

Quite a simple thing, but again a great time sizer.  Read more here.

Encrypted ExpressRoute Microsoft Peering

This was announced under the radar so to speak.  We now have the ability to natively encrypt ExpressRoute Microsoft Peering traffic to access PaaS services within Azure.

So if you are a department with a requirement to use Azure PaaS securely, this will likely be the solution you are looking for.

3 x New Azure Exams with 80% Off

azureMicrosoft have launched three new Azure exams with 80% off, aimed at Azure Administrators, focused on depth rather than breadth.

  • Taking the AZ-100 and AZ-101 will lead to the certification Microsoft Azure Administrator.
  • If you have the 70-533 already, you can take the AZ-102 to achieve the certification Microsoft Azure Administrator.

AZ-100: Microsoft Infrastructure & Deployment

Measures knowledge and experience of:

  • Azure Subscriptions and Resources
  • Implementing and Managing Storage
  • Deploy and Manage Virtual Machines
  • Configure and Manage Virtual Networks
  • Manage Identities

More information can be found here and the 80% discount code is AZ100TRAVELING

AZ-101: Microsoft Azure Integration & Security

Measures knowledge and experience of:

  • Evaluate and Perform Server Migration to Azure
  • Implement and Manage Application Services
  • Implement Advanced Virtual Networking
  • Secure Identities

More information can be found here and the 80% discount code is AZ101HIKING

AZ-102: Microsoft Azure Administrator Certification Transition

Measures knowledge and experience of:

  • Evaluate and Perform Server Migration to Azure
  • Implement and Manage Application Services
  • Implement Advanced Virtual Networking
  • Manage Identities
  • Evaluate and Perform Server Migration to Azure
  • Implement and Manage Application Services
  • Implement Advanced Virtual Networking
  • Secure Identities

More information can be found here and the 80% discount code is AZ102PLANS