CCNA: Security at Commsupport

I nearly forgot that my CCNA was due to expire, but Cisco sent me a few reminders, well I say a few, it ended up bordering on spam.  This meant that my efforts to gain the CCENT and the CCNA would soon be in demise and I would enter the realms of a ‘retired Cisco Certified Network Associate’.

With this in mind, I had a few choices to make:

Do Nothing this was close to being a front runner, however, if I’m being honest with myself, not being a Cisco Certified really bothered me.  It was almost like riding your bike everyday and then one day your dad saying ‘you aren’t allowed on the bike anymore’.  This thought process made we not want to loose the ‘bike’ in the first place.

Stay The Same to be fair this never really entered the equation.  Since starting in IT, one thing that I have always enjoyed is moving forward with skills, projects, vendors and technologies.  I don’t ever want to be a person who says I have 15 years experience in IT, well in fact, what you really meant to say is I gave up learning 12 years ago, so I only really have 3 years experience.

Move Forward this was the front runner, but I didn’t have enough time to self study as I had done before with the CCENT and CCNA (see blog posts CCENT Study Guide and CCNA ICND2 Study Guide) due to family and work commitments.

I spend some time over on CertForums and met a friendly fellow called Cisco Lab Rat who is the Owner/Senior Instructor at Commsupport.  His forum posts impressed me and when my employer was looking for for a new Cisco training provider, I recommended Commsupport’s services.

A few of my colleagues have used Commsupport, and the feedback has always been top notch.  So with this in mind, I decided to head to Commupport for my CCNA: Security training.  I knew that it was going to be a tough week as Joe AKA Cisco Lab Rat performs the course over six days with the average day being 9:00am to 6:00pm.

One thing of note, is that I would highly recommend that you have either the CCNA or have configured Cisco ASA’s and Routers out in the field.  During my time as an engineer I have been lucky enough to configure oodles of ASA 5510 in high availability and more site to site VPN’s than I could shake a stick at.

Anyway, back to the course, before it starts Commsupport provide you with access to there e-learning portal and they ask that you brush up on the basics so you are fully prepared for the course.

The course is held in Central Finchley (London) and this meant a two and half hour trek, door to door.  The first day was a Sunday which I have to say isn’t generally the trend in IT courses, but it was welcomed as I knew we had a lot of information to cram in.

The Commsupport offices are OK, they aren’t the Ritz but they certainly aren’t the ghetto.  You have to bear in mind the course cost, along with the equipment being used and the technical expertise giving the training.

Upon arrival, I was greeted by a slightly over excited Joe!  He instantly made me feel welcome and offered me a seat in front of a stack of Cisco equipment.

I was surprised by the amount of equipment we had to use:

3 x Cisco 1841 Routers
1 x Cisco 2801 Router
1 x Cisco 3560
2 x Cisco 3550
1 x Cisco ASA 5510
2 x Laptops

Normally, in most courses I attend, you have the initial meet and great, with the ‘Hi I work for x and do y’.  None of this, we cracked straight on with Cisco.

The way that Joe teaches you is excellent, he has a passion for networking, Cisco and ranting about random topics.  The overall work flow for each day is really structured, essentially, you have.

Step 1 – Joe Talks

Joe talks over the days plan giving us an overview of what we are going to achieve e.g. Client less SSL VPN from ASA over two routers with two lots of NAT.

He then draws out the network diagram and talks over the concepts of each area e.g. why you would use an SSL VPN rather than L2TP IPSEC or PPTP.

Step 2 – Joe Does The Lab

This part is cool, Joe then puts together the lab and explains all the IOS commands, ensuring you understanding what he is doing and why.

Step 3 – You Do It

Joe prints you out a set of instructions to configure your lab, this includes parts from the GUI (if you like that sort of thing) and also CLI.  One of the aspects that I really enjoyed was when you couldn’t get something to work Joe would spend the time and help you troubleshoot the issue.

Conclusion

Overall it was an excellent week, I gained a much deeper understanding of what it actually was that I was configuring rather than just making it work.  Joe’s ability to convey very technical information in a humorous fashion is second to none.  The lab you have to use is fantastic and the ability to access Joe before and after the course really helps when you have questions you are unsure off.

Would I recommend the CCNA: Security at Commsupport, yes definately.

Topics Covered

Common Security Threats

Describe common security threats

Security and Cisco Routers

Implement security on Cisco router
Describe securing the control, data, and management plan
Describe Cisco Security Manager
Describe IPv4 to IPv6 transition

AAA on Cisco Devices

Implement AAA (authentication, authorization, and accounting
Describe TACACS+
Describe RADIUS
Describe AAA
Verify AAA functionality

IOS ACLs

Describe standard, extended, and named IP IOS access control lists (ACLs) to filter packets
Describe considerations when building ACLs
Implement IP ACLs to mitigate threats in a network

Secure Network Management and Reporting

Describe secure network management
Implement secure network management

Common Layer 2 Attacks

Describe Layer 2 security using Cisco switches
Describe VLAN security
Implement VLANs and trunking
Implement spanning tree

Cisco Firewall Technologies

Describe operational strengths and weaknesses of the different firewall technologies
Describe stateful firewalls
Describe the types of NAT used in firewall technologies
Implement zone-based policy firewall using CCP
Implement the Cisco Adaptive Security Appliance (ASA)
Implement Network Address Translation (NAT) and Port Address Translation (PAT)

VPN Technologies

Describe the different methods used in cryptography
Describe VPN technologies
Describe the building blocks of IPSec
Implement an IOS IPSec site-to-site VPN with pre-shared key authentication
Verify VPN operations
Implement Secure Sockets Layer (SSL) VPN using ASA device manager

vSphere 5.1 – My Take On What’s New/Key Features

With the release of vSphere 5.1, it’s been tough keeping up with all the tweets and information from VMworld 2012 San Francisco.

With the plethora of data, I thought it would be handy to blog about what the key features that will have the biggest impact on my every day life.

Licensing

vRAM – It’s gone, licensing is back to per physical processor.

vSphere Essentials Plus – Now includes vSphere Storage Appliance and vSphere Replication.

vSphere Standard  – Now includes vSphere Storage Appliance, vSphere Replication, Fault Tolerance, Storage vMotion and vCentre Operations Manager Advanced.

Beneath The Hood

Monster Virtual Machines

Virtual Machines, can now have the following hardware features:

1TB RAM
64 vCPUs
> 1 Million IOPS per VM

Wonder if I will continue to have those we need a physical SQL server conversation?

This is made possible by Virtual Machine Format 9.

vMotion

vMotion no longer requires shared storage.  This has been achieved by combining vMotion and Storage vMotion into a single operation.  So when a VM is moved, it moves the memory, processing threads and disk over the network to it’s target.

Now what is really, cool it maintains the same performance levels as the older vMotion with shared storage!

Note, I recommend that you use multiple NIC’s for vMotion as per my post High Availability for vMotion

vSphere Replication

Enables virtual machine data to be replicated over LAN and WAN.  Previously to achieve 15 minutes  a-synchronous replication you need sub 2 ms latency.

vSphere Replication integrates with Microsoft’s Volume Shadow Copy (VSS) ensuring that applications such as Exchange and SQL will be in a consistent state if DR was implemented.

vSphere Replication can be used for up to 500 virtual machines.

The initial seed can be done offline and taken to the destination to save bandwidth and time.

VMware Tools

No more downtime to upgrade VMware Tools.

vSphere Web Client

This is going to be the tool for administrating vCentre.  Some pretty cool features like vCenter Inventory Tagging, which means you can apply meta data to items and then such on them e.g. group applications together for a particular department or vendor.

We now have the ability to customise the web client to give it ‘our look and feel’.

Always getting called away when you are half way through adding a vNIC to a VM, well we can now pause this and it appears in ‘work in progress’ so we never forgot to complete an action.

For the pub quiz fans, you can have 300 concurrent Web Client users.

Link Aggregation Control Protocol Support

Used to ‘bind’ several physical connections together for increased bandwidth and link failure (think Cisco Port Channel Groups), this is now a supported feature in vSphere 5.

Memory Overhead Reduction

Every task undertaken by vSphere has an overhead, whether this is a vCPU or a vNIC, it requires some attached memory.  A new feature allows upto 1GB of memory back from a vSphere host which is under pressure.

Latency Sensitivity Setting

vSphere 5.1 makes it easier to support low latency applications (something which I have encountered with Microsoft Dynamics AX).  The ability to ‘tweek’ latency for an individual VM is great.

Storage

We now have 16Gb Fiber Channel support and iSCSI Storage Driver has been upgraded. Some very impressive increases in performance.

Thin provisioning has always been an issue unless your array supported T10 UNMAP.  With vSphere 5.1 a new virtual disk has been introduced the ‘sparse virtual disk’ AKA SE spare disk.  It’s major function is to reclaim previously used space in the guest OS.  This feature alone is worth the upgrade.

Cisco 4510 & E1000 Virtual NIC Latency Issues

We received a report from a client that the local site Exchange DAG had been falling over on a regular basis.

After some investigation we noticed that Exchange 2010 had changed from block level replication to file level replication with Event ID 10036, MSExchangeIS Mailbox Store

‘Continuos replication block mode is unable to keep up with the data generation rate. Block mode has been suspended, and file mode has been resumed’.

We performed various tests and the net result that a ping from one DAG member to the other resulted in > 4ms latency.  Not good!

This meant that when the Exchange 2010 cluster threshold was reached which is latency above 1ms for a period of 5 seconds, the DAG failed over, causing users Outlook clients having to relocate which server they mailbox resided on.

For a quick fix we changed ran the following Exchange 2010 Power Shell commands:

cluster /list

cluster.exe ‘cluster name’ /prop

Increased the SameSubnetDelay threshold by running:

/cluster.exe /cluster:’cluster name’ /prop samesubnetdelay=2000
/cluster.exe /cluster:’cluster name’ /prop samesubnetthreshold=2000

This means that the DAG will no longer failover, however it doesn’t resolve the underlying issue with the network latency.

I was concerned from a vSphere perspective as the client has vCentre Operations Manager which hadn’t alerted on any issues with bandwidth utilisation and the no other latency issues had been reported by end users.

What did I do to diagnose the issue? As it’s always good to know your peers thought process!

– Checked all vSwitches Uplinks to make sure no configuration changes had been made and that they all reported back as 1000 Full – Pass

– Checked Load Balancing on vSwitches, default as ‘route based on originating virtual port ID’ – Pass

– Checked Network Utilisation in Exchange VM’s, all reporting < 10 Mpbs – Pass

– Checked Performance Charts for Network Utilisation on ESXi Hosts, not above 300 Mbps for past month- Pass

– Checked ESXTOP, to ensure that VM’s correctly balanced across uplinks see post What NIC is my virtual server using – Pass

– Checked physical servers on same LAN which always reported back <1ms response times – Pass

– Checked CPU/Memory utilisation on Cisco 4510 switches all below 20% – Pass

– Checked VMware Update Manager, some hosts needed updates (7 in total) – Failed

My colleague was looking at various port counters on the Cisco 4510 switches and he noticed that flow control was enabled and the TXPause counter was increasing on the ports that the ESXi hosts where connected.  We turned off flow control and didn’t notice any difference.

By default Flow Control is enabled on ESX and ESXi but only comes into play if the switch you are connected too supports it, see this article

We updated all of the ESXi Hosts using VUM as it had various E1000 adapter updates.  However the issues continued to persist.

At this point, we knew this issue wasn’t going to be a quick fix and would require some more investigation as the issue could be any of the following:

– E1000 vNIC
– Cisco 4510
– Broadcom NetXtreme II BCM5709 (standard for HP and Dell servers for onboard NIC)

On this particular configuration we have HP2810G switches which are isolated from the LAN and are used for vMotion, Fault Tolerance Logging and MS Clustering Heartbeats.

Step 1 – Pass

We setup a couple of test VM’s on different ESXi Hosts and created a new ‘test’ vSwitch using an Intel 82571EB Adapter with VMXNET3 adapters on an isolated VLAN.  Monitored this for a day we received all response times <1ms.

Step 2 – Pass

To ensure that the TCP/IP stack in Windows 2008 R2 VM’s was reset and to remove those pesky hidden network adapters, we ran the following commands:

netsh winsock reset
netsh int ip reset

Removed Intel 82571EB Adapter from ‘test’ vSwitch and replaced with a Broadcom NetXtreme II BCM5709 VMXNET3 adapters on an isolated VLAN.  Monitored this for a day we recieved all response times <1ms.

Step 3 – Fail

Ran the same TCP/IP commands for Windows 2008 R2 VM’s.

netsh winsock reset
netsh int ip reset

Stayed with the Broadcom NetXtreme II BCM5709 bu changed vNIC to E1000 adapters on an isolated VLAN. Monitored this for a day we recieved some response times <4ms.

Step 4 – Fail

We now know that the E1000 vNIC was a cause of the issue, however we needed to go back to the VMXNET3 on the main LAN.

Again we ran reset the TCP/IP stack to remove any hidden network adapters.

Stayed with the Broadcom NetXtreme II BCM5709 bu changed vNIC to VMXNET3 adapters on LAN. Monitored this for a day we received all response times <3ms.

Conclusion

What have we learnt? Well the first thing is to change the virtual machine vNICs to VMXNET3 which reduces the latency across the LAN, however this is not acceptable as it should always be <1ms unless you have a broadcast storm.

The second thing is to replace the 4510’s, as they have been end of life for over 2 years.

High Availabity for vMotion Across Two NIC’s

When designing your vCentre environment, good practice is to associate two physical network adapters (uplinks) to your vMotion network for redundancy.

The question is does VMware use both uplinks in aggregation to give you 2GBps throughput in an Active/Active configuration? the answer to this is no.

In the above configuration we have two uplinks both Active, using the load balancing policy ‘route based on originating virtual port ID’ this means that the VMkernel will use one of the two uplinks for vMotion traffic.  The secondary active adapter will be used but only if the uplink vmnic4 is no longer available.

You might say this is OK, I’m happy with this configuration, I say how can we make it more efficient?

At the moment you will have a single Port Group in your vSwitch which is providing vMotion functionality (in my case it’s also doing Fault Tolerance Logging)

And the vSWitch has two Active Adapters

What we are going to do is rename the Port Group vMotionFT to vMotionFT1, go into the Port Groups properties and change the NIC Teaming setting to the following:

So what have we changed and why? First of all we have over ridden the switch failover order, we have specified that vmnic4 is now unused and that we are not going to ‘failback’ in the event of uplink failure.

You may think hold on Craig, why have you done this now we have no HA for our uplinks, well the next step is going to be adding another Port Group as follows:

VMkernel Select
Network Label vMotionFT2
Use this port group for vMotion Select
Use this port group for Fault Tolerance logging Do Not Select
IP Address 192.168.231.8 255.255.255.0

Once completed, we are now going to edit the Port Group vMotionFT2, go back into NIC Teaming and over ride the switch failover order and set vmnic1 to unused and no for failback.

So what have we achieved?

1. vSwitch1 has two active uplinks
2. vMotionFT1 Port Group is active and uses vmnic1 for vMotion & Fault Tolerance Logging
3. vMotionFT2 Port Group is active and uses vmnic4 for vMotion
4. We can perform two vMotions simultaneously using 1GB of bandwidth each
5. If we have an uplink hardware issue vMotion continues to work

ESXi Networking Part 3

NIC Teaming

The load balancing on NIC teams in ESXi is based on number of connections (think Round Robin) on Windows Server 2003/2008.

Load balancing only occurs on outbound connections i.e. traffic from VM > vSwitch > LAN

ESXi has a number of load balancing options which are:

Route Based on the Originating Virtual Port ID

ESXi runs an algorithm to evenly balance the number of connections across multiple uplinks e.g. 10 virtual machines residing on one vSwith which contains two uplinks would means that each uplink has 5 virtual machines using it.

Once a VM is assigned to an uplink by the VMkernel it continues to use this until it is vMotioned to another ESXi Host or a uplink failure occurs.

Route based on the originating virtual port ID is the default setting in ESXi.

Route Based on Source MAC Hash

This is much like ‘route based on originating virtual port ID’ as the MAC address of the VM’s do not change and therefore they will continue to use the same connection path over and over.  The only way around this is to have multiple virtual NICs (vNIC’s) within the VM which will produce multiple MAC addresses.

Route Based on IP Hash

This uses the source IP and destination IP to create an Hash.  So on each new connection a different uplink path would be taken.  Naturally, if you are transferring large amount of data, the same path would be used until the transfer had finished.

When enabling ‘Route Based on IP Hash’ you will get an information bubble:

You need to ensure that all uplinks are connected to the same physical switch and that all port groups on the same vSwitch are configured to use ‘route based on IP hash’.

Use Explicit Failover Order

This isn’t really load balancing as the secondary active (vmnic1) uplink will only come into play if vmnic4 fails.

If you have an active and standby adapter, the same procedure applies.

On all Load Balancing policies it is set by default to notify switches, what does this actually mean? Well it means that the physical switches learn that:

– A vMotion occurs
– A MAC address is changed
– A NIC team failover or failback has occurred
– A VM is powered on

Virtual Switch Security

Virtual switch security has three different elements which are:

– Promiscuous Mode, this is where the vSwitch and/or Port Group can see traffic which is not for itself.
– MAC Address Changes, this is where the vSwitch and/or Port Group is interested if the incoming traffic into the vSwitch/Port Group has been altered.
– Forged Transmits, this is where the vSwitch and/or Port Group is interested if the outgoing traffic into the vSwitch/Port Group has been altered.

In all of the above configurations you have a choice to either Reject or Accept the traffic.

VMware recommends that all of these are set to reject.

However if you are using Network Load Balancing or devices with ‘Virtual IP Address’s’ such as Hardware Load Balancers often use an algorithm that produces a shared MAC Address which is different from the original source or destination MAC address and therefore can cause traffic not to pass.

If in doubt you can always turn all three to reject, however I would recommend letting the Server Team know first!