Devious Networks

Objective 3.02 of the 301a LTM specialist exam requires the candidate to have an understanding of the available cookie persistence options on a BIG-IP system. This post will detail the different options and look at implementing each one to understand what is going on.

Cookie Persistence

The BIG-IP system can make use of cookies to ensure the client is reconnected or persisted to the same pool member that they were originally connected to.

There are four different flavours of cookie persistence options, all available under the Local Traffic  ››  Profiles : Persistence menu:

• HTTP Cookie Insert
• HTTP Cookie Rewrite
• HTTP Cookie Passive
• Cookie Hash

HTTP Cookie Insert

Probably the more frequently used (in my experience) of the options above as it requires the least amount of effort to setup. The BIG-IP system inserts a cookie into the client response using the following format:

BIGipServer<pool_name>=<pool_member_address>.<port>.<0000>; expires=<date, time>; <path>

Where:

• BIGipServer = the name of the cookie created by the BIG-IP system.
• pool_name = the literal name of the pool that serviced  the request.
• pool_member_address = the encoded value (in decimal format) of the pool member that serviced the request.
• port = the encoded port number (in decimal format) of the pool member.
• 0000 = a special reserved field for future use.
• expires = the date and time this cookie expires. This is either session based or on a configurable setting.
• path = the specific URL the cookies applies to.

Note that you also have the option to set your own cookie name.

HTTP Cookie Rewrite

This method requires a bit more elbow grease to setup as you will need to liaise with the team that either manages the web servers or the application code itself. This method requires a blank cookie in a very specific format to be created on the web server and sent in the response. The BIG-IP system then intercepts this cookie and rewrites it to include the value just like the insert method above.

I use an Apache web server and to set this up I had to do the following:

1. Enable mod_headers:

sudo a2enmod headers

2. Modify the virtual hosts file to include the Set-Cookie attribute:

<VirtualHost X.X.X.X:80>
Header set Set-Cookie BIGipCookie=0000000000000000000000000000000000000000000000000000000000000000000000000000000000

In case you're wondering those are 120 zeroes. This is required.

3. Restart the Apache2 service:

sudo service apache2 restart

HTTP Cookie Passive

This method requires even more greasing of those elbows. In this method the BIG-IP system does not create, modify or intercept the cookie sent from the web server. Rather it passively passes it through to the client. What needs to happen is each back-end server must be configured to set the cookie itself but formatted in a very specific way to include the all the name/values described in the cookie insert method above. Thus, when the client returns traffic to the LTM it can read the cookie and determine which pool member the client must be sent back to.


In my lab environment I have three back-end web servers using the following IP addresses:

• Web server 1 = 10.128.2.150
• Web server 2 = 10.128.2.160
• Web server 3 = 10.128.3.170

These are part of a pool named "HTTP_Web_Server_Pool_1".

We first need to encode the IP addresses by representing the IP address of the pool members as a.b.c.d then using the following formula to derive the decimal format:

a + b*256 + c*(256^2) + d*(256^3)

10 + 128*(256) + 2*(256^2) + 150(256^3)

= 2516746250

To work out the encoded port we can use the following set of instructions:

1. Convert the decimal port value to the equivalent 2-byte hexadecimal value.
2. Reverse the order of the 2 hexadecimal bytes.
3. Convert the resulting 2-byte hexadecimal value to its decimal equivalent.

As I'm using port 80 I need to be aware that anything less than 256 must be padded with 0x00 at the beginning, so for port 80 this would be:

1. Decimal to Hex = 80 = 0x0050
2. Reverse order = 0x5000
3. Hex to decimal = 20480

If I repeat for the other two servers I can go ahead and create a cookie for each server. Note that there are other probably better ways to set cookies on web servers. This example is provided just to show the basic concepts:

Web server 1 cookie:

Header set Set-Cookie "BIGipServerHTTP_Web_Server_Pool_1=2516746250.20480.0000;path=/;Expires=Sun, Apr 30 2017 19:00:00 GMT"

Web server 2 cookie:

Header set Set-Cookie "BIGipServerHTTP_Web_Server_Pool_1=2684518410.20480.0000;path=/;Expires=Sun, Apr 30 2017 19:00:00 GMT"

Web server 3 cookie:

Header set Set-Cookie "BIGipServerHTTP_Web_Server_Pool_1=2852290570.20480.0000;path=/;Expires=Sun, Apr 30 2017 19:00:00 GMT"

Now each of these cookies must be set on the back-end server just as we did with the cookie rewrite method.

Cookie Hash

The final method available is interesting because it is the only method which actually stores a persistence record in memory. With this method the cookie must also be set on the web server, however, you are not restricted to the format as per the other methods described above. What you do need to ensure is that whatever you have named the cookie on the web server you must also enter into the cookie hash profile, they must match. The BIG-IP system will then:

1. Parse the response from the web server for the Set-Cookie header
2. Uses the Hash Offset & Hash Length settings defined in the profile to compose a value. This value is stored in the persistence table, but not in a hashed form.
3. Any subsequent request from the client will force the BIG-IP system to parse the header for a cookie and then uses the same offset/length values to determine if this matches one stored in the persistence table. If there is a match it sends the request to the pool member.

Cookie Security

As with anything, before implementing cookie persistence on the BIG-IP system you need to understand and take into account security as it applies to cookies.

Encrypting the cookies would be a good start. The BIG-IP encrypts using cookies using a 192-bit AES cipher.

If encryption cannot be used you need to take into account the fact that you can quite easily enumerate the back-end pool member IP just by looking at the returned cookie. This can help an attacker perform reconnaissance on your network.

By default the cookie name itself quite clearly spells out BIGIP and will make it very clear that you are using a BIG-IP system, which further helps them understand the topology of your network. Changing the cookie name would therefore be the preferable mitigation to this.

Configuration

HTTP Cookie Insert

We are presented with a few options when configuring this method:

• Cookie Method: fairy self-explanatory this one, choose which cookie method you want.
• Cookie Name: a user-defined name for the cookie to override the default.
• Always Send Cookie: this setting specifies whether the system sends a cookie with every response or just the first. This setting only applies if the expiration is set to 'Session Cookie'.
• Expiration: allows a user-defined expiration of the cookie. If the 'Session Cookie' setting is enabled you cannot configure an expiration.
• Cookie Encryption Use Policy: introduced in v11.5, this setting effectively turns on or off encryption. The three settings offered are:
• Preferred: the system generates an encrypted cookie but will accept from clients, an unencrypted cookie.
• Required: the system generates an encrypted cookie and will only accept an encrypted cookie.
• Disabled: turns of cookie encryption altogether. 
• Override Connection Limit: used to override any connection limits set at the pool member level for clients that are eligible to be persisted.

Here I have the config. In this example I have not used the Session Cookie option or enabled encryption:


The resulting cookie provided by the BIG-IP system:


Here we provide a user-defined name for the cookie and turn on encryption plus set the expiration to Session Cookie meaning the cookie will only be valid for the duration that the client browser is open.


The resulting encrypted cookie provided by the BIG-IP system:

HTTP Cookie Rewrite

The settings for this method are the same as the insert method. However, we will go ahead and create the blank cookie on the back-end web server.

We can see the web server return the blank cookie to the BIG-IP system:


And then the BIG-IP rewriting the blank cookie to the appropriate format as it sends it to the client:

HTTP Cookie Passive

The settings for this method are further stripped down. A key setting to note here is the Cookie Name field. This must be set and must match the cookie name configured on the web server. You can also encrypt this cookie as per the other methods.


We can see now the back-end web server return its configured cookie to the BIG-IP, which then passes it onto the client:


Any subsequent request from the client within the timeout period carries the cookie presented to the BIG-IP:


The BIG-IP system then sends the traffic to the same pool member thus achieving our persistence goal.

Cookie Hash

The final method introduces some further configuration options:

• Hash Offset: specifies at which point in the cookie value the system begins the hash. For example, if the cookie value is 'cookie' and the offset is 1, then the BIG-IP system will hash the value 'ookie'.
• Hash Length:  specifies the number of characters in the cookie value to use when calculating the hash value. For example using the above cookie value if we set the length to 3 this will hash the value 'ook'.
• Timeout: specifies the duration of the hash persistence entry in the persistence table.
• Mirror Persistence: specifies whether to mirror the persistence record to the standby peer.
• Match Across Services: attempts to send all persistent connection requests received from the same client, within the persistence time limit, to the same node only when the virtual server hosting the second connection has the same virtual address as the virtual server hosting the initial persistent connection. Both VSs must use the same IP address and backend pool members. The service port can be different on each VS.
• Match Across Virtual Servers: similar to Match Across Services, but it does not require the virtual servers to share the same IP address. This configuration allows clients to access different virtual servers, regardless of their IP address, and still access the same pool member. The VSs must still use the same backend pool members.
• Match Across Pools: allows the BIG-IP system to use a pool that contains a persistence record for a specific client. If you have an iRule that determines which pool member to send the request to and the client uses a different URI in their request, using this option may bypass the iRule logic and potentially break the application.

Here we set the value of the cookie and we'll use the hash values as per the description above and shorten the timeout to 30 seconds:


I have also configured a very basic cookie on the web servers:

Header set Set-Cookie Test_Cookie=Cookie

Now when I make a connection the BIG-IP system will perform a hash as per the configuration on the cookie value "Cookie". We can verify a persistence record has been created with the following command which also helps us to verify the Hash Offset and Length working:

#show ltm persistence persist-records all-properties
Sys::Persistent Connections
cookie - 10.128.3.120:80 - 10.128.2.160:80
----------------------------------------------
  TMM           1
  Mode          cookie
  Value         ook
  Age (sec.)    11
  Virtual Name  /Common/HTTP_Web_Server_vs_1
  Virtual Addr  10.128.3.120:80
  Node Addr     10.128.2.160:80
  Pool Name     /Common/HTTP_Web_Server_Pool_1
  Client Addr   10.128.3.1
  Owner entry

Summary

Cookie persistence does require some careful thought practically speaking and from a security perspective but works well and can help with your persistence requirements. All the above can of course be achieved using an iRule which can provide more flexibility.

 

The BIG-IP system comes with a number of profiles straight of the box. Depending on your needs, these could be good go as they are. You can, of course, create your own custom profile should you need to tweak any setting according to your needs. This is the second part of a series to look at profiles on the BIG-IP system. The previous post looked at the Stream profile. This post will briefly take a look at the concept of child and parent profile relationships and inheritance.

Custom and Parent Profiles

When creating a custom profile you have to specify a parent profile that the custom profile will inherit the base settings from. This parent-child relationship is fundamental to pretty much all profiles on a BIG-IP system and is important to understand.

Another key concept in this relationship is that of inheritance. Within inheritance any custom profile created will initially inherit all the settings from the parent profile. Any changes made to the parent profile will automatically trickle down to any child profile using that parent profile. If you have multiple layers of custom profiles whereby a custom profile is used as a parent, inheritance will trickle all the way down from the default profile unless you have overridden any settings in the custom profiles as explained below.

The following diagram illustrates the concept of the parent-child relationship and that of inheritance.



The diagram above shows three levels of profiles. We can see that in the middle tier we have a number of custom profiles that use the default profile as their parent. We then have the lower tier profiles that use the middle tier custom profiles as their parent.

When you create a custom profile you have next to each setting a check box:

The process of customising a setting on any custom profile automatically removes its ability to inherit that setting from its parent for that specific setting, meaning that modifying the parent's setting will have no effect on the child.

When you initially create a custom profile you are presented with one more of the default profiles that come pre-built for the service/type of profile you need to create. You are therefore restricted to choosing an appropriate default profile for the type of profile you are creating.

Although this may seem obvious it's worth mentioning that inheritance only works from top to bottom. Modifying a child profile does not affect the parent profile in any way.

Configuration

We'll do a simple example here just to demonstrate the concept. I am going to create two custom HTTP profiles:

• The first one will be the child of the default HTTP profile.
• The second the child of the first custom HTTP profile.
• The first custom profile will remove the Accept-Encoding Request Header
• The second custom profile will add a new custom header and add the Accept-Encoding header back in

If we then apply each one in turn to a virtual server we should be to easily observe the results.

Custom Profile 1:


Custom Profile 2:


Now if we apply custom profile 1 to our virtual server we can observe the absence of the Accept-Encoding header which should sit just below the Accept-Language header:


Finally if we apply the second custom profile we can observe the Accept-Encoding header is now back and we have a new custom header called New-Header:

Summary

In my opinion using a custom profile is the better way to go rather than using the default profiles as it allows you to modify the settings should you require. If you do then having an understanding of inheritance is important to ensuring you get the desired results from the profile.

Thanks for reading.

Objective 3.02 on the 301a blueprint is all about profiles. Profiles are ubiquitous on a BIG-IP system, they contain all the knobs to turn to control the behaviour of traffic as it flows through the system. This is the first of several posts that will look at all things profiles on the BIG-IP system.

Stream Profile

This post will look at the Stream profile. The Stream profile is all about modifying the content of the data as it flows inbound from client to server, and outbound from server back to the client, through the BIG-IP system.

As with most profiles, you can configure this via the GUI or CLI, making use of the built-in profile functionality. Alternatively you can use an iRule using the STREAM command. Why would you choose one over the other? Using an iRule is always going to offer extreme flexibility and Stream is no exception. Via an iRule, you can go to town with a regular expressions that allow you to be granular with your matching patterns.

The Stream profile comes with a few caveats:

• If you apply the Stream profile to a virtual server without a HTTP profile, and only a TCP profile, you are able to do a match & replace of the entire data portion of the TCP segment, that is, both the HTTP headers & the payload itself.
• If you apply the Stream profile to a virtual server with a HTTP profile you are restricted to match & replace on the HTTP payload only.
• If the back-end server performs compression this prevents the BIG-IP system from being able to read the content. By creating a custom HTTP profile and removing the Accept-Encoding header you can remove the instruction to the server to compress the data.

The following are simple examples of how you can use Stream to modify the content both in the header and the payload.

Configuration

Configuration of the Stream profile via the GUI is done by going to Local Traffic  ››  Profiles : Other : Stream & clicking Create.

You'll be presented with two fields that you can populate:

• Source: This looks at the string sent by the client that we wish to rewrite.
• Target: This specifies the string used to rewrite the source string.

Using the above diagram we can amend the headers using the following configuration. Important! The values you enter here are case-sensitive:


Now if we test the above we see the headers being modified from the client (10.128.3.1) to the back-end server (10.128.2.170):


Remember, modification of the HTTP headers is only possibly when no HTTP profile is applied. If I were to add a HTTP profile to the virtual server and attempt the connection again, the LTM would pass through the headers unmodified:


To amend the payload we would do something similar:


Now if we test we should see the payload amended:


Hold on a minute, this isn't right. We should expect to see the word 'Everybody' instead. If we go back to the caveats we see that we need to ensure that the back-end server does not compress the data. As it stands all requests to my server contain the Accept-Encoding header. We need to create a custom HTTP profile to remove this Request Header and apply this to the virtual server:

Now let's try again:


Success!

There is another way of doing things. The above example can actually all be done within the Target field completely. You can achieve this by making use of delimiters to separate out the replacements using the following syntax in the Target field of the profile:

@<search string1>@<replacement string1>@ @<search string2>@<replacement string2>@

This now allows you to replace multiple strings within a response. So, let's say that we now wish to replace multiple strings where the original response body contains the strings "Hello World! Get the man a beer!". We want to replace the response strings thus:

• World! -> Everybody!
• Beer -> Pepsi

We can configure the following in the profile:

ltm profile stream LAB-STREAM-PROFILE {
    app-service none
    defaults-from stream
    source none
    target "@World!@Everybody!@ @beer@Pepsi@"
}

I've shown the CLI config as the GUI form field is too small to show everything.

Now when we test, although the back-end server will respond to the LTM with "Hello World! Get the man a beer!" the Stream profile will replace the words as above:

Real World Examples

It's all well and good providing simple examples of how the Stream profile works but the question is how you use this in a real-world scenario. This DevCentral resource page (login required) provides some good examples of how Stream can be used.

An example would be to remove the 'Server' response header to help obfuscate the web server type as a measure of security. Whilst this can be performed at the server level, it is not always done.

Summary

The Stream profile is a handy little tool to easily replace data between the client and server without the need for any involvement of the development team. Or rather to provide a quick fix until the back-end application is updated.

The previous post we went through configuring HA between the two devices. In this post we'll look at how we can verify ConfigSync is working as expected. As noted on the very first post, I won't be convering failover or mirroring in this series.

Verification & Testing

At this point we should now have successfully setup an active/standby BIG-IP system as indicated by both devices showing as Online and In Sync.


We can also confirm this on the CLI:

root@(ltm-1)(cfg-sync In Sync)(Standby)(/Common)(tmos)# show cm sync-status
-------------------------------------------------------------------------------------
CM::Sync Status
-------------------------------------------------------------------------------------
Color     green
Status    In Sync
Mode     high-availability
Summary  All devices in the device group are in sync
Details
         ltm-2.lab.com: connected
         LAB-SYNC-FAILOVER-GRP (In Sync): All devices in the device group are in sync
         device_trust_group (In Sync): All devices in the device group are in sync

Now let's make a change on the active node and observe what happens. I'll create a dummy node object. Before I commit the change let's see what the status of the commit IDs are between the devices by using the run /cm watch-devicegroup-device command:

We can see the Commit ID all showing the same number which tells us the devices are fully synchronised. It also tells us which device originated the last Commit ID.

As soon as I create the object the Commit ID is incremented by 2 on the device I made the change.

This information is also sent to the peer via the MCP channel:

root@(ltm-1)(cfg-synChanges Pending)(Standby)(/Common)(tmos)# run /cm sniff-updates
Listening for commit_id_update on -i internal:h port 6699 (^C to exit)
[09:43:05] ltm-2.lab.com (v11.6.0) -> LAB-SYNC-FAILOVER-GRP: UPDATE CID 46.0 (ltm-2.lab.com) at 09:43:05

At this point the system should tell us that changes are pending:


If we click the Changes Pending hyperlink and then perform the sync we can see that the Commit IDs are now back in sync between the devices:


If we run the same command again we see this information again passed over the MCP channel. The output below tells us the new Commit ID (46.0) and the previous (44.0):

[09:48:34] ltm-2.lab.com (v11.6.0) -> LAB-SYNC-FAILOVER-GRP: UPDATE CID 46.0 (ltm-2.lab.com) at 09:43:05 FORCE_SYNC
[09:48:34] 10.128.2.110:49090 -> LAB-SYNC-FAILOVER-GRP: SYNC_REQ CID 44.0 (ltm-2.lab.com) at 09:37:01
[09:48:35] ltm-1.lab.com (v11.6.0) -> LAB-SYNC-FAILOVER-GRP: UPDATE CID 46.0 (ltm-2.lab.com) at 09:43:05

If we go to Device Management  ››  Overview, in the Devices section if we click Show Advanced View we can see some more useful information:

Here is what some of the less obvious items mean:

• CID Originator: The Commit ID originator displays the device that performed that last successful sync between the devices.
• CID Time: The Commit ID Time displays the most recent Commit ID time. This matches the output in the command above when the we first created the dummy node and clicked Create.
• Last Sync Time: Displays the time we initiated the ConfigSync. You'll see it matches the timestamp on the output just above the screenshot.
• Last Sync Type: Displays the last type of sync that was performed. If this is the first sync it will show as 'Manual Full Load'. If it is an incremental sync it will display 'Manual Incremental'. Note, it uses the word 'manual' because I have disabled Auto Sync.
• LSS Originator: This field displays the device that most recently performed a successful sync operation.  Typically this will match the CID Originator.
• LSS Time: This field displays the most recent successful sync. Typically this will match the CID Time.

Lastly on this point, you should see the following log, either locally or on your Syslog/SIEM:

Apr 15 09:48:34 ltm-2 notice mcpd[6806]: 0107168c:5: Incremental sync complete: This system is updating the configuration on device group /Common/LAB-SYNC-FAILOVER-GRP device %cmi-mcpd-peer-/Common/ltm-1.lab.com from commit id { 44 6409145281621881942 /Common/ltm-2.lab.com } to commit id { 46 6409146844437164723 /Common/ltm-2.lab.com }.

At this point the new configuration should now be on both boxes.

Summary

High availability on a BIG-IP system is clearly a big topic as there are a number of different moving parts. If implemented properly and if you have a good understanding of the process it should make troubleshooting any issues a little easier.

Thanks for reading.

The previous post in this three part series provided an overview of some design decisions and requirements to setup HA.

Now we're done doing all the pre-checks and have an understanding of the various DSC components we can go ahead and start configuring. In my experience the order in which you do these next few steps is not important except setting up the trust between the devices which must be done last.

Each of these steps is required on each physical device, with the exception of the trust configuration which can be done on either device.

Self IPs and VLANs

The very first thing you need to do when setting up HA between two BIG-IP devices is to configure the basic network settings. The two critical pieces of information are VLANs and a self-IP for each VLAN. Using the topology in the first post I go ahead and create each of the VLANs and self-IPs. Do not forget to create the floating IP(s) on both units.

ConfigSync

First off we will configure the ConfigSync. Go to Device Management  ››  Devices  ››  <DEVICE>  ›› Device Connectivity ›› ConfigSync and enter the Local Address.

Failover

Next we'll configure failover. In a production environment you may want to configure two Failover Unicast addresses - a TMM self IP and the management IP address. In this lab I am selecting the HA VLAN self IP. Device Management  ››  Devices  ››  <DEVICE>  ›› Device Connectivity ›› Failover Network.

Mirroring

F5s recommendation is to use a dedicated VLAN and interface for mirroring and to configure a primary and secondary address. In my lab I will use a single dedicated virtual interface and VLAN for this purpose. Device Management  ››  Devices  ››  <DEVICE>  ›› Device Connectivity ›› Mirroring:

Device Trust

We're now ready to setup the trust domain. If we go to Device Management  ››  Device Trust : Peer List click Add and enter the management IP address, administrator username and password of the peer and then click Retrieve Device Information. You will be presented this page:


So what is this being presented? As we are entering the management IP address of the peer to establish the device trust, the peer presents the certificate associated with this interface. This happens to be the certificate found in System  ››  Device Certificates : Device Certificate  ››  Device Certificate. In this instance I have yet to go through the process of getting a device certificate from a PKI so here the peer device is presenting a self-signed certificate.

After you click Finished, based on my observations the following sequence of events occur:

1. The BIG-IP device you initiate the device trust from acts as the TLS client and initiates a TCP 3-way handshake on port 4353 to its peer over its ConfigSync IP address. The devices then go through a TLS mutual authentication process with each other whereby each presents its identity certificate to the other.


If we take a look at the /var/log/ltm file during this process we see the following:

ltm-1 notice mcpd[6608]: 01071436:5: CMI listener established at 10.128.2.110 port 6699
ltm-1 notice mcpd[6608]: 01071431:5: Attempting to connect to CMI peer 10.128.2.111 port 6699
ltm-1 err mcpd[6608]: 0107142f:3: Can't connect to CMI peer 10.128.2.111, TMM outbound listener not yet created
ltm-1 notice mcpd[6608]: 0107143a:5: CMI reconnect timer: enabled
ltm-1 notice mcpd[6608]: 01071431:5: Attempting to connect to CMI peer 10.128.2.111 port 6699
ltm-1 notice mcpd[6608]: 01071432:5: CMI peer connection established to 10.128.2.111 port 6699
ltm-1 notice mcpd[6608]: 01071451:5: Received CMI hello from /Common/ltm-2.lab.com
ltm-1 notice mcpd[6608]: 01071038:5: Master Key updated by user %cmi-mcpd-peer-10.128.2.111
ltm-1 notice mcpd[6608]: 010714a0:5: Sync of device group /Common/device_trust_group to commit id 17 6407634457465257616 /Common/ltm-2.lab.com 0 from device /Common/ltm-2.lab.com complete.
ltm-1 notice mcpd[6608]: 0107143a:5: CMI reconnect timer: disabled, all peers are connected

What is going on here? Remember the CMI process is used as part of the ConfigSync process to allow the local MCP daemon to exchange MCP messages and commit ID updates to its peers. If we look at each line we can work out what is happening:

• CMI listener established at 10.128.2.110 port 6699
• The local MCPD has initialised and created a listener on port 6699 so it can now accept incoming CMI connections.
• Attempting to connect to CMI peer 10.128.2.111 port 6699:
• The local MCPD process is attempting to set up a CMI connection to its peer in the trust domain. 
• Can't connect to CMI peer 10.128.2.111, TMM outbound listener not yet created:
• Not an issue per se but the local process has not yet established a listener (or failed to bind the socket). 
• CMI reconnect timer: enabled:
• The local device is starting up a timer to try reconnecting every five seconds. 
• CMI peer connection established to 10.128.2.111 port 6699:
• This device has successfully created a CMI connection to another device in the trust domain. 
• Received CMI hello from /Common/ltm-2.lab.com:
• The peer device has established a CMI connection to this device. 
• Master Key updated by user %cmi-mcpd-peer-10.128.2.111 
• Not 100% on this but I suspect this has something to do with the exchange of certificates and the encryption used in the communication exchange(s).
• Sync of device group /Common/device_trust_group to commit id 17 6407634457465257616 /Common/ltm-2.lab.com 0 from device /Common/ltm-2.lab.com complete.
• The two devices are now part of the trust domain and are in sync for the default device_trust_group. The commit ID 17 is the ID for this transaction.
• CMI reconnect timer: disabled, all peers are connected
• Fairly self-explanatory this one.

There now exists a full mesh between the MCP processes on each device.


2. What happens next is that the failover communication channel is established between the devices.




3. Next the mirroring communication channel is established between the two devices.


4. We can now confirm there is an MCP mesh between the two devices via the CLI:

[root@ltm-1:Standby:In Sync] config # netstat -pan | grep -E 6699
tcp        0      0 ::ffff:10.128.2.110:40872   ::ffff:10.128.2.111:6699    ESTABLISHED 6608/mcpd
tcp        0      0 ::ffff:10.128.2.110:6699    ::ffff:10.128.2.111:57790   ESTABLISHED 6608/mcpd

[root@ltm-2:Active:In Sync] config # netstat -pan | grep -E 6699
tcp        0      0 ::ffff:10.128.2.111:6699    ::ffff:10.128.2.110:40872   ESTABLISHED 6806/mcpd
tcp        0      0 ::ffff:10.128.2.111:57790   ::ffff:10.128.2.110:6699    ESTABLISHED 6806/mcpd

Device Groups

At this stage all three channels of communication are now open and established and there exists an MCP mesh. The two devices will both be in the active state as we have not created a Sync Failover Device Group.

There is, however, a default Sync Only Device Group which is automatically created called device_trust_group. This is a system-generated and manage device group used to synchronise trust information across all devices.

As both devices are now in the Active state we want to go ahead a create a Sync Failover Device Group which will turn this into an active/standby pair.



For most implementations you'll have two members. As per F5 best practice recommendation the Network Failover checkbox is ticked.

The decision as to whether you enable Automatic Sync is going to be based on your policy. By having it enabled you reduce the number of steps during implementation, one less thing to think about. However, you lose the ability to confirm all your configuration before committing to the peer device.

In our network we do not enable Full Sync. I personally don't see the value in enabling it. By leaving it disabled the system will only send an incremental ConfigSync to its peer up to the value (size) in the box. If the ConfigSync is bigger than this a full sync will be done anyway.

Once created, we should now see the devices assume their active and standby roles. Both will show a prompt in the top-left corner displaying 'Awaiting Initial Sync':


If you click that it will take you to Device Management  ››  Overview. Here under the Devices section we want to click the name of the device we're currently logged into indicated by the word 'Self' and click Sync Device to Group then Sync.

The two devices are now in an active/standby configuration. In the next post we'll verify the ConfigSync is working as expected.

The 301a blueprint makes reference to High Availability and the technologies that are encompassed within in several of the objectives (2.05, 2.06). This is the first of three posts that will look into the detail of how HA works between two BIG-IP devices. This series will focus on the design, setup and monitoring of an active/standby setup. I have decided to focus more on the ConfigSync technology for verification as failover and mirroring are deserving of their own posts.

BIG-IP High Availability Design

A quick disclaimer - at the time of writing this the 301a blueprint is based on v11.2. This post implements HA using v11.6.0.

As much as I would like to have a full blown lab with hardware, the finances don't yet allow me, so I have to make do with a virtual lab. I am running two virtual edition (VE) instances of the BIG-IP product within VMware Workstation.  In a production environment, you will of course need to observe the F5 recommended best practices. These include:

• Use both directly connected hardwired failover and network-based failover.
• Use both either a dedicated or shared VLAN and management interface for failover.
• Use a dedicated mirroring VLAN & directly connected interface.
• Use a primary and secondary mirroring addresses.

Some other things to consider:

• When implementing HA on a VIPRION system you will be setting this up between the VCMP guests only.
• When implementing HA using VEs, you will want to deploy each VE on a separate physical host for resiliency.
• If you wish to deploy an active/active scenario take a look here.
• If the HA devices are Layer 2 adjacent, consider a non-routable VLAN for the failover link appropriately sized to conserve IP addressing space.

The following diagram shows the lab design. I will use a dedicated virtual interface and VLAN for each of the HA functions.

Device Service Clustering (DSC)

It's worth taking some time to understand the various components and technologies that make up DSC. Having a clear understanding also helps to ensure that the configuration will progress smoothly without issue and will help you troubleshoot and issues along the way.

DSC is just another name for HA. The key components of DSC include:

• Device trust & trust domains: basically all BIG-IP systems need to be able to trust each other before they can form a HA relationship. They do this by exchanging digital certificates. A trust domain is simply therefore a collection of BIG-IP devices that trust each other.
• Device groups: a DG is a collection of BIG-IP devices in the same trust domain that can synchronise and failover their configuration. There are two flavours:
• Sync-Failover: your typical/standard DG that allows for failover between two or more BIG-IP device.
• Sync-Only: a DG that allows you to synchronise specific data within folder. This post I wrote covers folder and Sync-Only DGs in more detail.
• Centralized Management Infrastructure (CMI): used with the ConfigSync process, this component allows the local MCP daemon (see below) to exchange MCP messages and commit ID updates to its peers.

There are also some pre-requisites that need to be in place before configuring DSC:

• Licensing: each device must have the same product licensing and module provisioning
• Software version: each device must run the same BIG-IP software version
• Management IP: each device must have a unique management IP address
• NTP: in a production environment NTP must be configured and in sync
• Ports: the following ports are required between the two devices:
• TCP 4353: Used for ConfigSync
• TCP 443: Used to setup up the device trust
• TCP 1029-1043 (v11.4+): Used for connection and persistence mirroring. A separate port starting at 1029 and incrementing by 1 is used for each traffic group.
• UDP 1026: Used for network failover
• UDP 1028 (v11.0 - 11.3): Used for connection persistence and mirroring. A single port is used for all traffic groups

A quick note on the ports above. It's always worth checking basic connectivity between the two peers as the physical network may be managed by a 3rd party. If deploying a VE, the host system, whether it's VMware of Hyper-V may also be managed by a 3rd party so definitely worth double checking.

BIG-IP Daemons

I thought it may be worth breaking this out into its own sub-section. A daemon is a computer program that runs as a background process, rather than being under the direct control of an interactive user. A BIG-IP system requires a number of core daemons to run in order for the system to be operational. In regards to DSC the main one that will pop up is the MCP daemon.

The Master Control Program (MCP) daemon allows communication between userland processes and the TMM. Needless to say, it's pretty important.

What is a userland process? This forms part of a computer system's memory & hardware protection mechanism. Basically, userland or user space processes are those that run outside of the systems kernel. A daemon is a userland process. If a daemon requires access to the system's hardware (input/output etc.) it has to go through the kernel. This Wikipedia page and this Stack Exchange post help to explain the userland concept further as the above description is a massive simplification.

In my minds eye this is how I visualise the above concept.

Using the above diagram as a reference point, when an configuration update is made the change is communicated to the local MCPD process. This communicates to the local TMM process which then sends the configuration & commit ID to the remote TMM process where the process is reversed.

BIG-IP Processes

Before deploying HA you should also check to ensure the following processes are running:

• devmgmtd: Establishes and maintains device group functionality.
• mcpd: Allows userland processes to communicate with TMM.
• sod: Provides failover and restart capability.
• tmm: Performs traffic management for the system. 

This can be checked with a simple command:

# bigstart status devmgmtd mcpd sod tmm

Certificates

As described above a trust domain requires the exchanges of digital certificates. So which certificates will the system exchange?

As described in part two of this series, when you setup device trust between the two peers, the first thing you must do is to enter the management IP address of the peer device to retrieve some information. This first TLS exchange uses the device certificate stored in System > Device Certificates > Device Certificate. This certificate can either be a self-signed certificate (the default) or you may wish to have your internal PKI sign the certificate.

During the process of establishing the MCPD mesh, the two peers communicate over TCP 4353 and use the identity certificates stored in Device Management  ››  Device Trust : Identity.

When you setup a BIG-IP system the system acts as its own Certificate Authority (CA) and creates it's own identity certificate. We can verify this to be true by looking at the certificates in the following locations:

• CA: /Common/dtca.crt
• Identity: /Common/dtdi.crt 

The following shows the CA cert and identity of each BIG-IP device prior to any HA configuration. As you can see each BIG-IP has signed its own identity certificate.


In the next post we'll go ahead and start configuring HA.