So what is this rate shaping all about?  Well for a kick off it is in fact simplicity itself to implement using Zeus ZTM v6 (or indeed previous ZXTM versions) SLM capabilities.

Solution Components

ZTM provides two technologies that are useful to in service monitoring/protection. The first is the SLM class and the second is the Rate class.

SLM  Class

An SLM or Service Level Monitoring class is a mechanism for monitoring the response times of the site/service you provide. The SLM class provides the facility through TrafficScript to report on the percentage of requests that confirms to the threshold configured in the SLM class that the response arrives back within 

Rate Class

The Rate Class is like a pipe with a definable capacity down which requests against your service flow. The pipe can only allow its maximum capacity and no more. The Rate Class (via TrafficScript) provides a overflow queue (lets think of it as a bucket catching the excess flow that isn’t getting through the pipe) that can be processed once requests have dropped below maximum capacity.

Put it all together: Step by Step

So lets set up a service that uses SLM and Rate Shaping.

I’m using the ZTM r6.02 virtual Appliance on my home lab and built this config as I wrote this blog (its that intuitive )

Create a Virtual Service

I’ve created one called “HTTP Service”

and a Pool called “HTTP Servers”

for the purpose of this post I have used Google to provide the web servers by simply adding the node www.google.com:80

Set Up a SLM

Click catalogue and then SLM tab

In this case I have called the Class “Subscription”

The SLM class offers several values to modify but I am only interested in the millisecond response time as I am going to use TrafficScript to test the other values.

That’s the SLM class created

Now apply it to the Virtual Service

Click Edit next to the Classes tab in the “HTTP Service” Virtual Service configuration summery.

select Subscription and click update

The Virtual service is now been monitored against the response_time value set in the SLM Class. In this case 40 milliseconds. 

Now we need to check the value and do something with it.

Rate Class

From the Catalogue tab select Rate and create a new Rate Class

I’ve called mine “Premium” you can have many rate classes and as is typical with ZTM the value used to determine which class to apply are numerous and highly configurable via TrafficScript.  E.G. it could be the host name, referrer, GeoIP check, username, cookie value etc etc etc that determines which class to apply.  

The values to configure in a rate class are simple and represent a volume of requests that your service can handle measured in requests per second and requests per minute. There are two values so that you can provide a quantification of what is sustainable by your service. If for example we could only configure 10,000 requests per minute in theory these could be delivered in the first ten seconds leaving 50 seconds where the rate class will not allow any additional connections.

Putting it all together

TrafficScript:

You need to do something if your service incredibly popular.

This TrafficScript checks if the service is conforming to our agreed SLA (Service Level Agreement) This is for you to decide what is acceptable. In this example its 95% of transactions been completed within the millisecond response time configured in the SLM Class “Subscription”. If our service drops below 95% the Rate Class is utilised to limit the number of connections that the service will handle.  This is a simple TrafficScript that achieves this:

connection.setServiceLevelClass( "Subscription" );
$conforming = slm.conforming( "Subscription" );

log.info("Percentage Conforming is : ". $conforming);

# Test our SLM threshold. If response times are degrading
# apply rate shaping class to protect service

if( $conforming < 95 ) {
rate.use("Premium");
}

The Rate Class is applied while the SLM Class detects that the service is performing below 95% conforming (to the configured 40 millisecond response).

The Rate Class limits the number of connections that will be processed and also provides a mechanism for queuing excess connection attempts. This queue will be held until the level of activity drops below the per second threshold OR the TCP connection times out (which is bad for user experience if left unhandled).

To handle the excess traffic a second TrafficScript is required to manage this:

# How many queued requests are allowed before we track users.
$shapeQueue = 10;
$backlog = rate.getbacklog("Premium");
if ( $backlog > $shapeQueue )
{
http.sendResponse( 503, "text/html", resource.get( "busy.html"), "" );
}

This script sets a value as an acceptable queue length( $shapeQueue ). While the Rate Class is applied, each request is checked to see if the number of connections in the queue is greater than the desired maximum queue length.

If it is then we can handle the connection in a number of ways. In this example I have configured the ZTMs to server a busy page and importantly used the HTTP Error 503 – Service unavailable in the response. The reason I have configure this is to prevent upstream servers from caching this response.    

Testing the configuration

The key to a successful deployment is making sure that millisecond response value is realistic and the number of connections configured in the Rate Class accurately reflect the threshold that the service can deliver – a small margin of error. 

In many cases this can be difficult to establish without sufficiently complex load testing. If you have a very module architecture with well established performance characteristics then simply plug in the values and go home for the weekend safe in the knowledge that everything is well with the world. 

If you are not so luck there is a nice way to monitor in real-time performance activity of your service in relation to the SLM and Rate Class configuration.

Example Interactive

I use Apache JMeter to create load. And that’s my next blog article  

and use the ZTM current activity monitors

To get real-time feedback.

Its been a long time since I posted mainly because all of the work I have been doing is covered by NDA (no disclosure agreements). One of the secret squirrel pieces of work involved Zeus ZXTM r5.2 beta testing. Well the good people of Zeus must have been having more moments of genius that usual and have decided that the great features due for release in r5.2 beta were not enough! So they have pulled out all the stops and are releasing v6.0 now known simply as ZTM (Zeus Traffic Manager).

This is a huge improvement over v5.1 with true load balancing of single IP, via a nifty loopback interface which can move between ZTM’s (That’s going to take some getting used to). Other changes include SSD backed cache, UI enhancements, enhanced support for cloud deployment (such as Amazon EC2) and extending TrafficScript functionality to name but a few.

Zeus are delivering a presentation where I work next week, I’ll be sure to post following this with more in depth information.

In the meantime here’s further reading

http://knowledgehub.zeus.com/news/2009/10/20/zeus_traffic_manager_6_0_released

Just a quick post to draw your attention to the excellent plug-in for eclipse IDE.
This is fantastic. You can now edit Traffic Script in a nice IDE (like a real developer ) and avoid having to spawn loads of browser windows to check traffic Script reference etc.

Deployment
Simplicity itself. You point the eclipse development environment at your ZXTM cluster (or clusters you can manage multiple environments) and the Plug-in provides the functionality that you have in the ZXTM Traffic Script GUI and then some! 

Context sensitive object browser

Zeus Knowledge Hub
More detail about the workings of the Plug-in here
http://www.zeus.com/documents/en/ZX/ZXTM_Plugin_for_Eclipse.pdf

and the Setup walkthrough here
http://www.zeus.com/documents/en/ZX/ZXTM_Plugin_for_Eclipse_walkthrough.pdf

You can download eclipse here

http://www.eclipse.org/downloads/

This is the link you need for Windows

http://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/galileo/R/eclipse-java-galileo-win32.zip

Zeus. – Why wait indeed …..

This post gives you a sneak preview of one of the enhancements that is due for release in ZXTM version 5.2

Rules prior to and including  ZXTM r5.1 

In the current release of ZXTM rules are managed with the order that the rules are processed been defined by moving them up or down in relation to their neighbour. The top rule is processed first working down the list from top to bottom.

This is fine for a few rules but once you have more than eight or nine rules on a virtual server the interface is a pain to manage.

ZXTM 5.2 Rules Sneak Preview

In the forthcoming release the rules section has been given an overhaul and now features the ability to drag and drop the rule into the position you want it. This will be very useful and speed up deployment of new rules.

Each rule has a little tab next to

Click on the tab

and the cursor changes to a cross

These screenshots show rule B moving from the bottom of the list to the top.

Video

You can view a video that demonstrates the new rules system in action here

Valued

It is fantastic to be able to feedback to Zeus and see the results so quickly.

I can assure you that your feedback, observations and enhancement requests are greatly appreciated. The development team will evaluate the request for feasibility. You could have your request implemented in a subsequent release.

This post relates to RHEL 5.x

After you install the OS

Set up user accounts

useradd username
passwd password

I create an account called remote that I can us to login via SSH.

Network

Set up networks to provide access to internet

For example here is my VM config ( /etc/sysconfig/network-scripts/ifcfg-eth0 file for eth0 network interface:)  use nano or vi to edit this as required. 

DEVICE=eth0
BOOTPROTO=static
BROADCAST=192.168.1.255
HWADDR=00:0F:EA:91:04:07
IPADDR=192.168.1.111
NETMASK=255.255.255.0
NETWORK=192.168.1.0
ONBOOT=yes
TYPE=Ethernet

Static routes:

You would put your default gateway in “/etc/sysconfig/network” using “GATEWAY=x.x.x.x”

You would typically add static routes into a startup file the system will read on boot e.g.

/etc/sysconfig/network-scripts/route-eth0

#Route Description

10.8.0.0/24 via 10.0.0.1 dev eth0

Once configured you can get the system to re-read the files as follows:

“service network restart” (redhat specific)

Or

“/etc/init.d/network restart” (Works with just about any Unix box)

RHEL 5 Registration

rhn_register

Follow onscreen dialogue to register RHEL – You obviously need to have purchased a subscription.

Install Java

yum install java

its that easy

General OS Update

To update RHEL simply type Yum update

ZXTM pre Install

This section assumes that you are doing the install remotely from Windows machine. You must have port 22 access through any firewalls between zxtm and remote console.

Download following utils

winscp http://winscp.net/eng/download.php

putty http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

Use winscp to upload ZXTM.tar file to /home/remote

Use putty to SSH to host as remote

Then sudo to root

Untar the zxtm install file

Tar –zxf /home/remote/ZXTM_51r1_Linux-x86.tgz

This creates a folder called ZXTM_51r1_Linux-x86

Navigate to the folder it contains zinstall

Type ./zinstall

Once the install is completed you may need to configure the firewall (iptables) to allow access to the administration port.

RHEL Firewall

iptables – open ports required for administration

e.g. this command allows access to ZXTM default admin ports from 192.168.1.1

iptable –A INPUT –d 192.168.1.1 –p –tcp –m tcp –dport 9080:9090 –j ACCEPT

/etc/init.d/iptables save

List command allows inbound connections on 9090

iptables –A INPUT –p tcp –m tcp –dport 9090 –j ACCEPT

Obvious but if you are installing ZXTMs into existing platform consider the infrastructure such as firewalls and routers. You may require static routes on the ZXTM “internal” interfaces to route to you web servers. Other consideration is to make sure that the ZXTM interfaces can ping the gateway address set in the network configuration.

Useful ports to open on any firewall/s in related infrastructure. You may also need to configure iptable on zxtm hosts as well.

SNMP (161) – to infrastructure server (monitoring).

SSH (port 22) – between administrative console & infrastructure server (back up).

RADIUS (1812) – to Radius server if required

HTTP (80) – to all relevant servers via internal interface and to external to internet/network that the clients come from

HTTPS (443) – to all relevant servers via internal interface and if providing SSL passthru to external to internet/network that the clients come from   

DNS (53)

ZXTM specific OS

Areas of the OS to be familiar with from a ZXTM configuration perspective (assumes Redhat RHEL)

Zeus Install directory

/usr/local/zeus/zxtm/

Logs:

/usr/local/zeus/zxtm/log/

They include

errors – this is were log.info() output is logged

audit – Log shows security/change activity viewed via Diagnose/Audit Log

Plus any logs that you have created for virtual servers (Activity/View Logs/Virtual Server Request Logging)

Extra Files

This is were extra files live such as IP white list or html assets that you want to have served by the ZXTM in the event of loss of connectivity to or outage of web servers.

/usr/local/zeus/zxtm/conf/extra

Config Script

If you need to make changes to core ZXTM install the config script is located in

/usr/local/zeus/zxtm/

to execute type ./configure

Debugging

To view logs in real time for debugging

tail –f /usr/local/zeus/zxtm/log/errors

Hope this is useful

If like me you have spent most of you IT life working with a Windows environment you have never really had to consider the case that you write scripts in. The odd login script or batch file aside its not the mainstay of the work concentrates on GUI environments.

While working on a particular task recently I spotted this little issue with issuing a 302 redirect with traffic script.

Linux is case sensitive so login.aspx is not the same as lOgin.aspx

To this end its important that you consider case If you are using ZXTMs to terminate SSL and restrict access to resources served from a none Linux based web servers.

e.g.

This script looks for any URL containing login.aspx, signup.aspx, /thismustbessl/userdetailseform.aspx, /admin/ for the website www.website.net.

$url = http.getRawURL();
$host = http.getHeader (“host”);

if (($host == “www.website.net“) && (string.contains($url, “Login.aspx”)) || (string.contains($url, “Signup.aspx”)) ||

(string.contains($url, “/ThisMustBeSSL/userdetailseform.aspx”)) || (string.contains($url, “/admin/”))) {

http.sendResponse( “301 Moved Permanently”, “text/html”, “”, “Location: https://”.$host . $url);
}

So this script does what we need right? Wrong

If you request http://www.website.net/ThisMustBeSSL/userdetailseform.aspx the script matches all conditions and the redirect will be issued to make the site HTTPS.

However if you request http://www.website.net/thismustbessl/userdetailseform.aspx

The traffic script will not match and the page will be served as HTTP. Disaster!

To avoid this occurring a minor but crucial change is required. First do a string conversion on the url, I force the url to be lowercase but you could equally use uppercase if you wish. Then make sure that all of the strings you are comparing are also the same case (lowercase in my example). This will allows match regardless of the case that the original request is submitted as.

$url = http.getRawURL();
$host = http.getHeader (“host”);

$s = string.lowercase($url); # set $s to equal lowercase $url
$url = $s;  ~ now set $url to equal

if (($host == “www.website.net“) && (string.contains($url, “login.aspx“)) || (string.contains($url, “signup.aspx“)) ||

(string.contains($url, “/thismustbessl/userdetailseform.aspx“)) || (string.contains($url, “/admin/”))) {

http.sendResponse( “301 Moved Permanently”, “text/html”, “”, “Location: https://”.$host . $url);
}

Now everything is cool

Happy Days….

When you write traffic script make sure that you comment you scripts with plenty of debug information to facilitate testing.

Example:

$responseLocation = http.getResponseHeader(“Location”);
$body = response.get();
$responseCode = http.getResponseCode();

log.info( “Location at Zone A ZXTM is : ” . $responseLocation );
log.info( “Body is : ” . $body );

# Test for HTTP 302, Location is HTTP and body contains HTTPS
if (($responseCode == 302) && (string.startsWith($responseLocation,”http://”)) && ( string.contains($body, “a href=’https://”)) ){

# Rewrite location header
$responseLocation = string.replace($responseLocation, “http:”, “https:”);
log.info( “Location has been rewriten to : ” . $responseLocation );
http.setResponseHeader( “Location”, $responseLocation );

}

View Log

Then when you are testing your scripts SSH to the ZXTM and run the following command to view the log as events are written to it.

tail –f /usr/local/zeus/zxtm/log/errors

You can then view the log as you test to check that your script is behaving as expected.

Once complete rather than amending your script, which could potentially introduce bugs. change the logging level in the ZXTM config to not log info messages.

You simply turn this on and off as required for testing.

Happy Debugging

If you want to filter based on IP address you have a number of options with Traffic Script.You can filter based on subnet mask, file based white list and regular expressions.

Mask Based Filter

This example tests the remote IP against the entire 10.0.0.0 address range if the address resides within that range the connection is closed.

$ip = request.getRemoteIP();

if string.ipmaskmatch($ip, “10.0.0.0/8″)) {
connection.close( “500 Unauthorised\r\n” );
}

File Based White List

In this example we need to restrict access to specific area of the web site to specific staff computers. First of all you need to create a file in ZXTM install location if your using the default install location its

/usr/local/zeus/zxtm/conf/extra

You can create as many files as you like for different white list purposes. Following the Zeus Knowledge Hub example this file is called trusted_ips

$siteaddress = http.getHostHeader();
$rawurl = http.getRawurl();
$ip = request.getRemoteIP();
$trusted_user_file = “trusted_ips”;
$trusted_ips = resource.get( “trusted_ips” );

if (($siteaddress  == ( “www.website.com” )) && ( string.contains( $rawurl, “/payments/secure/” ))) {

# Check IP
if( string.contains( $trusted_ips, $ip ) == 0 ) {
connection.close( “500 Unauthorised\r\n” );

}
}

Regular Expressions

If like i did, you think this looks like ramblings of a mad math professor, stick with it. Its actually quite straightforward and very very powerful

The key to this is the reg ex
When you put the value you want to test in () the value is assigned to $1 thru $9

So this example has $1 and $2

string.regexMatch($ip, “([0-9]+)\\.([0-9]+)\\.[0-9]+\\.[0-9]+”);

This example has variables $1 $2 $3 and $4

string.regexMatch($ip, “([0-9]+)\\.([0-9]+)\\.([0-9]+)\\.([0-9]+)”);

We can then test the values of $1 $2 etc

e.g.
if $1 == 192 {
do something
}

This is taken from a script I produced that had to allow certain IPs within any /24 subnet from a within 192.4.160

Example

$ip = request.getRemoteIP();

string.regexMatch($ip, “[0-9]+\\.[0-9]+\\.([0-9]+)\\.[0-9]+”);

# log.info (“Matched the mask to 192.4.0.0/16″);
# Match 192.4.160.x to 192.4.161.x OR
# Match 192.4.8.x to 192.4.9.x

if (($1 >= 160 && $1 < 162) || ($1 >= 8 && $1 < 10)) {

#         log.info (“Matched at RegEx”);

} else
connection.close( “500 Unauthorised\r\n” );
}
}

A bit more on Reg Ex

The key to this is the reg ex

When you put the value you want to test in () the value is assigned to $1 thru $9

So this example has $1 and $2

string.regexMatch($ip, “([0-9]+)\\.([0-9]+)\\.[0-9]+\\.[0-9]+”);

This example has variables $1 $2 $3 and $4

string.regexMatch($ip, “([0-9]+)\\.([0-9]+)\\.([0-9]+)\\.([0-9]+)”);

We can then test the values of $1 $2 etc

e.g.

if $1 == 192 {

do something

}

ZXTM traffic Script conforms to the PRCE Regular Expression compatible library.
More info here http://perldoc.perl.org/perlre.html

Putting it all together: Example Script

This example is used to match a set of web sites host names and then test the client IP to see if the request is from a machine considered to be internal in this network. If its not internal the client is directed to a pool that requires two factor authentication.

$siteaddress = http.getHostHeader();
$ip = request.getRemoteIP();

if ($siteaddress  == ( www.siteone.co.uk ) ||
$siteaddress  == ( “www.sitetwo.co.uk ” ) ||
$siteaddress  == ( “www.siteone.net ” ) ||
$siteaddress  == ( www.siteone.com ) ||
$siteaddress  == ( “somethingelse.net” )) {

log.info ( “2FA Rule ” . $ip . ” Host = ” . $siteaddress );

if(string.ipmaskmatch($ip, “10.0.0.0/8″)){
string.regexMatch($ip, “([0-9]+)\\.([0-9]+)\\.[0-9]+\\.[0-9]+”);

# log.info (“Matched the mask to 10.0.0.0/8″);
# Match 192.0.0.0 to 192.150.255.255  OR
# Match 192.158.0.0 to 192.201.255.255

if (($1 == 192 && $2 < 151) ||
($1 == 192 && $2 >= 158 && $2 < 202)) {
#         log.info (“Matched at RegEx”);
pool.use ( “HTTP” );
} else {
# Address is External
pool.use ( “RADIUS” );
}

Enjoy

First off I tried to eliminate any ISA functionality that may cause this behaviour by design as I have ZXTMs in the environment I disabled link translation globally and checked that the HTTP filter on the listener wasn’t configured to manipulate response headers in anyway.

Here’s the problem (This is a complex local government network with many thousands of users at tens, if not hundreds of WAN linked locations).

The Web Server has a site that controls SSL. The client requests a URL that needs to be SSL. The web server issues a HTTP 302 redirect see below. Between the Internal Interface of ISA server and the External Interface the Location Header in the response to the client is modified from HTTPS to HTTP. this is bad and results in an infinite loop of request and 302 response. 

Request and HTTP 302 Response

Wireshark capture at Zone B ISA

We still have a HTTPS URL in the Location Header.

Wireshark Capture at Zone A ISA

And now we don’t have a HTTPS Location Header

Anyone who know why ISA would modify the Location header in the response please feel free to enlighten me . The ISA configuration is a basic firewall configuration with web publishing rule sending requests to ZXTM IP behind the ISA internal interfaces.

In a less complex deployment you would probably would not have this issue but this solution had ISA due to client requirements to use some of the functionality provided by ISA Server.

ZXTM traffic script to the rescue!

This is a problem that can be addressed in a number of ways. You could add a custom HTTP header which you checked once the response hits the external Interface on the ZXTM or in this example the response body also contained the full https URL so the traffic script below looks for this and modifies the Location Header if it finds a 302 response like this:
 

$responseLocation = http.getResponseHeader("Location");
$body = response.get();
$responseCode = http.getResponseCode();

log.info( "Location at Zone A ZXTM is : " . $responseLocation );
log.info( "Body is : " . $body );

# Test for HTTP 302, Location is HTTP and body contains HTTPS

if (($responseCode == 302) && (string.startsWith($responseLocation,"http://")) && ( string.contains($body, "a href=’https://")) ){

# Rewrite location header
$responseLocation = string.replace($responseLocation, "http:", "https:");

#log.info( "Location has been rewriten to : " . $responseLocation );

http.setResponseHeader( "Location", $responseLocation );
}

The traffic script is so powerful and this is why I love working with ZXTMs!

Happy Scripting….

Zeus Knowledge hub has an article here but I thought I would elaborate a little for the benefit of the Windows Admin’s

This assumes that you have a Windows machine on which to do the conversion.

1) Install OpenSSL

Windows binary here http://www.slproweb.com/products/Win32OpenSSL.html

2) Export Private Key from .PFX

Once you have installed openSSL

Do the following:

Copy your .PFX file to local file system on the windows machine you have installed OpenSSL on -

If you follow default install navigate to c:\openssl\bin\ and enter

openssl.exe pkcs12 -in <drive\path\name.pfx> -nodes -out drive\path\name.pem>

e.g. openssl.exe pkcs12 -in C:\cert\govuk.pfx -nodes -out c:\cert\PKgovuk.pem

Type the password for the PFX file

You should see a .pem file for the private key in your folder.

3) Export Certificate

Now repeat the process but this time use following syntax to export the certificate

openssl.exe pkcs12 -in C:\cert\govuk.pfx -nokeys -out c:\cert\Certgovuk.pem

You should see this:

and a new file

4) Import into ZXTM

Open admin console and navigate to catalogue \ SSL \ Server Certs

Select Import Certificate

Give your cert a name and populate the location of your cert and private key .pem files. Click “Import Certificate”

You should now see following.

It is more than likely that you will require an intermediary Certificate to complete the key chain.

5) Intermediary Certificate

If the Cert requires an Intermediary to complete the certificate chain do the following:

Download the appropriate certificate from the issuing Certificate Authority. In this example the CA is global sign

Cert is Here:

http://www.globalsign.com/support/intermediate/domainssl_intermediate.php

VeriSign here: http://www.verisign.com/support/install2/intermediate.html

And Thawte requires login here: http://www.thawte.com/roots/index.html

Download the intermediate certificate, this is usually via copying the cert from the web page and saving in a text file. Call the file intermediate.pem

Open the Imported SSL Cert and (scroll down) select install intermediate.

Populate the box with the location of the cert and then click upload.

You should see something similar to below

6) Finished

Test by navigating to the site and verify the certificate via the browser. The Certificate should be valid and display the complete key chain.

- FIN –