ClusterFunk
May 29
I recently had to debug this tricky little problem with ISA 2006. ISA decided for some reason (that I am yet to get to the bottom of), that it didn’t like the Location response header received from the web servers and changed the header from HTTPS://hostname/url to HTTP://hostname/url .
First off I tried to eliminate [...] [...more]
Posted: under ISA Server, Zeus ZXTM.
I recently had to debug this tricky little problem with ISA 2006. ISA decided for some reason (that I am yet to get to the bottom of), that it didn’t like the Location response header received from the web servers and changed the header from HTTPS://hostname/url to HTTP://hostname/url .
First off I tried to eliminate any ISA functionality that may cause this behaviour by design as I have ZXTMs in the environment I disabled link translation globally and checked that the HTTP filter on the listener wasn’t configured to manipulate response headers in anyway.
Here’s the problem (This is a complex local government network with many thousands of users at tens, if not hundreds of WAN linked locations).
The Web Server has a site that controls SSL. The client requests a URL that needs to be SSL. The web server issues a HTTP 302 redirect see below. Between the Internal Interface of ISA server and the External Interface the Location Header in the response to the client is modified from HTTPS to HTTP. this is bad and results in an infinite loop of request and 302 response.
Request and HTTP 302 Response
Wireshark capture at Zone B ISA
We still have a HTTPS URL in the Location Header.
Wireshark Capture at Zone A ISA
And now we don’t have a HTTPS Location Header
Anyone who know why ISA would modify the Location header in the response please feel free to enlighten me 
. The ISA configuration is a basic firewall configuration with web publishing rule sending requests to ZXTM IP behind the ISA internal interfaces.
In a less complex deployment you would probably would not have this issue but this solution had ISA due to client requirements to use some of the functionality provided by ISA Server.
ZXTM traffic script to the rescue!
This is a problem that can be addressed in a number of ways. You could add a custom HTTP header which you checked once the response hits the external Interface on the ZXTM or in this example the response body also contained the full https URL so the traffic script below looks for this and modifies the Location Header if it finds a 302 response like this:
$responseLocation = http.getResponseHeader("Location");
$body = response.get();
$responseCode = http.getResponseCode();
log.info( "Location at Zone A ZXTM is : " . $responseLocation );
log.info( "Body is : " . $body );
# Test for HTTP 302, Location is HTTP and body contains HTTPS
if (($responseCode == 302) && (string.startsWith($responseLocation,"http://")) && ( string.contains($body, "a href=’https://")) ){
# Rewrite location header
$responseLocation = string.replace($responseLocation, "http:", "https:");
#log.info( "Location has been rewriten to : " . $responseLocation );
http.setResponseHeader( "Location", $responseLocation );
}
The traffic script is so powerful and this is why I love working with ZXTMs!
Happy Scripting….
Apr 22
This is based on Microsoft ISA Server 2006 SP1 the network configuration best practice is equally pertinent to ISA2004.
I have just battled for several days to get Multicast NLB (Network Load Balancing) set up on the external interfaces of an Enterprise deployment of ISA Server. For good measure I also wanted to [...] [...more]
Posted: under ISA Server.
This is based on Microsoft ISA Server 2006 SP1 the network configuration best practice is equally pertinent to ISA2004.
I have just battled for several days to get Multicast NLB (Network Load Balancing) set up on the external interfaces of an Enterprise deployment of ISA Server. For good measure I also wanted to have HP teaming on the interfaces to provide host nic resilience.
This post assumes you want to set up a ISA server with a Firewall network configuration rather than a single leg proxy setup.
The key to success is to configure you networks correctly
Firstly set up you teaming (if required) Microsoft confirm that NIC teaming is supported for Multicast and Multicast with IGMP configurations. Its actually supported for all NLB configurations its just that Unicast requires a change to the host MAC that may conflict with the teaming utilities automated configuration (this is how failure is accommodated by dynamically modifying the MAC of the passive nic in the team). I must clarify that this relates to HP NFT (network fault tolerant) team only.
HP Team Configured as below
The following is screenshots from my Laptop so you don’t see the teaming protocols in the screenshots (but that’s not a problem 
)
External Interface
Configure as follows:
Disable File and Print Services
Best Practice dictates that you configure you external nics with the default route (Gateway Address) but leave the DNS entries blank.
Also on the WINS tab disable NetBIOS over TCP/IP
You can leave Enable LMHOST lookup if you want to reference your External Nic’s of the other ISA servers in your NLB configuration
Internal Interface
On the Internal Interface enter the IP and DNS servers
If you’re using LMHOSTS for any reason on the External interfaces disable LMHOSTS lookup on your internal Interfaces to avoid name resolution conflicts.
In order to implement ISA integrated NLB on the external Interfaces you also need to enable Integrated NLB on the Internal interfaces also.
NIC Binding Order
Set NIC up like this
Inter-Array Communications
Given the requirement for NLB on internal and External interfaces it is strongly recommend that you have an additional Interface assigned for inter-array communication. It should be available for ISA servers in your array and includes Configuration Storage Server/s
In the image below you can see the “heartbeat” nic for inter-array communications.
I also name each interface including PCI slot and port this makes testing and hardware fault tracking considerably easier in the event of hardware issues.
Use ISA Tools (search for Microsoft KB article 938550) KB938550.wsf script to confirm configuration.
This is an example of the output
If you have problems establishing NLB I recommend that you try to establish NLB using the OS based NLB GUI first. Once you have been successful remove the OS NLB config and use the ISA Integrated NLB configuration wizard to establish NLB config.
Conclusion
Given a choice I would avoid utilising NLB for ISA server. In my experience it is problematic to set up and the feedback is both limited and unhelpful when trying to fault find.
My main gripe is that documentation around this subject is sketchy at best. ISA provides very poor error logs if integrated NLB fails to converge. For example the error I received stated “there is a local error”. Great!
While it can be adequate in the right environment, given the total cost of ISA 2006 Enterprise server solution, I would strongly recommend designing a suitable load balance solution into the initial design. If you are using ISA to publish web sites and you don’t need to take advantage of its excellent firewall capabilities then I would go for Zeus ZXTM over ISA every time.
Further Info:
Tool to remove NLB setting (should you have a problems) for ISA 2006 Enterprise
RemoveAllNLBSettings Tool for Internet Security and Acceleration (ISA) Server 2006 Enterprise Edition
http://www.microsoft.com/Downloads/details.aspx?FamilyID=0cfc0231-2000-488f-a879-de00087f7a46&displaylang=en
An update enables multicast operations for ISA Server integrated NLB
http://support.microsoft.com/kb/938550
Troubleshooting NLB
http://download.microsoft.com/download/3/2/3/32386822-8fc5-4cf1-b81d-4ee136cca2c5/NLB_Troubleshooting_Guide.htm
Q. Can NLB be used with network interface card teaming adapters?
A. Network interface card teaming solutions are offered by most major network adapter vendors and provide adapter fault tolerance (AFT). Network interface card teaming permits grouping network adapter ports for a connection to a single physical segment. If connectivity through one port is not working, another port is activated automatically. This operation is transparent to the operating system and other devices on the network
NLB can be used with certain teamed adapters, however, this requires careful coordination between NLB and the teamed adapter configuration software. Refer to Knowledge Base article INFO: Using NIC Teaming Adapters with Network Load Balancing May Cause Network Problems (http://go.microsoft.com/fwlink/?LinkId=18366), for details.
Using teaming adapters with network load balancing may cause network problems
http://support.microsoft.com/kb/278431
Network Load Balancing in ISA Server 2004 Enterprise Edition
http://technet.microsoft.com/en-us/library/cc302580.aspx
Excellent ISA server blogs
http://blogs.isaserver.org/shinder/
http://blog.msfirewall.org.uk/
Apr 01
While doing a ISA Server deployment recently I came across this tricky little problem:
Event ID: 12260 Source: Microsoft ISA Server Job Scheduler
This error may be due to a corrupted certificate or insufficient permissions to access the certificate.
Locate the reported certificate in the local machine store. If required, reinstall the certificate and its private key, or [...] [...more]
Posted: under ISA Server, Tool, Tips and Tricks.
While doing a ISA Server deployment recently I came across this tricky little problem:
Event ID: 12260 Source: Microsoft ISA Server Job Scheduler
This error may be due to a corrupted certificate or insufficient permissions to access the certificate.
Locate the reported certificate in the local machine store. If required, reinstall the certificate and its private key, or delete it.
http://www.microsoft.com/technet/support/ee/transform.aspx?ProdName=Internet+Security+and+Acceleration+Server&ProdVer=4.0.3443.594&EvtID=12260&EvtSrc=Microsoft+ISA+Server+Job+Scheduler&LCID=1033
Why?
This occurs because of the way that the certificate is applied. If you import an SSL cert from a .PFX file via the the method described below the problem occurs:
…"Place all certificates in the following store" should be selected, and below it, the Personal node should also be selected. Press "Next". Press "finish" on the next screen and your certificate has been successfully imported.
If you do this the cert is placed in the Personal node of the Current User not the Personal node of the Local Computer.
If you open the Certificates store MMC you will see the following:
This is also the case if you choose the “Automatic select option” option
The obvious thing to do to resolve this misplacement is to open the Certificate store, cut and paste the Cert from Current User into Local Computer Personal store. You open the Cert and everything looks fine the certificate chain all works and the certificate says that it has its private key. Great….
Then you go into ISA and configure your web listener and select your Cert only to find that ISA is not happy…
Notice that the Certificate is correctly installed according to the GUI, but ISA is not happy. If you look in the application log you will see an error 12660.
To resolve this is:
First delete the Certificate in the Local Computer Personal store
Then right click on the Local Computer Personal store and select Import
Navigate to your PFX file
Follow the wizard
Bingo
Here is an example errors
This is what you see if you allow auto placement or follow the instructions without installing via an import in the certificate store. Notice that the Private Key is correctly installed but the Certificate store is wrong.
This is what you see if you move the Cert to the correct Local Computer Personal store. The Certificate store is correct but the Private key now shows an error.
And finally how it should all look
The event id 12660 also mentions permissions. You need to check that the certificate store is accessible by the system account. In order to check this navigate to \Documents and Settings\All Users\Application Data\Microsoft\crypto check that SYSTEM has full control on this folder.
Further info
How to install and use certificates for SSL connections in ISA Server 2006
http://support.microsoft.com/kb/840614
Troubleshooting SSL Certificates in ISA Server 2004 Publishing http://technet.microsoft.com/en-gb/library/cc302619.aspx
Jan 17
I have recently completed a consolidation / redesign of an ISA2006 infrastructure.The resulting platform publishes multiple web farms both internally and externally. Two factor authentication for the external clients via Radius was a requirement. The internal clients were presented via an internal firewall cluster with the external clients presented via an edge firewall cluster with [...] [...more]
Posted: under ISA Server, Networking, Zeus ZXTM.
I have recently completed a consolidation / redesign of an ISA2006 infrastructure.The resulting platform publishes multiple web farms both internally and externally. Two factor authentication for the external clients via Radius was a requirement. The internal clients were presented via an internal firewall cluster with the external clients presented via an edge firewall cluster with ISA farm sitting between the two firewall clusters.
Like this:
There were a number of problems that need to be addressed and so I have decided to highlight these. Some are common to all load balanced environments such as application state. Others only arise with multi-tier firewall / load balancer environments such as routing/firewall spoofing issues.
Network routing
One of the key requirements business have is to measure the popularity of the web applications that they publish. This can be for a large number of reasons such as to establish the effectiveness of an advertising campaign, to track usage trends or even inform strategic decisions regarding the organisation future. The Information that is available from data recorded in the logs is therefore strategically and operationally valuable.
So we want the clients details recorded in the web server logs. Easy right? That’s done by default…..
Internet user requests a web page via the published IP hosted by the ISA servers. ISA Server passes the request through to the web back-end. The response is sent (to the client) as shown below. Everyone’s happy
Problems!
So now we consider the Internal users.
The INT VLAN has a static route that enables the internal client request to be routed to the ISA server EXT VLAN. The ISA Server publishes the request to the web servers. The web servers send the response to the client IP which in this case is a VLAN with a route via the Internal Firewall. the Internal Firewall process the response expecting the response to be routed to the ISA INT VLAN. The Firewall closes the connection. The response is considered to be spoofed traffic (see below) due to the mismatch in source and destination IPs. The destination port on the firewall for the response is different to the port that the request was received on.
Firewall Anti-Spoof checking
This mechanism protects against activity from spoofed or forged IP addresses, mainly by blocking packets appearing on interfaces and in directions which are logically not possible.
This diagram shows the routing problem
In order to resolve this issue, the Internal Client IP is replaced by the ISA Servers Internal NIC IP, NATing the client behind ISA servers NIC address.
So now when you look in your web server logs every request seems to come from your ISA server Internal NIC.
Immovable object meets irresistible force
So the business wants the data but the network topology prevents this from happening. What do you do?
You have several options
Don’t Log at the Web Servers
Use ISA Server logging capability to provide the information you need.
ISA server logs a huge amount of information (full list here) . ISA Server also logs to SQL Server and Microsoft provide example reports for SQL Reporting Services . However it maybe that you must get the client IP to arrive at the web server.
Modify the HTTP Header
The industry standard approach is to insert the X_FORWARDED_FOR in the http header. Using this method offer the ability to get the client IP recorded in the web server log as it is part of the HTTP request header.
ISA Server does not natively support this for two reasons.
1) X_FORWARDED_FOR is not a ratified standard. There is no Internet Engineering Task Force (IETF www.ietf.org) documentation.
2) The value recorded in the HTTP Header is easily spoofed and is not authenticated so cannot be relied on to actual represent the clients IP.
There is a plug-in for ISA Server 2004/2006 that inserts the value into the HTTP header.
http://www.winfrasoft.com/X-Forwarded-For.htm
And an article about how to use it here
http://www.isaserver.org/tutorials/X-Forwarded-For-ISA-Firewall-Track-Originating-Client-Web-proxy-Chain-IIS.html
Alternatives
There are other options but they may or may not be viable. If you are building the solution from scratch design out routing issues. Another option is to utilise functionality of other devices in you network topology. E.G. Cisco ACE (Application Control Engine) Load Balance modules can insert the Client IP into the HTTP request.
In this Infrastructure* I would remove the ISA servers and use Zeus ZXTM v5.1 instead. The Internal and External Firewalls provide the TCP/IP packet inspection, so actually reduce the ISA server farm to a Proxy / Web Publishing solution. This is ZXTM’s core capability and for this environment there is no comparison. ZXTM’s load balancing, traffic inspection and modification is far superior to ISA Server 2006 R1. ZXTM v4.2 and above automatically adds a value "X-Cluster-Client-IP" to HTTP Header which will be recorded in the web server logs.
* There are a number of reasons for this not just the Client IP. The next post includes more of these reasons
In the next post I will look at Load Balancing, Application State and DNS.
Added to my RSS reader
