This is based on Microsoft ISA Server 2006 SP1 the network configuration best practice is equally pertinent to ISA2004.
I have just battled for several days to get Multicast NLB (Network Load Balancing) set up on the external interfaces of an Enterprise deployment of ISA Server. For good measure I also wanted to have HP teaming on the interfaces to provide host nic resilience.
This post assumes you want to set up a ISA server with a Firewall network configuration rather than a single leg proxy setup.
The key to success is to configure you networks correctly
Firstly set up you teaming (if required) Microsoft confirm that NIC teaming is supported for Multicast and Multicast with IGMP configurations. Its actually supported for all NLB configurations its just that Unicast requires a change to the host MAC that may conflict with the teaming utilities automated configuration (this is how failure is accommodated by dynamically modifying the MAC of the passive nic in the team). I must clarify that this relates to HP NFT (network fault tolerant) team only.
HP Team Configured as below
The following is screenshots from my Laptop so you don’t see the teaming protocols in the screenshots (but that’s not a problem
)
External Interface
Configure as follows:
Disable File and Print Services
Best Practice dictates that you configure you external nics with the default route (Gateway Address) but leave the DNS entries blank.
Also on the WINS tab disable NetBIOS over TCP/IP
You can leave Enable LMHOST lookup if you want to reference your External Nic’s of the other ISA servers in your NLB configuration
Internal Interface
On the Internal Interface enter the IP and DNS servers
If you’re using LMHOSTS for any reason on the External interfaces disable LMHOSTS lookup on your internal Interfaces to avoid name resolution conflicts.
In order to implement ISA integrated NLB on the external Interfaces you also need to enable Integrated NLB on the Internal interfaces also.
NIC Binding Order
Set NIC up like this
Inter-Array Communications
Given the requirement for NLB on internal and External interfaces it is strongly recommend that you have an additional Interface assigned for inter-array communication. It should be available for ISA servers in your array and includes Configuration Storage Server/s
In the image below you can see the “heartbeat” nic for inter-array communications.
I also name each interface including PCI slot and port this makes testing and hardware fault tracking considerably easier in the event of hardware issues.
Use ISA Tools (search for Microsoft KB article 938550) KB938550.wsf script to confirm configuration.
This is an example of the output
If you have problems establishing NLB I recommend that you try to establish NLB using the OS based NLB GUI first. Once you have been successful remove the OS NLB config and use the ISA Integrated NLB configuration wizard to establish NLB config.
Conclusion
Given a choice I would avoid utilising NLB for ISA server. In my experience it is problematic to set up and the feedback is both limited and unhelpful when trying to fault find.
My main gripe is that documentation around this subject is sketchy at best. ISA provides very poor error logs if integrated NLB fails to converge. For example the error I received stated “there is a local error”. Great!
While it can be adequate in the right environment, given the total cost of ISA 2006 Enterprise server solution, I would strongly recommend designing a suitable load balance solution into the initial design. If you are using ISA to publish web sites and you don’t need to take advantage of its excellent firewall capabilities then I would go for Zeus ZXTM over ISA every time.
Further Info:
Tool to remove NLB setting (should you have a problems) for ISA 2006 Enterprise
RemoveAllNLBSettings Tool for Internet Security and Acceleration (ISA) Server 2006 Enterprise Edition
An update enables multicast operations for ISA Server integrated NLB
http://support.microsoft.com/kb/938550
Troubleshooting NLB
http://download.microsoft.com/download/3/2/3/32386822-8fc5-4cf1-b81d-4ee136cca2c5/NLB_Troubleshooting_Guide.htm
Q. Can NLB be used with network interface card teaming adapters?
A. Network interface card teaming solutions are offered by most major network adapter vendors and provide adapter fault tolerance (AFT). Network interface card teaming permits grouping network adapter ports for a connection to a single physical segment. If connectivity through one port is not working, another port is activated automatically. This operation is transparent to the operating system and other devices on the network
NLB can be used with certain teamed adapters, however, this requires careful coordination between NLB and the teamed adapter configuration software. Refer to Knowledge Base article INFO: Using NIC Teaming Adapters with Network Load Balancing May Cause Network Problems (http://go.microsoft.com/fwlink/?LinkId=18366), for details.
Using teaming adapters with network load balancing may cause network problems
http://support.microsoft.com/kb/278431
Network Load Balancing in ISA Server 2004 Enterprise Edition
http://technet.microsoft.com/en-us/library/cc302580.aspx
Excellent ISA server blogs
http://blogs.isaserver.org/shinder/
http://blog.msfirewall.org.uk/