wybaczcie mój słaby angielski - robiłem co moglem aby wypadło dobrze,
miłego czytania (artykuł jest w trakcie "rozwijania" tak więc jego treść może się zmieniać w czasie).
Configuring fault-tolerant XMPP enviroment using Cisco SLB.
at beginning: sorry for my bad English - i do my best to improve my language skills ;), note that this guide is still "in process of writting" so it can contain updates
Many people searching good solutions for reducing downtime of theirs xmpp servers due to failures or software/hardware upgrades. RFC provide partial solution: that is DNS SRV records. Unfortunately this is not good solution for now, here are few points that explain why:
- no client software support for SRV (there is no client that fulfill dns srv rfc). there are some jabber clients that claim to support srv records but practical tests showing that the support is weak or badly implemented. There are patched PSI builds as well as latest versions of Pandion, Exodus and few others but all of them not fully implement RFC. As for now there is only one known client that support SRV well - and in addition also supports weights - it is svn version of Tkabber.
- not all servers support SRV (server need to fulfill srv rfc in order to accomplish connection to alternate server, and this must be done with rules specified by DNS entries)
- SRV for now is general bad idea due to lack of DNS support. (there is few workarounds related to setting up DNS to use SRV - but still this are workarounds!)
- we have better solutions!
ok, so what we have? We of course have Cisco SLB! SLB stands for "Server Load Balancing" and it is used to balance traffic and to "spread" this traffic to server farms to reduce load on one particular real server. As a "side effect" SLB also can discover if one of real servers is operational or not (bingo!). That is so great, but what i need to use SLB? Well SLB is a feature of Cisco IOS and in some configurations it works with software (software mode) support or with hardware support (dispatched mode). You need to check if your router/switch support this feature, if not you can try to upgrade IOS. In this article i describe SLB that is used on Catalyst 6509 NEB and work in directed mode (also can work in dispatched but for simplifying this article i describe easier mode). Also i will not write here about using SLB in redundancy mode (that mean SLB works on two 6509NEB reducing risk of "single poinf of failure") as this would be beyond this topic.Ok, first we need to setup ejabberd nodes:
Follow this guide (Chapter 6: Clustering) to setup ejabberd. If you are building environment for large user base there are some Erlang features that could improve performance: kernel poll, threads and HIPE (high performance erlang). Although there are people saying that HIPE does not improve ejabberd performance at all, in general
test@node:~$ erl
Erlang (BEAM) emulator version 5.4.13 [source] [hipe] [threads:0] [kernel-poll]
Eshell V5.4.13 (abort with ^G)
If you have succesfully created ejabberd cluster, we can finally start to configure SLB on our switch. Note that this guide covers mnesia setup as database backend only and it is not considering any SQL backend, but it should not be any problem to use other DB backends as well (but this mean - if DB is not clustered - DB can be "single point of failure")
Take a look at this graph that we will follow to build our example (click here to get full size image):
We are going to build cluster containing 4 nodes: 3 public accesible nodes and 4th node that will act as "spare" node (clients will be allowed to access only when we need to down other nodes or load increases and we need to do more agressive balancing), also 4th node will be used for maintaining additional services for example: gateway transports, bots, etc. SLB need 2 VLANs, one for virtual servers, second for real servers (server farms). Wait! But how it works? The service is seen by the word as a single IP address - this address is a virtual server that is configured on Cisco switch/router. Behind vserver there is "server farm" consist with many "replicated" or clustered servers called "reals". Normaly those reals are not accesible from internet (that is another simple function of SLB - firewall), but in this guide i focuse on most simply deployment - so it bases on public IP addresses. When new connection arrives to vserver, SLB during decision process considers: with reals are operational, current load with connections, number of failures, weight and algorithm (round-robin or leaseconn), after that the new connection is directed to proper real, if connection is acknowledged (ACK) nothing special happens, but when there is no response from real (no ACK), SLB reassign this connection to other real and increases failed counter on first real. When particular threshold is acquired real is marked as failed. SLB also check if failed real "back to life" by sending some incoming new connections to failed reals - if connection succed, real is marked back as operational. Simple and effective...
So now we know how it works so let's setup vservers (i assume that you know basics of configuring cisco IOS). Look at show running-config:
!
ip slb vserver P5222
virtual 213.134.161.161 tcp 5222
serverfarm XMPP
inservice
!
ip slb vserver P5223
virtual 213.134.161.161 tcp 5223
serverfarm XMPP
inservice
!
ip slb vserver P5269
virtual 213.134.161.161 tcp 5269
serverfarm XMPP
inservice
!
as You can see we have 3 vservers. For every xmpp port we must setup one vserver. First line in every entry describe vserver name, second is its IP address, and tcp or udp port, third line is a name of serverfarm to with traffic will be directed (we use only one serverfarm for xmpp), and the last line shows if particular vserver is enabled or disabled by the configuration.
To configure vserver you must enter in configure terminal mode on IOS.
now let's see how reals configuration looks like:
ip slb serverfarm XMPP
nat server
predictor leastconns
!
real 213.134.162.162
weight 5
reassign 4
faildetect numconns 16 numclients 5
retry 30
inservice
!
real 213.134.162.163
weight 5
reassign 4
faildetect numconns 16 numclients 5
retry 30
inservice
!
real 213.134.162.164
weight 5
reassign 4
faildetect numconns 16 numclients 5
retry 30
inservice
!
real 213.134.162.165
weight 5
reassign 4
faildetect numconns 16 numclients 5
retry 30
no inservice
!
it is little "tweaked" configuration for specyfic xmpp traffic so it includes some extra options. First it is name of server farm, second line (nat server) is needed for "rewriting" src addresses to reals, it instructs SLB to rewrite src address to reals. Next line selects leastconns as algorithm for this farm. SLB utilizes 2 algorithms: round-robin and leastconnection. Round-robin means that new connection is directed to reals by the circular fashion, least connections means that new connection is directed to real that have fewest active connections. Now we have real configuration next. First is IP address of real, next line shows weight used by "selection" algorithm to calculate with reals are more "loaded" with traffic and where new connection must be directed. Refer to SLB manual for detailed instruction how to set up weight in various conditions. In our example all reals have the same weight. Third line defines reasign value. Reassign value is a number of unacknowledged TCP SYN that are directed to real server before the connection is reassigned to a different real server. In our experience default value(3) for xmpp is too low, so we have increased it to 4. Next is failedetect value that defines how many connections must be reassigned before real is considered as failed and the value numclients tells SLB how many unique connections failures are allowed before real is considered to have failed. Again our experience shows that this value must be tweked. In Your enviroment other values can be more proper. Retry - interval in seconds, to wait between the detection of a failure on the real server and the next attempt to connect to the server. Finally - inservice - means that this real is "active" in farm and ready for connections. Note that last real have this parameter set to "no inservice" - becouse we use that node (real) as spare server.
This is pretty much with simple configuration, if everything goes right you can now issue command: show ip slb serverfarms detail and you will see:
XMPP predictor = LEASTCONNS, nat =SERVER, interface(s) =
virtuals inservice: 4, reals = 4, bind id = 0
Real servers:
213.134.162.162, weight = 5, OPERATIONAL, conns = 415
213.134.162.163, weight = 5, OPERATIONAL, conns = 414
213.134.162.164, weight = 5, OPERATIONAL, conns = 415
213.134.162.165, weight = 5, OUTOFSERVICE, conns = 1
Total connections = 1245
Final notes:
- reals comunicate with eatch other by local switching, and they don't "see" vserver IP. so you must be carefull to use local dns names. For example - you must add proper entry to your /etc/hosts file if needed
- observe if all services on reals are matching those defined in vservers. If you define vserver that does not exist - connections will always be see as "failed" and your server farm will be marked as failed.
- if possible plan downing your servers "day before", by setting particular server as "no inservice" as this is better way then just unplug ethernet cable (users will be then forced to reconnect)
- adjust "keepalive"s in your server /proc/sys/net/ipv4/tcp_keepalive_
- if you don't use cisco you can still gain the same efect using alternative methods (software or hardware) like Linux Virtual Server. There are few open-source projects that aims to provide software load-balancing and fault-tolerant enviroment
- also you can try to play with other xmpp server software as well, but still for now: ejabberd is simply the best
- Read SLB documentation: there is lot of things to consider, this guide don't cover even tiny piece of them
- The example described abowe is almost exact configuration on with jabber.autocom.pl server works
that's it, if you see any errors, or want to ask, just make a comment or contact me: zbyszek(@)toliman.pl
