The cluster is formed by four physically separated bunches of Intel-based PCs. The master node is a 1.0GHz PIII dual-processor PC named Orpheus. It was formerly a 1.0GHz PIII single-processor PC, but we assigned the master node rol to former node 08, provided that duals are a bit more reliable. This computer has four net cards. The first one is used to connect the whole cluster to the internet, and the other three to three of the four different net switches (the fourth switch is connected to one of the first three ones). The PBS queue system is organized in such a manner that different queues run in physically distinct computers, so each node is devoted to a certain queue, and only that queue.
It is important to note that our setup is not an optimized one, and even if it were, different users might have different needs. The queue/CPU biunivocal assignment, for example, may have unwanted features for some users (some queues can be empty while others are overloaded), but, as everything else in our setup, it does work for us. The fact that two switches are connected in cascade lowers the performance of the data transfer (vital in many parallel calculations), so some users might desire a better tuned connectivity for the nodes. Anyway, our measurements of net traffic load during parallel computing seems to point out that the speed limiting factor is definitely not the communication bandwidth, but the computation time in each node. For faster CPUs, or more communication-demanding calculations, a faster net could be needed, but for us it's a waste of money.
The cluster has 82 nodes, 18 of which are dual-processor ones, giving a total of 100 CPUs. A detailed breakdown is shown in the following table:
| Node number | CPU type | Speed | Queue | Added |
| 01-07 | Dual PIII | 1.0 GHz | Medium | 12-Dec-2001 |
| 09-14 | PIII | 1.0 GHz | Medium | 12-Dec-2001 |
| 15-20 | Dual PIII | 600 MHz | Short | - |
| 21-28 | PIII | 866 MHz | Mini | - |
| 29-44 | P4 | 1.7 GHz | VShort | 10-May-2002 |
| 45-60 | P4 | 1.7 GHz | Large | 10-May-2002 |
| 61-78 | P4 | 2.4 GHz | VLarge | 18-Dec-2002 |
| 100-104 | Dual PII and PIII | 450-500 MHz | Slow | 10-Feb-2003 |
As mentioned before, the different queues run in different CPUs. Each queue is designed to cover some needs of the group. It is worth noting that the queueing system has some mayor advantages over "interactive" job sending, specially for groups of users, and large number of available CPUs. With a well set queue, the user doesn't have to take into account which nodes are working or free. Just send the job to the queue, and it will enter the cluster when the appropriate nodes are free. With the time limits the queues have (all but VLarge and Mini), the user can expect some CPUs to get freed in a reasonable amount of time, while being encouraged to send the jobs to queues with settings appropriate to its nature (a big freq calc should go to VLarge, but a tiny NBO calculation could as well be done in Slow). Recently (Jun-03) we upgraded Mini to no time limit (from a former 24h limit), under the pressure of more really long queues needed.
We have currently 7 queues running, with the following specs:
| Queue name | Number of CPUs | Speed of each CPU | Peak GHz | CPUs per job | GHz per job | Time limit | TFlop per job |
| Slow | 10 | 450-500 MHz | 4.78 | 10 | 4.78 | 60h | 806.6 |
| Mini | 8 | 866 GHz | 6.91 | 4 | 3.46 | - | 299.3/day |
| VShort | 16 | 1.7 GHz | 27.12 | 4 | 6.78 | 36h | 878.7 |
| Short | 12 | 600 MHz | 7.16 | 12 | 7.16 | 72h | 1855.9 |
| Medium | 20 | 1.0 GHz | 19.99 | 4 | 4.00 | 96h | 1382.4 |
| Large | 16 | 1.7 GHz | 27.12 | 4 | 6.78 | 184h | 4491.1 |
| VLarge | 18 | 2.4 GHz | 42.59 | 4 | 9.55 | - | 825.1/day |
|
|
|