Open main menu

CDOT Wiki β

Changes

Enterprise Hyperscale Lab

525 bytes removed, 00:03, 2 August 2015
no edit summary
=== Power ===
Power to each cabinet is provided by two 115 volt, 15 amp, 60 cycle circuits that power two independent APC 1.5 kW rackmount power supplies. Each of the power supplies has a network interface for remote monitoring and control.
The UPSes feed two Raritan Dominion power distribution units (PDUs), mounted vertically up the sides of the cabinet at the back, plus a third 1u horizontal unit. The PDUs act like long power bars, but have control and monitoring systems so that the per-outlet current consumption can be measured and remotely monitored over the network, using snmp or http protocols. Each outlet can also be switched on or off under network control.
Where possible, devices have been configured with dual power supplies. This means that many of the devices in the EHL are plugged into both two PDUs and can continue to operate when power is cut to one of the two supplies. This permits the EHL equipment to be rewired without any downtime; it also guards against downtime due to PDU, UPS, or PSU failure.
Devices which are not configured with dual PSUs are connected to just one of the PDUs.
The total draw of the EHL equipment installed in the first cabinet as of January 2015 was approximately 1.8 kWh under load. The current power system can support a little over 3 kWh; the cabinet supports thermal exchange of about 8 kWh.
 
'''Cabinet 2'''
 
Similarly to the first cabinet, power is provided by one APC 1.5kW rackmount power supply. The UPS then feeds one Raritan Dominion PDU. The configuration of these pieces are near mirrors of cabinet 1 and thus they have the same capability to remotely control and monitor the status of each piece of the network. The PDU for cabinet 2 currently is utilized only by the AMD Seattle.
=== Environmental Monitoring ===
One of the PDUs in each cabinet is equipped with a string of three environmental sensors. These are laid out diagnonally across the EHL, so that the air intake temperature, mid-cabinet hot zone temperature, and air exit temperatures are monitored, as well as the humidity.
In January 2015, typical intake temperatures on the EHL were 25-27C and exhaust temperatures were 36-37C.
=== Networking ===
One At least one Cisco 24-port gigabit switch and one Netgear 24-port 10-gigabit switch are installed in the back of each EHL cabinet. The 10g switch provides switches provide both 10GBASE-T and SFP+ connections. Where possible, SFP+/DA (Direct Attach) copper cables and LC fibre optic cables are used where because they are simpler and less expensive than fiber optic cables, and yet offer much lower latency than 10GBASE-T connections (2 nS vs 2 mS - one million times less latency); other connections are made with fibre optic transcievers or 10GBASE-T copper connectiosn are connections as required. Devices which do not support 10 gigabit connection are connected with 1 gigabit or 100 Mbit ethernet.
The connection between the EHL cabinets is made by a fibre optic 10 gigabit connection.
Connections between the EHL LAN and the outside world are provided by a [http://www.compulab.co.il/utilite-computer/web/utilite-overview Utilite], a small ARM computer installed in cabinet 1. This computer acts as a an ARM64 [http://www.diablotin.com/librairie/networking/firewall/ch04_02.htm dual-homed host] that provides firewall, NAT, forwarding, DNS, and VPN endpoint services.
=== Storage ===
=== Terminal Server ===
Many of the computers installed in the EHL do not have video output (because they're are not intended for desktop applications). Most of these have a serial port; in many cases, this is a virtual serial port which is accessed using the ICMP IPMI SOL (Serial-over-LAN) or similar protocol on the network. Since this is not a TCP/IP protocol, the client must run somewhere on the LAN.
For systems that do not have a working ICMP IPMI engine, each EHL cabinet is equipped with a [[Cyclades Terminal Server]] which provides remote access to 32 serial ports. A remote user can connect to a selected port to monitor and control the connected system.
=== Display ===