By David Boston, Director of Facility Operations Solutions for TiePoint-bkm Engineering and a Data Center World speaker.
If your data center operation is like most, you involve multiple departments in the installation and removal of computer hardware. This practice originated largely because vendors were initially used to perform the work. Companies providing network cabling services generally do not employ electricians nor offer electrical services. As data center owners increasingly moved cabling functions in-house, they continued to view power and network cabling as separate entities and structured departments accordingly.
A few organizations have chosen to break from this tradition, recognizing the majority of downtime events within a data center occur as a result of people working within computer rooms. They acknowledge errors are greatly reduced when the number of people involved in change activity is minimized. By conducting this work with a single department utilizing consistent procedures and training methods, risk is further diminished.
Error potential is high in the typical organization model employed for managing the data center operation. In general, written agreements defining which department is responsible for specific cabling related tasks are absent. The line where one department’s ownership of cabling tasks ends and another’s begins is usually blurred.
Facilities operations commonly owns power cable installation from the UPS to the power distribution unit (PDU) or wall panel and to the remote power panel (RPP). They normally install or hire a contractor to install the power whips to a location near the server cabinet or mainframe device. It is frequently unclear whether the facilities group, Information Technology group, or network group is responsible for connecting the power whip to the cabinet or mainframe. There is often confusion over who will make the power connection to the power strip (or in-rack PDU) within the cabinet and the physical connection to each server.
Information Technology and Network Operations departments most often perform installation of network cabling in a data center, from the telecom rooms through patch panels and out to server cabinets. If more than one group is responsible for elements of this work, the same potential for confusion exists as for power cabling functions.
Several factors compound the aforementioned risks associated with multiple owners and lack of clarity and accountability in cabling task ownership:
- Failure to minimize the number of people involved in performing each cabling function
- Failure to utilize clear and consistent procedures for each task
- Lack of a consistent, repetitive, site-specific orientation and training program for all involved
- Failure to work in pairs, preventing “read and repeat back” confirmations when executing individual steps of a procedure
One Fortune 100 company with multiple data centers chose to break from the traditional model for cabling work following a server downtime event they correctly attributed to the risks of multi-department cabling ownership. Their original multi-department approach lacked consistent training programs, did not recognize minimal technical skills needed, failed to adequately qualify employees, and often resulted in lack of accountability and finger pointing when a problem occurred.
This organization’s leadership considered the substantial investment they had made in fault tolerant infrastructure systems and realized their organizational model did not meet the same objectives. Risks associated with installation and removal of computer equipment in the multi-department model threatened to negate the resiliency their facility design had achieved. With their multi-department arrangement, dual power functionality could easily be defeated at the server cabinet.
David Boston will present "Achieving Continuous Operation Through Site-Specific Training" on March 14 at Data Center World in San Antonio. View the complete program here.
Once the decision was made to change, the company concluded a single department performing cabling tasks would deliver the greatest reliability. Managers determined it would be more efficient to train electrical technicians to install and remove network cables than to take the opposite approach. They dictated their cabling technicians must have a Journeyman Electrician’s certification, which includes five years of apprenticeship. Previous experience working in a data center environment is preferred.
Once hired, a new cabling technician is assigned as a helper to a more experienced cable technician for a period of four to eight weeks. During this time, the new employee will complete the department’s Level One training program, which includes three to four hours of classroom time and covers company and department philosophy, impact of downtime, department member roles and responsibilities, and orientation to infrastructure systems and cabling configurations. The new employee must score at least 95 percent on a written test and receive positive feedback from their peers and supervisor on their grasp of knowledge, as well as their interactions with others.
The new employee initially helps with cable fabrication for both network and power cabling installations. They learn to cut cables to prescribed lengths and to install connectors. Next, they are paired with another team member to remove cabling from the computer room environment. The trainee learns to verify the correct cable will be removed. Removal work is performed section by section, taking out no more than five foot lengths at a time, to reduce the risk of disturbing other cables.
After successfully mastering cable fabrication and cable removal processes, the new employee is paired with an experienced employee to learn the network cable installation process. This includes dressing the cabling neatly within the server cabinet, terminating the cable, and effectively labeling each cable with grid, switch, and port locations.
The final part of the orientation process involves fully training the new employee in the installation of power cabling.
Technicians are hired based on electrical experience and are taught network cabling tasks and related procedures on the job. At a single data center supporting over 2,400 servers, the cabling department is comprised of eight people, including one lead person for power cabling activities and one for network tasks. Roughly half the group is assigned to network tasks and half to power, though this varies depending on demand. New employees are designated for primarily power or network tasks based on proficiency demonstrated during their orientation process. Those demonstrating strongest skills are assigned to the power cabling function, where risk is greatest.
Cabling technicians for this company work as a sub-group within the Facilities Operations department, primarily dedicated to computer power and network cabling installation and removal. This keeps their focus on the critical cabling tasks, ensuring timeliness and task specific experience. They will normally not be assigned electrical and mechanical infrastructure support systems PM and repair work, which is conducted by another sub-group in the department.
In addition to the training program described, the cabling group relies on the use of consistent procedures for each of their critical tasks. As installation and removal activities are conducted, team members work in pairs for safety and to ensure they always apply the read and repeat back process before implementing each procedure step and to verify they are working on the intended equipment. This is commonly referred to as “pilot/co-pilot” behavior, due to its use by commercial airlines.
Only facilities department technicians have permission to open the raised floor space within this company’s computer rooms, where all network and power cables are located. Cable trays for copper cables are secured to floor pedestals twenty-four inches off the floor within the thirty-six inch space. Fiber optic cables are installed in troughs in the floor plenum and liquid tight flexible conduit is run along the floor.
Additional commendable practices include:
- Network cables cut to size prior to installation
- Network cables labeled on each end to designate patch panel location and work order number
- Power cables custom ordered to specific lengths
- Power cables color coded to designate separate power paths
- Cable manager brackets within cabinets prevent cable connection separation due to vibration
- Colored adhesive dots are placed next to breakers within PDUs to indicate powered on; powered off; reserved for future installation; etc.
- Panel schedules within PDUs are updated at the time a change is made
Because most cabling team members have an electrical systems background, the facilities manager has increased flexibility in offering infrastructure systems sub-group technicians additional help for a specific task. The ability to retaining facilities team members is enhanced because the occasional opportunity for an employee to transfer to the other sub-group provides individuals opportunities to take on new challenges.
This organization cites the reduction of human error potential as the ultimate benefit of their transition to single department ownership of computer room cabling. Their new organization model ensures minimal people are involved, common procedures are consistently employed, levels of experience and skill match the need, and all involved are clear on critical objectives and the methods required to meet them. Their successful record of uptime has confirmed the organizational change and consistent application of procedures and training has paid off.
