The Night Watch: Working the night shift at CERN

The Night Watch: Working the night shift at CERN

by Dr. Khoo Teng Jian

Illustration by Charis Loke.

Illustration by Charis Loke.

It is 3 a.m. on midwinter night, and I futilely wish for a nasi kandar and kopi tarik to keep me going. In their place, I have a couple of Swiss francs saved for vending machine coffee and chocolate, but the snacks have to wait for the 5 a.m. slump. The ATLAS Control Room (ACR) resembles a starship bridge at a lower level of alertness, with some individuals attempting to write analysis code, one of my neighbours watching a movie and another making conversation over Skype. We’re also nervously eyeing the shift leader’s banjo case, wondering when the instrument might come out to play. Welcome to the ACR late night special.

Being on call and taking night shifts is a standard part of a Medical Doctor’s life, but it is also not unknown to the less practical sort of doctor. In our case, however, we haven’t a stream of ailing patients to mend. Instead, our ministrations are reserved for the fragile titan that is the ATLAS detector. Our 7,000 tonne instrument is a complex camera/microscope, blasted 40 million times a second by energetic radiation and doing its best to record the tracks of minute particles as they emerge from the interaction point. Given this harsh environment, ATLAS’s vital signs are kept under the round-the-clock surveillance by half- and fully-baked PhD’s.

Most shifters can be thought of as semi-skilled workers, knowing just enough to be aware of problems as they arrive. When there’s something strange on the monitor, who’re you gonna call? Specialist counterparts to the shifters are the on-call experts, who have more hands-on experience with the detector systems. They have the know-how to resolve common issues, and are able to diagnose and repair less typical problems. They are also responsible for calibrating and upgrading the control systems. Experts usually visit during the daylight hours to perform these tasks, and are a source of pleasant company as well as specialised information about the detector. The ACR in the daytime can be like a hive, as various interested parties pass through to report or gather information about what needs doing and the odd tour party stares and strobes camera flashes. Night shifts are usually more quiet, being dedicated to data-taking.

During a data-taking run, which can last for more than 24 hours, shifters monitor the activity of detectors, flagging incidents where a particular component regularly reports errors and watching out for any problems with the flow of data. For example, an electronic component that develops a high level of noise can saturate the trigger system, causing useless events to be read out, wasting bandwidth and disk space.
Such components need to be “masked” out and ignored. As the rate of collisions (i.e., the instantaneous luminosity) changes over a run, the trigger “menu” needs to be updated to maximise the efficiency of data-taking. Checks on temperatures, coolant levels and other markers of detector health are also made routinely, whether or not the LHC is delivering collisions.

Checks on temperatures, coolant levels and other markers of detector health are made routinely, whether or not the LHC is delivering collisions

With sensitive sensors situated centimeters from sizzling streams of accelerated protons, what could possibly go wrong? Quite a lot. Every LHC experimentalist’s worst nightmare is a stray beam slicing through the silicon of a particle tracker, carving through circuit components and simply making junk of expensive and irreplaceable detectors.

Fortunately, our counterpart accelerator physicists in the CERN Central Control Room (CCC) are similarly working around the clock to keep the protons confined in their orbits. Indeed, the most important call to action in the ACR is the “machine people” announcing STABLE BEAMS, assuring us that the crossed proton streams will remain precisely placed in the detector’s hollow centre to deliver collisions.


A jubilant ATLAS team when the LHC achieved its first 7TeV collision, the highest energy  ever recorded for a particle collider. Photo: CERN.

A jubilant ATLAS team when the LHC achieved its first 7TeV collision, the highest energy
ever recorded for a particle collider. Photo: CERN.

From this moment, the detector is safe to switch on, and a flurry of activity ensues. Silicon strips and pixels are raised to high voltage and the trigger system kicks into gear, firing off signals to record data every time an especially interesting event is spotted. All the while, the human operators scan screens for signs of software or hardware malfunctions, transitioning from eagle-eyed to blurry-eyed over the course of an 8-hour shift. The buzz continues until eventually the fine focus of the beams degrades or a beam monitor spots an orbital deviation.

Then, the LHC dumps its cargo of protons, unceremoniously announced in the ACR by the sound of a toilet flush. A final set of checks is made to ensure that the beam loss has not caused any problems, and the detector is returned to standby, while the CCC refills the collider for the process to start anew.

Eventually, 7 a.m. rolls around, and we are relieved of our posts by fresher colleagues. Home is a 15-minute cycle away, past snowy fields and sleepy cows. As the sun rises and the world awakens, the night watch retires. Come 11 p.m. we’ll return as the relief force, another day passing in the ceaseless collection of inverse femtobarns.


ATLAS: A Toroidal Large Hadron Collider (LHC) Apparatus, the biggest particle detector experiments of the LHC.

Inverse femtobarn:  1 / (the cross-section of a uranium nucleus times 10-15), it translates to “enough data to make 21,000 Higgs bosons at the LHC with 8 TeV proton collisions”.

About the Author

Dr. Khoo Teng Jian is KL-born, Penang-bred, and an Old Free. A graduate of Williams College, Massachusetts, he completed his PhD in experimental particle physics at the University of Cambridge. His PhD thesis “The hunting of the squark” won an ATLAS Thesis Award in February 2014. As a member of the ATLAS collaboration, he searches for supersymmetric particles and investigates reconstruction techniques involving invisible objects. In 2013, he took up a Junior Research Fellowship at Jesus College, Cambridge. He can be contacted at [email protected] Find out more about Teng Jian at