An Inside Look at the DRW Linux Services Team

I've been at DRW for a little over a year now and I've been doing systems administration, DevOps, and Linux kernel debugging for almost 25 years. Like most systems administrators, we're generalists on the Linux Services Team, but my primary focus is writing code to automate our systems administration processes. Because our trading desks and developers rely on infrastructure that is reliable, ensuring deployed Linux systems are in a consistent state contributes to DRW's success in the markets.

While my focus has consistently been on automation, that automation has touched a variety of different issues and yielded insights in how we can perform our jobs better and more successfully. Instead of only focusing on automating the provisioning and changes to systems themselves, modeling what, specifically, a change will do has increased the confidence and success of our changes. I have created tools to mine the existing data about a system's current and desired state and can present it in a succinct and meaningful way. I personally find a lot of satisfaction in improving me and my colleagues’ work by removing one more unknown and one more source of worry when they touch critical production systems. Automating change and provisioning work has been especially important with the shift to remote work over the last six months. With the Linux Services Team working remotely on laptops and often in spaces shared with children and family, any incremental improvement in hands-off automation of provisioning new systems or changing existing systems leads to a real reduction in stress and real increases in productivity.

Ensuring the consistency of a fleet of systems can sound like a deceptively simple, and consequently boring, job. However, when you multiply that across several trading operations with their own performance requirements and needs for what's installed and configured, it's a complex and interesting problem space. Additionally, with the firm running on physical hardware for key performance sensitive jobs (obviating the ability to utilize many of the modern "cloud scale" orchestration frameworks) it becomes a target rich environment that can constantly improve our automation processes. I find it enormously satisfying to take this complex task and turn it into a reliable and consistent process that users of our systems are less concerned with than the reliability of their refrigerator.

In many ways, being a Linux Engineer can look a lot like being a contractor that's responsible for paving a racetrack. But this is no ordinary paving job: the track has to be glass smooth, perfectly banked, meet the drivers' needs for corners and straights, and you have a very specific and narrow window of time in which you can pave the surface. There's no way you can accomplish this task without a high level of automation and a deep understanding of what the end user's needs are. The opposing constraints of a high-quality job done in a narrow window of time and the ability to conceptualize a problem space at a high level while having a very deep technical understanding are what keep this job consistently interesting. Being a Linux Engineer at DRW isn't anything like being stuck on the break/fix hamster wheel that systems administrators fall into; it's a real engineering job that requires a consistent, disciplined, creative, and thoughtful approach to complex problems.

It's no mystery that no matter how interesting the challenges, it's the people you work with that make or break your experience. Not only does everyone on the Linux Services Team constantly challenge each other to up their game with coding solutions to problems, but we also believe in providing the team with the resources to do so. We've always got each other's back and I know that no matter what, I can count on my colleagues to give me a hand. Additionally, we have a management chain that works to enable people to be constantly growing and it's a winning and highly satisfying combination. And finally, everyone on the team approaches the work with a perspective of humbleness and humor that really makes work enjoyable. One of the harder parts of working remotely due to the pandemic was that I missed the group dynamic and being around the team. Hopefully, given my propensity for dad jokes, they felt the same.

One of the great things about DRW's culture is the willingness to use the right tool for the job and to explore how new and different technologies can help us succeed in the markets. Many of the tools we use for managing systems that reside on physical hardware, Ansible and Python primarily, are widely used by other firms in the market, but we also have our own on-premises Kubernetes cluster because it fits the needs for certain users applications best. The diversity of tools in our toolbox and the evaluation of new and (possibly better) tools means that this is a job where I can constantly keep growing. This is the sort of job where I look forward to Mondays because it's incredibly engaging and interesting. I am always learning.

Over the years I've learned that in order to be a successful Linux Engineer, it's not so much about what you know in the moment, but what you can learn in the future. The ability to take your accumulated experience and feed that back into finding new areas to explore, new languages to learn, and constant incremental improvements is what drives long-term success. Just as important as staying hungry for constant improvement is listening. As an engineer it's often counter to my nature to listen to not only what people have to say, but what they don't say. My first instinct is that I want to jump in and start proposing technical solutions because that's what drives engineers. However, if I stop and listen with full intent, I not only end up with a more complete understanding of the technical challenge, I start to gain insight in someone else's perspective. That insight is key to a truly deep understanding of not only the technical challenge but what the other person or team truly needs to be successful.

It’s important to remember to have a sense of humor and perspective. It's easy to get lost in the weeds and, let's face it, not all technical challenges are incredibly satisfying puzzles. Even with all the advances in automation over the last 25 years or so, a fair amount of systems administration involves cleaning up messes. My teammates and I like to joke that being a systems administrator is 50% janitorial work and 50% murder mystery in which case you might also be the murderer. Sometimes that momentary break in frustration and the perspective of just how much better things are than a year ago is just what I need to make it through to the other side of the (hopefully rare) rough day.

Interested in joining DRW as a Linux Engineer? Head to our careers page to learn more.