Responding to an Unpredictable Pandemic Means Painstaking Capacity Planning
This month The New Stack is talking about — what else?— the COVID-19 pandemic and how it is affecting our teams and our tech. While we’ve definitely uncovered some common threads on the technology side, it’s often more interesting to learn how tech orgs are working to help get their teams through these trying times.
Capacity planning is one-way organizations have intentionally prepared for the worst and hoped for the best. From how teams now communicate from the comfort — or distractions — of their homes to what to do if your leadership becomes ill to how to create cross-company redundancy, today we learn how two teams have prepared as best they can be for the challenges we’re currently facing.
For Some Companies, the Tech Stabilizes as Usage Dramatically Changes
Renee Orser, vice president of engineering at NS1, says most of the changes it’s seeing aren’t technically oriented, as NS1 is built to be responsive to the way the internet runs and scales. The networking and traffic flow automation company offers a mission-critical service that has to deliver 100% uptime, no matter the external conditions.
“We are the center at much of the internet traffic at the moment, but the way that our technology is set up is that we are very flexible and adaptable. When internet traffic increases, the load in our platform doesn’t have to change,” she explained.
She said NS1 is intentionally positioned to be responsive not reactive.
“I think of reactive as not being able to anticipate some of the changes down the line. Responsive is having a wider variety of plans available to us,” which Orser says allows them to be flexible and resilient.
This includes weekly team “fire drills” or chaos engineering to prepare for the unknowns like internet attacks, disaster recovery and continuity.
Chaos Engineering w/ Kris Beevers
Shadi Rostami, executive vice president of engineering at Amplitude, which offers product intelligence tooling, says this company also seen an uptick as its app customers are trying to figure out changes in end-user behavior. If some of these apps can’t attract new paying customers right now, it’s even more important to understand user behavior to retain existing ones.
Amplitude is noticing some of its customers, like in food delivery, not only seeing a dramatic increase in users, but in different kinds of users. Their customer 8×8 has not only seen a 50-fold traffic increase, it has gone from companies having to videoconference across organizations to also enabling teachers and students to continue learning over them. 8×8 also uses Amplitude’s high-level regional information on user profiles to help determine how to shape their traffic and where to add more servers.
Amplitude hasn’t seen any downtime despite an increase in user sessions. Like what we heard from 8×8, Rostami attributes a lot of its ability to scale reliably to its base of Kubernetes and Amazon Web Services. As part of its capacity planning, they make sure it has a lot of bandwidth available. And when it makes a major change, it does it slower with progressive delivery.
Resiliency of Staff Is Equally Important
As the pandemic came over the horizon, Orser said NS1 reexamined all their already rigorous business continuity plans. As NS1’s 180 employees went from distributed across seven cities around the world — and about 40% remote — to 100% work-from-home, it looked to understand how to respond quickly, including how to pivot operations where people are now working from and when health issues come up.
One of the first things it did was to register and map out every home internet provider and region for each on-call engineer across the U.S., Europe and Vietnam, and set alerts to know if and when any outage happens to bring another engineer on board.
NS1 also enacted what Chief Operating Officer Brian Zeman described as “broad contingency planning at a personnel level.” From the CEO and CTO down, if a member of the staff gets sick, who is next? What if multiple key players are hospitalized at once? And it’s not about just understanding NS1’s systems but also cross-training any third-party apps the company uses like Zoom and Salesforce.
Some Roles Will Need a Complete Reworking
Zeman’s role covers all go-to-market strategy including all sales and marketing. He says his team bought some remote applications to gamify sales engagement — Troops and LevelJump — while they are at home.
But it’s not as easy as friendly competition. It’s hard to sell in times of extreme crisis, loss and economic downturn. Zeman says they have to, more than ever, take a human approach to sales.
“You can’t take the same approach to call someone about an opportunity when the country’s in lockdown. We cannot attach our tech to COVID. We have a #BuiltResilient campaign. We focus on injecting positivity and how our employees are being resilient versus directly trying to capitalize on COVID-19,” Zeman said.
They (remote) coached sales leaders and executives that yes they can still talk to their prospects about how NS1 can solve their problems, but they cannot call the “last week of quarter five times in quarantine.”
This first-grade teacher shows how we're #builtresilient by visiting students having a hard time and reading them stories from a safe distance #InThisTogether @FoxNews https://t.co/zpedEJnSkj pic.twitter.com/ZdX63ddabk
— NS1 (@NS1) April 23, 2020
Amplitude saw a drastic HR pivot when it recently acquired ClearBrains predictive analytics company. The first day the new team joined the company ended up being the first day of San Francisco’s shelter-in-place mandate, leaving Amplitude to dive right into 100% remote onboarding.
In order to meld the two teams and to make sure there is a good balance of work and camaraderie, Amplitude has created some remote traditions:
- Hosting weekly live trivia during lunch on Wednesdays.
- Hosting a twice-weekly “stretch & reset” session.
- Twice-weekly virtual yoga with their instructor who normally visits their office.
- Hosted a talk on managing stress and uncertainty with Mental Health Expert Dr. Jennifer Akullian.
- Hosted a live Q&A with two teammates who normally work remotely.
- Created a few new Slack channels — for example, one to share work from home tips, another for “cook-alongs” for colleagues to cook together.
- Hosted a virtual “be kind to a human week” where they sent each other anonymous notes of gratitude through a custom slack-bot, as well as amplified ways teammates are giving back.
- Hosted a virtual Nowruz celebration to mark the Iranian new year.
- HR managers are also actively talking about how to help parents who are doing double duty.
How to Create ‘Hallway Moments’ Online
Rostami said that Amplitude has always had a very face-to-face, co-located culture. In March, the team went rapidly went from colleagues with connections to China requesting to work from home to within a few days Amplitude requesting colleagues to WFH to then “No you cannot come, shelter in place.”
The first gap for them to overcome was the valuable hallway conversations that weren’t happening. Voluntary virtual lunches began. They created a Slack channel or thread for each topic. If the thread dragged on, then it’s a good time to hop a Zoom.
With a headquarters based in pricey San Francisco, some of Rostami’s engineers live with five other roommates. As soon as she saw an engineer calling in from their bed, the company created a stipend to allow teammates to buy and expense things that can improve their at-home productivity, like desks and monitors.
She said that working from home with young kids means tripling the workload. Amplitude provided sessions for managers to offer flexibility and empathy.
Enforced Work-Life Balance
Orser says she and her fellow managers are trying to focus on the power of one-to-one meetings instead of increased video calls.
“Some people love and some people hate to be remote. It’s important for people to have space to talk about what’s going on,” she said.
The most positive employee feedback the company has had so far is the promotion of balance. Prior to the pandemic, it was already promoting flexibility, like making sure parents block out school pickups and bedtimes, so the company can schedule around employees’ schedules. Now they need to double down on that message.
“The base assumption is that what you are doing for friends and family is respected and valuable, supporting family and neighbors,” Orser said.
Everyone is experiencing stress for different reasons and in different ways. And remote work is known to see people work longer hours. At NS1, they had to make sure to foster a business environment that allows for “almost mandatory” paid time off — without even having to ask.
NS1 is a sponsor of The New Stack.
Feature image by Gerd Altmann from Pixabay.
