I had the privilege to take part a panel discussion on “scaling technology” for go1’s offsite. It got me thinking…
Software products are (usually) written to meet product outcomes. Initially, these outcomes tend not to include deliverables like “used by millions” or “developed by 1000s”. As a product grows and matures, these concerns start to bubble to the top. Sustaining growth requires businesses to take on “scaling” activities. They help build a lasting product.
Scaling tends to be forced upon a team on 2 directions:
As market adoption increases, there’s more inbound traffic and opportunities. Product team will start to pile on new features to improve conversion and stickiness. Organisation may acquire horizontal pieces of technology to integrate and grow the business. With such rapid growth, software applications starts to breakdown under its own weight.
With more customers, comes more funding and interest from the wider organisation. That means more developers start to push code into the mix. Operations start to slow down as overheads, dependencies and complexity of collaboration increases.
Scaling systems will both help continued growth and reduce some of these issues. Often, that’s not enough to be successful. We also have to evolve people’s mindset to support the growth in business (i.e. scaling people).
Scaling systems
This is a large topic. There are lots of books about achieving scalable systems. Here, I’ll focus on some ideas I’ve seen teams miss out on.
Scaling systems is a lot about having a strong architectural direction, aligned to business strategy. Where do pieces of your system go and how do they fit in? “Highly cohesive, loosely coupled” pieces that support where the business is taking the product.
The architecture cannot be “too clever” and should be easy to reason with. Sometimes, we forget that we will have more people with diverse skill levels working in the system. With more people comes more misunderstandings and confusion. Simplicity does well here… as long as simplicity doesn’t adversely affect the competitive advantage a business has.
One mistake I’ve seen made over and over is that we tend to design for the problems we face today. Scaling requires imagination and creativity to identify and agree on a future vision. This has to be done with stakeholders from other disciplines. Tradeoffs and constraints applied for future has to be made together deliberately.
Think about a road vs a F1 car design. What changes and tradeoffs are made to achieve the outcome of going around a circuit significantly faster? F1 car only carries 1 person. It has to have extreme turn around time when swapping out modules like tires and pieces of the body. It’ll support lots of configurations and costs a fortune to maintain and improve one. Scaling is a lot about trying to figure out how you can turn your road car to a race car.
Architecture should be thought as an organic/fluid element of the product. There is little chance that a living product reflect any idealised architecture design. Some parts will lag behind while others will evolve rapidly. That’s the nature of large software applications. Systems are always in motion, much like the business itself.
Think about where “rubbish is collected”. Every system produces waste as more development occurs. Waste usually floats to the top of the stack because these parts tend to be more volatile. Can you isolate areas of high waste production so they can be renewed with low friction?
Example of this is edge services (or backend-for-frontend APIs). Most changes goes into the edge to drive changes in clients. Overtime, they are natural places for bloat. They need to be regularly refactored.
Lastly, any new architecture should have an incremental migration plan. Businesses continue to operate and make money. This means that rewrites are expensive and downright dangerous. Go incremental. If you are not convinced, read the big redesign in the sky.
Scaling people
Scaling people is much harder. Even when the technology is thoughtfully designed, it may not yield good results if people didn’t learn to think from a point of scale.
All teams and companies have an equilibrium: a “default” mode of operation. New systems and processes tend to devolve back to the state of legacy systems when nothing’s done to move existing equilibrium state. Many disregard this as an issue because it is the current playbook to company’s success. Problem is, to make more money, to have scale, you have to move your equilibrium.
Example... You may build a new CI/CD system that takes an application from monthly releases to 15min. It is hailed as a victory for how quickly issues can be resolved in production. However, team’s scrum process may dictate that sprints can’t change after they are set every 2 week. This leads to bugs sit in queue for 2 weeks before picked up anyway. Team may decide it is not valuable to release all the time since the testers are over worked with hourly releases. Couple of mishaps have led to some bad performance reviews. Slowly, things move back to monthly releases. Pipelines degrade because speed is no longer a priority. PR sizes go back up since no one’s trying to push atomic changes to production.
The “way of work” has a direction, even if no ones designed it to have one. The way people work is what yields the system we are trying to replace. So we have to pair the system change with a people direction change as well. These changes have to compliment each other so we get the most out these changes.
Continuing our CI/CD example... Team may move to a Kanban model (or swim lanes for issues). Testers have to be integrated to the process in a different way. A new support process with daily triaging and SLAs will push the team towards monitoring quality. Reporting on revenue from customer value delivered may motivate the team.
On the thinking front, individuals can end up in 2 places.
- People who have seen high collaboration at smaller scale would long for glory days of the past and continually fight for it to return. They’d want to work with a trusted few and smash out features like they used to — “who cares about design systems and business domains?!”.
- Others will create hard boundaries in their minds and never go past to make outcomes happen. A lot of “that’s not my job” proclamations and meetings about “ownership”.
Neither of these help to achieve scale and get shit done. Most leaders lean on continuing to maintain hard boundaries and tinkering with process. These tend to have low yield.
Changing mindsets and how people think is the ultimate challenge for an organisation. Everyone has to be given a path to mentally transition from where they are to where they need to be.
…I am a frontend developer who had full autonomy to prototype and release a new UI patterns to production. Now that we are larger, customers need to have a consistent set of patterns to work with. This will help them navigate the product better. It is still important to innovate so I should discuss the new pattern with a designer to understand wider concerns. I should pitch my prototype as an opportunity to gather data on customer behavior. It will help us decide if we should deploy this pattern across the product to have wider impact than my single concern…
Elevating people who reflect the mindset required to achieve scale can help. Allowing them to take up leadership positions and making decisions can guide the rest of the organisation to see new thinking in action. It shift the organisational equilibrium quickly.
There is a lot more to changing processes and mindsets at a company. I’ve spent the last 5 years pushing for a new identity of “Maker” within Expedia to aid its transition. I hope to be writing about it more in the future.
For now, I hope these thoughts will help you scale your company better.
