scale

Learnings from "Seeing like a State" by James C Scott

Seeing like a State inspects attempts of states bringing “administrative order of nature and society” that’s led to disastrous outcomes. It is an interesting window into inspecting how large, people based systems work at scale.

High Modernism

The book inspects state's appetite to use “high modernism” to solve issues at scale. "High modernism is a form of modernity, characterised by an unfaltering confidence in science and technology as means to reorder the social and natural world."(1) There is less consideration given to history and practical knowledge held in the grassroots.

States employ these high modernist schemes to simplify, standardise and make things more legible (easier to catalog and understand). Simplification and legibility makes it easy to monitor and control outcomes. For example, cataloging land ownership makes for easier track assets and income, leading to more efficient taxation.

It is incredibly difficult to simplify how a community operates. These societal systems tend to have multiple hidden outcomes; they tend to be complex ecosystems. Forcing change solely around an outcome the state is focused can cause a lot of disruption. They can go against the natural order of things. Often, the result is failure.

Scientific Forestry

In 18th century Prussia, the government tried to determine the revenue potential a plot dedicated for timber yield. This measurement resulted in forests being optimised for lumber, leading them on a path of monoculture (1 type of tree), planted in a strict grid pattern. While it made for maximised yield initially, subsequent generations collapsed. This attempt to simplify nature ignored how complex a forest ecosystem is and its need for that complexity to achieve continued growth.

There are learnings here we can apply to large teams and companies. Teams are complex ecosystems too. They have many hidden outcomes that are not visible to "people at the top". Therefore, we have to be careful when we apply new processes to get specific outcomes or do reorgs to align to new direction. It might look great on a spreadsheet or a diagram (high modernist?) but does it work for the humans who have to adopt it?

Visual Order

One of the key points made in this book is this idea of visual order over system order. States (and its planners) appreciate visual order, which is mistaken for actual order in a system. If something looks visually simpler and easier to understand, it must be better! However, it is often the case that visually appealing macro order leads to micro confusion. Complexity has meaning in a large systems; their order requires more context and knowledge to appreciate. A small example might be how a jet engine works. It may look complex to the untrained eye. However, its complex design has order, which yields the desired outcome.

If you are a leader of a company or someone designing software architecture, it is well worth thinking about this idea of “visual order”. Are teams laid out in a way that is easy to collect reporting reflect how work actually happen? Is the architecture side stepping the complexity of the business, which in turn gives it a competitive advantage? Are we causing confusion on the lower decks?

Le Corbusier’s plan for Paris redesign: grids and perfect right angles. Read more about Le Corbusier and Jane Jacobs here.

Mētis

“Mētis” stands for “practical wisdom” in greek. The author talks through examples of every day innovations that come from having context and experience. Not all problems can be solved at scale because the conditions of these systems are unique.

Excerpt from the book:

While doing fieldwork in a small village in Malaysia, I was constantly struck by the breadth of my neighbours’ skills and their casual knowledge of local ecology. One particular anecdote is representative. Growing in the compound of the house in which I lived was a locally famous mango tree. Relatives and acquaintances would visit when the fruit was ripe in the hope of being given a few fruits and, more important, the chance to save and plant the seeds next to their own house. Shortly before my arrival, however, the tree had become infested with large red ants, which destroyed most of the fruit before it could ripen. It seemed nothing could be done short of bagging each fruit. Several times I noticed the elderly head of household, Mat Isa, bringing dried nipah palm fronds to the base of the mango tree and checking them. When I finally got around to asking what he was up to, he explained it to me, albeit reluctantly, as for him this was pretty humdrum stuff compared to our usual gossip. He knew that small black ants, which had a number of colonies at the rear of the compound, were the enemies of large red ants. He also knew that the thin, lance-like leaves of the nipah palm curled into long, tight tubes when they fell from the tree and died. (In fact, the local people used the tubes to roll their cigarettes.) Such tubes would also, he knew, be ideal places for the queens of the black ant colonies to lay their eggs. Over several weeks he placed dried nipah fronds in strategic places until he had masses of black-ant eggs beginning to hatch. He then placed the egg-infested fronds against the mango tree and observed the ensuing week-long Armageddon. Several neighbors, many of them skeptical, and their children followed the fortunes of the ant war closely. Although smaller by half or more, the black ants finally had the weight of numbers to prevail against the red ants and gain possession of the ground at the base of the mango tree. As the black ants were not interested in the mango leaves or fruits while the fruits were still on the tree, the crop was saved. (2)

Successful workplaces allow employees to innovate and solve problems they observe on the ground. Sounds weird, given that is very much the job of an employee. However, processes and constraints put in place can discourage people from applying ingenuity altogether. Couple of things to do:

  1. The culture of your workplace has to embrace individuals using judgement to sidestep guidelines and rules that doesn’t make sense. When people blindly follow processes, leaders must push for using common sense. This is difficult for leaders as they are more concerned with order and ensuring everyone follows the same set of rules. Best to have less rules :).
  2. We must fight our impulse to solve every problem for people we manage/mentor/advice. It is a leader’s job to anticipate issues and clear the path ahead for teams. However, fixing every issue is unnecessary. Let the professionals on the ground figure it out by encouraging a bottom-up approach. Make it okay for them to be makers (think → do). Instead, work on issues that are more widespread that attack your value system or product direction.

Conclusion

We do have to solve issues at scale. This book doesn't discourage that. However, it is well worth understanding the complexity of systems we change and be careful not to oversimplify. It is difficult to have a full picture of how or why an ecosystem works. There are always second and third order effects for all new ideas introduced. We need to evolve systems in a way that can leverage scale to its advantage while not destroying the hidden goodness that drive outcomes forward.

Seeing like a State was well worth the read, despite its, sometimes, tedious academic writing :). I left out a wealth of good ideas presented in the book. Hope this summary helps you carefully inspect complexity that exist in your workplace.



Notes

Couple of detailed reviews of the book:

Citations

  1. James C. Scott, Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed (New Haven, CT: Yale University Press, 1999), p. 4.

  2. James C. Scott, Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed (New Haven, CT: Yale University Press, 1999), p. 333.

Thoughts on “Scaling Technology”

I had the privilege to take part a panel discussion on “scaling technology” for go1’s offsite. It got me thinking…

Software products are (usually) written to meet product outcomes. Initially, these outcomes tend not to include deliverables like “used by millions” or “developed by 1000s”. As a product grows and matures, these concerns start to bubble to the top. Sustaining growth requires businesses to take on “scaling” activities. They help build a lasting product.

Scaling tends to be forced upon a team on 2 directions:

  1. As market adoption increases, there’s more inbound traffic and opportunities. Product team will start to pile on new features to improve conversion and stickiness. Organisation may acquire horizontal pieces of technology to integrate and grow the business. With such rapid growth, software applications starts to breakdown under its own weight.

  2. With more customers, comes more funding and interest from the wider organisation. That means more developers start to push code into the mix. Operations start to slow down as overheads, dependencies and complexity of collaboration increases.

Scaling systems will both help continued growth and reduce some of these issues. Often, that’s not enough to be successful. We also have to evolve people’s mindset to support the growth in business (i.e. scaling people).

Scaling systems

This is a large topic. There are lots of books about achieving scalable systems. Here, I’ll focus on some ideas I’ve seen teams miss out on.

Scaling systems is a lot about having a strong architectural direction, aligned to business strategy. Where do pieces of your system go and how do they fit in? “Highly cohesive, loosely coupled” pieces that support where the business is taking the product.

The architecture cannot be “too clever” and should be easy to reason with. Sometimes, we forget that we will have more people with diverse skill levels working in the system. With more people comes more misunderstandings and confusion. Simplicity does well here… as long as simplicity doesn’t adversely affect the competitive advantage a business has.

One mistake I’ve seen made over and over is that we tend to design for the problems we face today. Scaling requires imagination and creativity to identify and agree on a future vision. This has to be done with stakeholders from other disciplines. Tradeoffs and constraints applied for future has to be made together deliberately.

Think about a road vs a F1 car design. What changes and tradeoffs are made to achieve the outcome of going around a circuit significantly faster? F1 car only carries 1 person. It has to have extreme turn around time when swapping out modules like tires and pieces of the body. It’ll support lots of configurations and costs a fortune to maintain and improve one. Scaling is a lot about trying to figure out how you can turn your road car to a race car.

Architecture should be thought as an organic/fluid element of the product. There is little chance that a living product reflect any idealised architecture design. Some parts will lag behind while others will evolve rapidly. That’s the nature of large software applications. Systems are always in motion, much like the business itself.

Think about where “rubbish is collected”. Every system produces waste as more development occurs. Waste usually floats to the top of the stack because these parts tend to be more volatile. Can you isolate areas of high waste production so they can be renewed with low friction?

Example of this is edge services (or backend-for-frontend APIs). Most changes goes into the edge to drive changes in clients. Overtime, they are natural places for bloat. They need to be regularly refactored.

Lastly, any new architecture should have an incremental migration plan. Businesses continue to operate and make money. This means that rewrites are expensive and downright dangerous. Go incremental. If you are not convinced, read the big redesign in the sky.

Scaling people

Scaling people is much harder. Even when the technology is thoughtfully designed, it may not yield good results if people didn’t learn to think from a point of scale.

All teams and companies have an equilibrium: a “default” mode of operation. New systems and processes tend to devolve back to the state of legacy systems when nothing’s done to move existing equilibrium state. Many disregard this as an issue because it is the current playbook to company’s success. Problem is, to make more money, to have scale, you have to move your equilibrium.

Example... You may build a new CI/CD system that takes an application from monthly releases to 15min. It is hailed as a victory for how quickly issues can be resolved in production. However, team’s scrum process may dictate that sprints can’t change after they are set every 2 week. This leads to bugs sit in queue for 2 weeks before picked up anyway. Team may decide it is not valuable to release all the time since the testers are over worked with hourly releases. Couple of mishaps have led to some bad performance reviews. Slowly, things move back to monthly releases. Pipelines degrade because speed is no longer a priority. PR sizes go back up since no one’s trying to push atomic changes to production.

The “way of work” has a direction, even if no ones designed it to have one. The way people work is what yields the system we are trying to replace. So we have to pair the system change with a people direction change as well. These changes have to compliment each other so we get the most out these changes.

Continuing our CI/CD example... Team may move to a Kanban model (or swim lanes for issues). Testers have to be integrated to the process in a different way. A new support process with daily triaging and SLAs will push the team towards monitoring quality. Reporting on revenue from customer value delivered may motivate the team.

On the thinking front, individuals can end up in 2 places.

  • People who have seen high collaboration at smaller scale would long for glory days of the past and continually fight for it to return. They’d want to work with a trusted few and smash out features like they used to — “who cares about design systems and business domains?!”.
  • Others will create hard boundaries in their minds and never go past to make outcomes happen. A lot of “that’s not my job” proclamations and meetings about “ownership”.

Neither of these help to achieve scale and get shit done. Most leaders lean on continuing to maintain hard boundaries and tinkering with process. These tend to have low yield.

Changing mindsets and how people think is the ultimate challenge for an organisation. Everyone has to be given a path to mentally transition from where they are to where they need to be.

…I am a frontend developer who had full autonomy to prototype and release a new UI patterns to production. Now that we are larger, customers need to have a consistent set of patterns to work with. This will help them navigate the product better. It is still important to innovate so I should discuss the new pattern with a designer to understand wider concerns. I should pitch my prototype as an opportunity to gather data on customer behavior. It will help us decide if we should deploy this pattern across the product to have wider impact than my single concern…

Elevating people who reflect the mindset required to achieve scale can help. Allowing them to take up leadership positions and making decisions can guide the rest of the organisation to see new thinking in action. It shift the organisational equilibrium quickly.


There is a lot more to changing processes and mindsets at a company. I’ve spent the last 5 years pushing for a new identity of “Maker” within Expedia to aid its transition. I hope to be writing about it more in the future.

For now, I hope these thoughts will help you scale your company better.

Attributing value to teams in large organizations

In a large tech company, there are layers and layers of teams that contribute to the success of a product or service offered. Many of these layers tend to only have indirect impact on customers. Teams that have a direct relationship with customers tend to have a lot of power. They bear the burden of delivering perceived value, in exchange for larger investments.

As organisations grow, responsibility of focusing on critical parts of a product gets split up and assigned to various teams. Many of these teams would would be in the middle to lower layers, not having a direction relationship with customers. Their contribution often go unchecked and under-appreciated, as their value isn’t easily captured and accounted for. This starts to erode their understanding of the value they provide. Focus shifts.

Let’s take an example of a team tasked with creating a data analytics library for a website. Collecting analytics on a product is critical to improving it. How important is it compared to contributions from the frontend or services teams? Should the analytics library team focus their efforts on accuracy, library performance, or perhaps, code quality?

As different parts of an organisation starts to think about value very differently, it becomes much harder to innovate and continue to be competitive. The product itself starts to suffer.

The key thing to work on here is “attribution of work” at every layer. If the business is making 100M a year, what percent of that is thanks to the analytics library? The relationship is not direct, and therefore, no obvious. But it is a worthwhile exercise to understand attribution.

Once attribution is understood, we can share growth targets based on that. Teams at each level can start to work towards a common goal. This facilitates collaboration among teams. Decisions like prioritising backlogs and staffing investments starts to become easier.

If the analytics library team understands that they will be at least 5% contributors to the 20M growth target for online sales, they can start to power features towards the growth target itself. They can start to ask questions like, “would innovating a new feature enable teams to evaluate and capture market segments that wasn’t understood before?”

Speaking a common language of value is a hard thing. Understanding exactly how “the janitor helps put a man on the moon” is tough too Doing these hard things can lead to a much more cohesive working environment and innovation.