Thoughts on “Scaling Technology”

I had the privilege to take part a panel discussion on “scaling technology” for go1’s offsite. It got me thinking…

Software products are (usually) written to meet product outcomes. Initially, these outcomes tend not to include deliverables like “used by millions” or “developed by 1000s”. As a product grows and matures, these concerns start to bubble to the top. Sustaining growth requires businesses to take on “scaling” activities. They help build a lasting product.

Scaling tends to be forced upon a team on 2 directions:

1. As market adoption increases, there’s more inbound traffic and opportunities. Product team will start to pile on new features to improve conversion and stickiness. Organisation may acquire horizontal pieces of technology to integrate and grow the business. With such rapid growth, software applications starts to breakdown under its own weight.

2. With more customers, comes more funding and interest from the wider organisation. That means more developers start to push code into the mix. Operations start to slow down as overheads, dependencies and complexity of collaboration increases.

Scaling systems will both help continued growth and reduce some of these issues. Often, that’s not enough to be successful. We also have to evolve people’s mindset to support the growth in business (i.e. scaling people).

Scaling systems

This is a large topic. There are lots of books about achieving scalable systems. Here, I’ll focus on some ideas I’ve seen teams miss out on.

Scaling systems is a lot about having a strong architectural direction, aligned to business strategy. Where do pieces of your system go and how do they fit in? “Highly cohesive, loosely coupled” pieces that support where the business is taking the product.

The architecture cannot be “too clever” and should be easy to reason with. Sometimes, we forget that we will have more people with diverse skill levels working in the system. With more people comes more misunderstandings and confusion. Simplicity does well here… as long as simplicity doesn’t adversely affect the competitive advantage a business has.

One mistake I’ve seen made over and over is that we tend to design for the problems we face today. Scaling requires imagination and creativity to identify and agree on a future vision. This has to be done with stakeholders from other disciplines. Tradeoffs and constraints applied for future has to be made together deliberately.

Think about a road vs a F1 car design. What changes and tradeoffs are made to achieve the outcome of going around a circuit significantly faster? F1 car only carries 1 person. It has to have extreme turn around time when swapping out modules like tires and pieces of the body. It’ll support lots of configurations and costs a fortune to maintain and improve one. Scaling is a lot about trying to figure out how you can turn your road car to a race car.

Architecture should be thought as an organic/fluid element of the product. There is little chance that a living product reflect any idealised architecture design. Some parts will lag behind while others will evolve rapidly. That’s the nature of large software applications. Systems are always in motion, much like the business itself.

Think about where “rubbish is collected”. Every system produces waste as more development occurs. Waste usually floats to the top of the stack because these parts tend to be more volatile. Can you isolate areas of high waste production so they can be renewed with low friction?

Example of this is edge services (or backend-for-frontend APIs). Most changes goes into the edge to drive changes in clients. Overtime, they are natural places for bloat. They need to be regularly refactored.

Lastly, any new architecture should have an incremental migration plan. Businesses continue to operate and make money. This means that rewrites are expensive and downright dangerous. Go incremental. If you are not convinced, read the big redesign in the sky.

Scaling people

Scaling people is much harder. Even when the technology is thoughtfully designed, it may not yield good results if people didn’t learn to think from a point of scale.

All teams and companies have an equilibrium: a “default” mode of operation. New systems and processes tend to devolve back to the state of legacy systems when nothing’s done to move existing equilibrium state. Many disregard this as an issue because it is the current playbook to company’s success. Problem is, to make more money, to have scale, you have to move your equilibrium.

Example... You may build a new CI/CD system that takes an application from monthly releases to 15min. It is hailed as a victory for how quickly issues can be resolved in production. However, team’s scrum process may dictate that sprints can’t change after they are set every 2 week. This leads to bugs sit in queue for 2 weeks before picked up anyway. Team may decide it is not valuable to release all the time since the testers are over worked with hourly releases. Couple of mishaps have led to some bad performance reviews. Slowly, things move back to monthly releases. Pipelines degrade because speed is no longer a priority. PR sizes go back up since no one’s trying to push atomic changes to production.

The “way of work” has a direction, even if no ones designed it to have one. The way people work is what yields the system we are trying to replace. So we have to pair the system change with a people direction change as well. These changes have to compliment each other so we get the most out these changes.

Continuing our CI/CD example... Team may move to a Kanban model (or swim lanes for issues). Testers have to be integrated to the process in a different way. A new support process with daily triaging and SLAs will push the team towards monitoring quality. Reporting on revenue from customer value delivered may motivate the team.

On the thinking front, individuals can end up in 2 places.

• People who have seen high collaboration at smaller scale would long for glory days of the past and continually fight for it to return. They’d want to work with a trusted few and smash out features like they used to — “who cares about design systems and business domains?!”.
• Others will create hard boundaries in their minds and never go past to make outcomes happen. A lot of “that’s not my job” proclamations and meetings about “ownership”.

Neither of these help to achieve scale and get shit done. Most leaders lean on continuing to maintain hard boundaries and tinkering with process. These tend to have low yield.

Changing mindsets and how people think is the ultimate challenge for an organisation. Everyone has to be given a path to mentally transition from where they are to where they need to be.

…I am a frontend developer who had full autonomy to prototype and release a new UI patterns to production. Now that we are larger, customers need to have a consistent set of patterns to work with. This will help them navigate the product better. It is still important to innovate so I should discuss the new pattern with a designer to understand wider concerns. I should pitch my prototype as an opportunity to gather data on customer behavior. It will help us decide if we should deploy this pattern across the product to have wider impact than my single concern…

Elevating people who reflect the mindset required to achieve scale can help. Allowing them to take up leadership positions and making decisions can guide the rest of the organisation to see new thinking in action. It shift the organisational equilibrium quickly.

---

There is a lot more to changing processes and mindsets at a company. I’ve spent the last 5 years pushing for a new identity of “Maker” within Expedia to aid its transition. I hope to be writing about it more in the future.

For now, I hope these thoughts will help you scale your company better.

Why Facebook is having a tough time regulating content?

It is a diversity problem.

Facebook sees its users as a singular global community, much like how we are all citizens of planet Earth. Facebook regulates by having a common set of rules it has decided as the community standard. This standard evolves and changes all the time as it is challenged with new and interesting scenarios.

Good background listen: RadioLab — Post No Evil Redux

As communities scale up to 1000s, 10000s, millions and billions, these standards start to break down.

Each of us hold a very different opinion of our world due our diverse backgrounds. Usually, we find ways to form groups and micro-communities to be with their like-minded counterparts. People in these communities start to congregate and solidify ideas around some key principles that guide their path in the world. Micro-communities strengthen the core beliefs of their members. While each individual may have slightly differing viewpoints, it is often overridden by their appetite to be in a community. Humans are, after all, communal creatures. This has allowed us to survive and dominate the planet for all this time.

For humans, the differences of their ideology is just as important. Their backgrounds and histories fuel them to appreciate and maintain the differences. These attributes help us grow and progress as people. This diversity of thought makes us creative and innovative. Therefore, it becomes important for us to support the idea of multiple micro-communities with diverse viewpoints.

What this really means is that there is no one standard we can apply to a billion people. If you look around, these diverse sets of standards are everywhere in our physical world. Whether it is countries states and districts or competing supermarket chains, people follow ideologies and split themselves up to identifiable groups. It helps them have a sense of belonging, self-worth and feel safe.

So, Facebook should stop maintaining their big list of opaque community standards. There is just no one way to define what is an acceptable level of nudity in the platform (among many other debates…).

They should move towards a model like Reddit’s where there is only a base set of company standards everyone adheres. These standards are just minimal expectations. Each of the micro-community (subreddit) moderators have the opportunity to set their own rules that build upon the base ones. This model appreciates the diversity in our community.

The problem no one has definitively solved is the one of allowing these diverse ideas to mix. How do we meaningfully facilitate the crossover of these ideas? Learning from each other is very much part of us growing as a community. For a platform like Facebook, it is also their business model. There is huge monetary value and opportunity in being able to inject new ideas into existing groups of people.

This is where simplistic metrics like “user engagement” starts to work against big platforms. A peaceful community is at medium engagement at best. Everyone is talking about new ideas and learning from one another. With higher levels of engagement, we might be looking at conflict points. Algorithms have to take more care; they are the spreaders (and catalysts) of ideas in these modern day social platforms after all. They have to work against their own appetite to engage users. Or perhaps, work as a set of algorithms that compete for balance of some sort.

If we inspect how moderation happens in the physical world, you can think of legal and justice systems as micro-communities too. Some individuals and groups respond to the calling of maintaining peace and safety. Platforms like Facebook shouldn’t be creating “a global court system”. It feels like a silly idea stemming from the belief that a community has a global standard. Instead, they should encourage and facilitate the creation of entities that uphold peace and safety. Their base standards should only allow to restrict poor business practices and guard against more universally accepted destructive human behaviour. They need to elevate groups that hold communities together and hold them accountable to the standards they set. Means for content moderation should be built on the back of these key community interactions.

We have to stop thinking of ourselves as one big idealistic western-valued society. We are not. This is working against our progress as a society. We are a diverse group, learning to exist together in a common place. Facebook should embrace the diverse nature of humanity.

Attributing value to teams in large organizations

In a large tech company, there are layers and layers of teams that contribute to the success of a product or service offered. Many of these layers tend to only have indirect impact on customers. Teams that have a direct relationship with customers tend to have a lot of power. They bear the burden of delivering perceived value, in exchange for larger investments.

As organisations grow, responsibility of focusing on critical parts of a product gets split up and assigned to various teams. Many of these teams would would be in the middle to lower layers, not having a direction relationship with customers. Their contribution often go unchecked and under-appreciated, as their value isn’t easily captured and accounted for. This starts to erode their understanding of the value they provide. Focus shifts.

Let’s take an example of a team tasked with creating a data analytics library for a website. Collecting analytics on a product is critical to improving it. How important is it compared to contributions from the frontend or services teams? Should the analytics library team focus their efforts on accuracy, library performance, or perhaps, code quality?

As different parts of an organisation starts to think about value very differently, it becomes much harder to innovate and continue to be competitive. The product itself starts to suffer.

The key thing to work on here is “attribution of work” at every layer. If the business is making 100M a year, what percent of that is thanks to the analytics library? The relationship is not direct, and therefore, no obvious. But it is a worthwhile exercise to understand attribution.

Once attribution is understood, we can share growth targets based on that. Teams at each level can start to work towards a common goal. This facilitates collaboration among teams. Decisions like prioritising backlogs and staffing investments starts to become easier.

If the analytics library team understands that they will be at least 5% contributors to the 20M growth target for online sales, they can start to power features towards the growth target itself. They can start to ask questions like, “would innovating a new feature enable teams to evaluate and capture market segments that wasn’t understood before?”

Speaking a common language of value is a hard thing. Understanding exactly how “the janitor helps put a man on the moon” is tough too Doing these hard things can lead to a much more cohesive working environment and innovation.

A litmus test for strategy

Recently, I was questioning why I felt uneasy about some strategy talks I’ve been having with my peers. While something felt off, I was unable to articulate the why. Things didn’t come together till my friend, Scott Horn, threw in this great heuristic. It is a wonderful litmus test if you are thinking strategically about your approach.

If the opposite of your strategy doesn’t exist, it is not strategy at all.

Imagine we made a Music app with the highlight feature being “Playlists”. In order to compete with many dozens of music apps out there, You start looking around. We have…

• Pandora making intelligent playlists out of music attributes
• Google making playlists out of gathered data
• Spotify allowing customers to share playlists with each other

Perhaps, our playlist solution needs to be “curated by experts”?

If our competitors are already doing this well, then it wouldn't be a great approach. It would not grant us any competitive advantage.

If a strategy is what all competitors are already executing on, then it wouldn't mean much. It would be “business as usual”. It wouldn't help the product grow as much.

---

Update: Having done more reading, this is possibly where these words of wisdom originated from: https://www.bridgespan.org/insights/library/strategy-development/roger-martins-unconventional-wisdom#.VRW6sZOsUYc

Wonderful read with far better examples!

Product development is like...

...driving a car.

Sometimes, the path is clear. You can see far ahead. You can go fast.
Other times, the road is windy. You have to take it slow.
Some roads lead nowhere. Some are shortcuts.

What’s important is to keep the destination in mind.
And enjoy the ride.

Do "basics" well

High performing teams are about interperson chemistry that is built on a foundation of the “basics”.

Take sports for example.

Learning to play a sport require you to learn the rules and develop basic skills to play. For soccer, it would be passing, tackling, lobbing and shooting.

When you join a team, there’s drills, runs and short set plays. These are repeated over and over during practice. During a full game, these disconnected pieces mix together to form fluid play. Play together with your team for long enough and magic starts to happen — team members anticipate each others moves to outplay opposition.

Drills, runs and set plays allow you to improve and build on the basic skills: passing, tackling, lobbing and shooting. They improve player fitness and execution during gameplay.

The act of continually working on the basics seem to lead to continued advancement. This is an interesting idea as it would be easy to assume the opposite — to advance further, one has to work on increasingly complex drills. I think it comes down to the fact that most complex things are built up from very basic elements. The more confident and efficient we are with basics, the more opportunity we have to put them together to do complex things.

Therefore, I believe, basics and chemistry form important attributes for any team.

Chemistry can be a complex thing to establish, with time, environment and the individuals themselves. It’s a whole other discussion.

It is easy to establish basics when talking about soccer. What are the basics outside the context of sports? I’ve been trying to figure out what forms the basics for a software engineering team. Is it design patterns or code reviews? Is it practicing readiness to production issues? Or perhaps, it is our ability to document and capture tasks in Trello? These seem to be the essential building blocks in a life of an engineer; solving problems by applying technical concepts into code, that outputs a product to deliver value.

Software is more than just code. At the heart of it, it is empathy and collaboration. It is a bunch of people, observing and acknowledging pain felt by themselves or others, deciding to take upon themselves to resolve that pain. Solutions doesn’t just come from one individual. It is the work of many, fusing their minds, disciplines and skills together to make something truly valuable. For me, the basics for a team are contained in these core values.

Basics has to be more around the idea of problem solving. That is all we do as a team, every day. Even our individual disciplines themselves are various ways dimensions of problem solving — research, design, engineering, analysis etc.

This idea became clearer when I recently read the book “The Culture Code” by Daniel Coyle. The book investigates the elements of highly successful groups. There is a great case study around Danny Meyer’s wildly successful restaurants. In a restaurant, how should one go about preparing staff for both conveying the right feel to customers and be able to react to new and complex issues that come up? It is impossible to cover all the bases and situations.

Danny uses the idea of “heuristics”. He captures the behaviours and his ideology in short, catchy phrases that are used in high frequency. Staff learn to apply these heuristics in every day complex situations, leading to very delightful customer experiences. Applying these heuristics seem to be the basics Danny get his staff to focus on.

These “heuristics” aren’t the same as rules. They are shortcuts for understanding and applying “guiding principles”. They convey a behaviour that is core to situations that arise in a restaurant environment. They allow staff to be thoughtful and creative, while being aligned towards providing customers with a unique, delightful experience.

Heuristics based in problem solving… these can be our basics.

Here are some ideas for heuristics I’ve been collecting with the help of others:

• Just start.
• Try and try again.
• Optimise for fast recovery.
• Act like it is your company.
• Coaching over management.
• Challenge assumptions.
• Understanding the problem is half the solution.
• Ask why.

“Just start” — we can apply this heuristic over and over in many occasions, whether people are stuck on a decision or unsure of estimates. It embraces the culture of a “maker” in having a bias for action.

Whenever someone asks me why we allowed a bug to go to production, I like to say: “Making mistakes is part of the job. We’d be better off if our system was optimised for fast recovery instead.”

We do workshops to help everyone understand how to write a good problem statement. Because “understanding the problem is half the solution.”

I believe that practicing these basics of problem solving can lead to powerful teams. They can provide the fundamentals that enable magic to take place.

Keep practising. Keep doing basics well.

Time for a reset

I have been writing my Squarespace blog for about two years now. Yet, I can never achieve a consistent writing routine. Months go by without any posts. Over the Christmas break, I thought about why this was the case.

While the blog represents my thoughts, they do not belong to a topic or theme. I don’t have a strong goal set to break droughts, or to police pieces that are quickly becoming novels. I have many abandoned posts in my iA Writer folder that have simply grown too large.

I’m getting organised this year. I “started with why”, like Simon Sinek prescribes.

So why does this exist? 

I believe there is a practical and meaningful way to live a good, healthy, balanced life in this technology dominated world.

Given that I am working in the industry, I have a unique insight to this crazy tech world. I want to explore and tell people about products and ideas that help me achieve a balanced life. I want to talk about how technology influences what I do.

Therefore, I am renaming my blog. It is no longer “Android with an Apple shaped heart”. It is a little more agile, little more relevant and a whole lot simpler. It is going to set the theme for the posts from now on:

Incrementally Better

Yes, it is a reset. If Star Wars, along with so many other Hollywood franchises, can do reboots, so can my blog. It’s gonna be awesome!

2 things your app can focus on to be successful

Apps that do really well on the App Store focus on PASSIONS & PRODUCTIVITY. The higher you rank in the scale on these two categories, the better chances you have of making a kick-arse app concept that resonate with its users.

People are passionate about all kinds of crazy things. What's important is that passions makes us very emotional. And everybody knows emotional hooks are the best way to sell something to a consumer. Your job becomes exponentially easier if you don't have to convince them why something will be of value.

The effectiveness of the 'passion-hook' increases as the reach and the prestige of the passion gets higher. For example, fitness is a great category to be in. Everyone's emotionally invested in it, so your market size is large. More importantly, people are willing to pay to get fit.

Productivity is why IT exist in the world. Instead of figuring out what 28912 x 92891 is in your head or on paper, you have a calculator. If you can save time, make life easier, and return the human back its lazy couch potato stage as quickly as possible from real work, you are doing a great service. This service can worth a lot of money.

People pay to make things go away all the time. So if you are designing an app to help people be more productive, you have a great chance to make it. The bigger the problem you are solving, the higher you can charge for it. If it targets companies instead of individuals, then there is even bigger bucks to be made.

Then there are really good apps that combine the two. Food apps like Posse or Urban Spoon are great examples. People are very passionate about what they eat, and what says about them to the people around them. They take pride in finding little gems around the city and recommending to friends. At the same time, they are extremely productive. It saves me from having to Google for places to get lunch from when I'm hungry and irrational.

Of course there are other angles to succeed by making an app. You can look at improving communications and social interaction. But these are markets that may not follow the general pay for a download model that well. They are also very hard to succeed in.

If you want to keep it simple, make something that target passions and productivity. Analyse where you stand in the two scales when you are at the concept stage. Bring features in to add value in both categories as you go on. Find your own little edge in these scales that make your idea better.

As long as you are helping lazy humans be extra lazy, and help enjoying what they rather be doing, there's a chance of success.

Why make an iOS app first?

This is a regular discussion that happens under every article on The Verge about some new iOS app. So far they are all flames thrown at each camp. This poster had a really good answer with good supporting facts. I thought it was worth sharing:

mfocazio

It’s not “gibberish” – it’s fact:

1. Android users spend less time with apps. http://www.businessinsider.com/android-users-use-apps-less-2013-6
2. Android users are less likely to pay for apps: http://phandroid.com/2013/07/19/free-apps-android-users/
3. Android users don’t buy stuff with their mobile devices as much: http://bgr.com/2013/12/02/ios-android-black-friday-online-shopping/

There is a meaningful difference, and if you’re trying to make money in the mobile world, you start with iOS.

Taken from the http://www.theverge.com/2013/12/5/5178432/best-new-apps-level

The only thing I'll add to this is the ease of building and maintaining apps due to a more consistent group of devices with users who upgrade to new versions of the OS regularly.

Don't mistake this for developers not succeeding on Android. That is not true -- plenty of developers make a good living out of Android.