Transform an existing architecture - 6 - define and transform culture

 Define and transform culture

This blog is part of the series "Transform a large-scale architecture guide", see:

• Before the landing
• The first 30 days, and the first half year
• Create the vision
• Get the first product started
• Execution and delivery
• Define and transform culture

Why Define a Culture?

Culture guides the daily behaviors of a team, recognizes and rewards people, identifies great, good, and bad behavior, creates an environment, and defines autonomy. A good cultural environment is necessary for the team to scale and repeat success. Defining culture guidelines also defines the rewarding and recognition system. The right leaders will naturally emerge based on culture, leading the product, technology, and people to success.

A culture that cannot meet the growth of the industry or customer requirements will eventually harm the company. Complaints like being unable to recognize talented people, reward heroes, or prioritize self-interest over customers will eventually be sorted out by competition.

What is Culture?

Culture is a set of habits that people understand and follow well. For example, customer success is a culture, and the team's focus should be on customer success. 

Good behavior that creates customer success creates a habit/culture. A single habit cannot guide all behaviors. 

Customer success itself defines the guiding principles for work, but other important things may be overlooked. For example, a platform team needs to be simple and consistent. Simple means the platform API should be designed consistency for easy understanding. The concept across the platform should be consistent so that customers don't need to think twice. 

Culture defines the hiring, promotion, rewarding system, and guiding principles. Amazon specifically asks and trains people to use them in various documents. My team uses this for hiring, recognition, and promotion.

Define Culture

The first 20 people hired define culture. Culture is not obvious in the first month of running a business. It becomes more and more obvious when interacting with various stakeholders and customers.

The founder of the team defines the culture by hiring the first 20 people. It wasn't clear to me when I started hiring, but all the people hired reflect the founder's values and instincts.

The right culture makes customers and businesses happy. Identify the early stage successes and formalize them. For example:

• Customer success: Customer success generates NPS, promoting and scaling business success.
• Simple and Consistent: Simple interfaces and consistent behavior and APIs simplify the customer onboarding process and make internal and external team communication easier.
• Open Collaboration: In a 2B business environment with many dependencies, finding opportunities to collaborate helps deliver requirements easier and simplifies communication upward or externally.
• Data-Driven and Geek Spirit: Products are built by engineers. Good quality systems are data-driven. Engineers are hackers and are not afraid of technical issues. They always find ways to improve the systems.

Transform Existing Culture

80% of behavior is already decided when a person is hired. The existing people have already adopted a culture, so culture transformation is hard and needs to be done in phases.

1. Hire the right people. Hire according to the founder's spirit, identify and decide across the process to ensure the right people are hired. It is a hard process. Interviewing 10 people may generate one candidate, and the market is also competitive. Get help from HR, hire through personal leads, and hire through friends.
2. Promote the right people. With the vision and products defined earlier, promote the right people with the right behavior. The old behavior that does not make the business successful should be discussed during 1:1s with serious outcomes. Promote and reward the right people with the right behavior.
3. Find positions for people with inappropriate behavior. Talk with peers and leaders to arrange open positions for them. For example, one Ops Engineer we worked with moved to another Ops team and was promoted to a leader because his behavior matched perfectly as an Ops engineer.
4. Formalize culture and guide through hiring, promotion, and recognition. Communicate with leaders and team members to build a transparent working environment to help everybody be on the same page.

Transforming an existing culture is challenging since there are many connections in between. Start with a commonly understood vision, create as many allies as possible, and then transform with strength.

Transform a large-scale architecture - 5 - execution and delivery

 

Execution and Delivery

This blog is part of the series "Transform a large-scale architecture guide", see:

• Before the landing
• The first 30 days, and the first half year
• Create the vision
• Get the first product started
• Execution and delivery
• Hiring, promotion, and performance evaluation

Ownership

Dividing a product into different milestones and executing them may seem like a straightforward approach, but it is a common pitfall. It is more important to have the right people with ownership to execute than the milestones themselves.

Recognizing the right people with ownership starts with trust in project/feature delivery. If the owner has a track record of on-time delivery and quality attributes, that person can be used as a role model to create an ownership culture. This may be the single most important thing for execution.

The ownership team culture is not easy to build, and project delivery should start from zero trust first. It should improve by tracking details end-to-end, building trust step-by-step, and then building momentum.

Communication

Conceptual and high-level designs set the direction for product delivery, but they need to be divided into different components. Each component will require collaboration among the APIs, UIs, and customer feedback. Things will evolve, and the product and people will need to change. What was designed at the beginning may not be exactly the same as the initial design.

The gap comes from communication. One-way communication is bad. Two-way communication and group communication are important. As an architect, fostering communication is vital to success.

Communication has style and content challenges. Documenting as many details as possible helps to bridge the gap. Take the time to write them down, as clearly and visually as possible. That document can be passed along from the beginning to the end, and it will help bring more context along the way. Sometimes things change, and documents become outdated, but the high-level design should still be the same. Key principles should last. Try to document key changes as appendices to the original design.

Communication is not only about high-level/detailed level design. It is also about code communication, which is the code-review process. Equal communication and bringing people along in the process help people learn and create programmatic programmers.

Fostering communication in a professional way and bringing people along the way creates an equal way of communicating. All the team needs is to create a fantastic product to help customers succeed. So, put personal ego aside and let's collaborate!

Meetings

As in Lean, Plan-Do-Check-Act is a four-step process for execution. For product development, once the initial design is settled, divide it into different iterations for execution.

Plan: Sprint planning is about setting the right expectations on task quality. Design, coding, and testing should meet the expectation. The planning should consider vacation time, holidays, release time, dependencies, etc. The planning should be from bottom-up and top-down. Bottom-up as for time estimation, top-down as for priority and definition.

Check: Review, testing, and feedback from customers can serve as checkpoints. During the development process, different checkpoints can be made, such as design review, code review, testing, and feedback. Finding feedback earlier benefits the product and the customers more.

Act: Retrospective or private communication can serve as action items. Retrospective is for team communication to have an open forum for discussing and improving as a whole. Private communication is for personal advice such as feedback, time management, quality issues, and performance evaluation.

Keeping Focus

Focus is about saying no. Different types of distractions during product development include production issues, bug tracking, escalations, and chaotic communications.

Production issues unfortunately affect the team's performance since customers come first. But capping that into a single person or half person helps. Find an engineer who is good at dealing with those and can also deliver. During sprint planning, consider that. During performance review, also account for that.

Bug tracking comes with a priority. High-risk priority should still need to be considered. During sprint planning, create another lane for that. Account for delivery and timeline schedule.

Escalations come from production issues and bug tracking. If they are not managed properly, they surely come with an escalation. Customer service is the first to calm down the customers. Communicate with direct timeline and schedule. Customers typically can resonate.

Chaotic communications should be delegated to one proper person. That person handles all the private communications and then optimizes by knowledge base, bots, etc.

Transform a large-scale architecture - 4 - get the first product started

 

Get the first product started

This blog is part of the series "Transform a large-scale architecture guide", see:

• Before the landing
• The first 30 days, and the first half year
• Create the vision
• Get the first product started
• Execution and delivery
• Hiring, promotion, and performance evaluation

Finding Priorities

When creating a vision, you need to determine the north star and overall issues in each domain area. However, with limited resources, it can be challenging to tackle all the problems. So, how do you prioritize which issues to address first and which have better return on investment (ROI)?

Conversations with customers can help prioritize concerns, but these conversations may be opinionated. The individuals you speak with are responsible for a business and focus on what's most important to them. While they can provide data and facts, it may be specific to their priorities and not the entire business.

Gathering data and facts from the bottom up can lead to an overwhelming amount of information. However, grouping them together can help identify the most important things for the systems to operate. These may not be the most urgent issues to address, but they are still essential.

Examining how customers and engineers spend their time is another way to gather information on how to improve their happiness. Understanding why they're unhappy and what they're spending time on can provide insights into priorities.

To find the right balance between the most important and most urgent issues, list out all the concerns, examine them with data and conversations, and then use your experience and judgment to prioritize them. List out the most important things and verify with leaders by having a group conversation with facts and data.

Once you've established priorities, allocate your resources in a laser-focused manner. Losing this focus will affect your timeline and patience from funding partners/leadership.

Defining Product Scope

The vision and priorities guide the project's direction, but the leaders have expectations for results within a specific timeframe. While you can define a perfect product that takes years to deliver, business leaders may lose patience and need to see results sooner.

Ultimately, the product should gain a certain amount of market share within a specific time frame. In my experience, for a 2B product, it can be divided into several milestones:

• Proof of Concept/Buy-in from flagship customer (3 months)
• Delivery of the first product and migration from flagship customer (6-9 months)
• Expansion of the product into the market to gain a certain amount of market share (9-12 months)

The Proof of Concept (PoC) can be a prototype or a simple website that showcases the experience. The PoC should define the flow of the experience and how it simplifies customers' experiences.

The PoC should define the scope of each experience and list out the things you will do in the beginning and the things you will NOT do in the first phase. Not defining it well can create high expectations that cannot be delivered later on in the expected timeframe.

The product delivery and migration should correlate well with the PoC. The requirements should be listed out with a risk factor, including the migration. The product delivery will also need to set up the timeline and expectations right, so there is a two-week or monthly reporting. This is the time to create trust and gain support.

Expansion of the product is an iterative process that requires a pipeline of customers, conversations, plans with the customers, and should be added as part of the product roadmap, iteratively.

High-Level Design

High-level design should focus on the domain-driven architecture. Each domain should either have its own logical or physical boundary. There will always be arguments on micro-services or monolithic services. In my experience, each domain or several similar related domains should be a micro-service.

Defining a micro-service will have its boundaries, including security, database, API, CAP, etc. Define each micro-service and what it should and should not be doing.

Defining the key algorithms/user flows that flow through the micro-services ensures major cases will be working well. Along with it, define the major testing cases to verify they are working excellently.

Defining the operating model for the major services, including failure case fallbacks and pre-defined standard-operating-procedures (SOPs) or automations.

Review them with the team all together, brainstorming on each one of them, and making sure everyone understands their responsibilities, scope-of-work, and delivery expectations.

Finally, make a delivery milestone for each one, work as a team, and get started.

Making Trade-Offs

An Amazon quote says "Is this a one-way door decision?" but it depends on the context. For a product owner, the ultimate question for making trade-offs is whether it is a key experience for customers at this phase and whether it correlates with the key concepts.

The key concepts and architecture define the boundary of the ultimate vision and product. The key concepts should be primitive and present from day 1. Adding/changing them will be expensive.

Then there is a domain area for each product to deliver, and there are domain concepts for each. The domain concepts are key experiences that cannot make trade-offs. Then comes the use cases and experiences, which can make trade-offs.

Customers always have ways to get things done, but the key concepts that will affect their core experiences, including security, cost, and simplicity, cannot be sacrificed.

The best way is always that things can be iterated and incremental. Even with small web-page customization, as long as the concepts and flow are there, as more and more customers onboarded, the small customizations will eventually change and it is a low cost.

Transform a large-scale architecture - 3 - create the vision

Transform a large-scale architecture: Create the architecture vision

This blog post is part of the series "Transform a Large-Scale Architecture Guide," which includes the following parts:

• Before the Landing
• The First 30 Days, and the First Half Year
• Create the Vision
• Get the First Project Started
• Execution and Delivery
• Hiring, Promotion, and Performance Evaluation

The Vision

A vision is a mission statement that should be precise, easy to understand, and memorable. Defining a vision is a collective thinking exercise that requires a lot of understanding, time, and effort. There is no one way to define a vision, but the following are necessary:

Be Open-Minded and Help People Solve Their Pain Points

Be open-minded and always think about win-win solutions. Software is created by humans, and any changes to the current software product will involve people. Communication with people requires mindfulness about the conversations. The goal is to have honest conversations, but conversations can easily go nowhere.

Be a multiplier by asking questions to enable things. The current business may already have ten systems running for over ten years and believe changing them would be too disruptive. Redefining them may not bring necessary business value. Instead, think about why you are doing the same thing. Put that thinking aside and ask questions about the challenges of the existing systems and how you can help.

Understand the Current

Research the current systems to understand their relationships, functions, technologies, and business values. Use an Excel sheet to list all the functions and business use cases. Abstract them into different domains and establish the relationships among them.

Understanding the current also means having conversations with people familiar with the systems. Learn about how they are using them, what the patterns are, the current challenges, and any ideas for change. The conversations about the current will be fragmental but are representative of daily life.

Think from the current business perspective and connect them through business metrics data. If there isn't any, try to find something similar or representative.

Think Outside the Box

The current systems are still running, but maybe not very well. There are backgrounds on the existing business, and it's not just the technical reason why it is still here today. Think outside the technical box to understand the organizational, business, and industry growing history part of it.

The industry is evolving every day, and there are well-established technologies and newly innovated solutions or tools. For the domains identified from the current, think about industry trends, innovations, or new ideas. As an experienced engineer, pick the right trends for the current business to invest in, work with, or create new ones.

Keeping track of the latest trends in research areas and the industry can also help to keep the mind fresh and know where to find more detailed information when necessary. Joining open research channels, podcasts, conferences, etc. regularly can be helpful.

Value Proposition

Once you understand the current and think outside the box, there are apparently things the business can do better, either through using new technology or just creating a better product to achieve better results. But what are the better results, and how can you justify them through data?

Collecting data through the current is not easy since, if they have already understood the current metrics so easily, why are they not starting to change? The metrics data may be buried in various issues, notes, emails, or manual works, but not in a systematic way. Finding a representative data that is similar can also work. Maybe through a customer survey, data mining on current notes, issues.

Collecting data from the industry means reading reports and surveys on the industry-leading companies. Market researching companies like Gartner provide insight into who the leaders are in each domain and why they are the leaders. The leaders are usually innovators who educate the market on what are the things and provide insights on how can they use their product to achieve better results.

Use the industry and current data to propose the values, collect feedback, and reach agreement to fund the projects.

Architecture Framework

Architecture is an abstraction of each domain and the relationship among them. There are layers from bottom to top and connectors in between. Defining an architecture can be easy or hard depending on the level:

• Don't talk about too many details, create new concepts;
• The architecture boundary should be stable, not changed by details;
• The architecture should be flexible so that each domain can be prioritized differently.

Create New Concepts

There are always many details. Try to scope them into different domains and name them with different concepts. Verify the concepts through different conversations, document them, and communicate over-and-over to see people's reactions.

The concept name should not be completely new that nobody understands at first hearing. Try to find the concepts through people's talkings so that they can match their expectations more easily. Keep explaining new concepts over-and-over means education costs at latter phases.

The concepts should not leave out the technical details, including protocols, access and identity, databases, security considerations. Don't talk about them. Instead, focus on the concepts and what the use cases are.

Stable Architecture Boundary

Architecture is for guidance on technology. It is a high-level abstraction of the system. It should cover all the cases researched before and will cover the cases according to the industry research. Each domain architecture is an abstraction of the current business and technology and should not move in the many years to come.

The architecture is also forward-looking. Any new changes coming should be easily extendable to the current architecture, which means that the core concepts are unified, stable, and also extendable. Change to core concepts is a disaster to the whole system, so the concept and life-cycle around it need to be well-communicated and stabilized.

Each business domain is composed of the business cases around it. The core concepts need to be loosely-coupled. Each interface for each business domain should be stable and backward-compatible. It should have its business perspective, and the design should focus on its perspective. Even if there is a problem around each domain, it should be easily replaceable without affecting other domain services.

The book "Clean Architecture" has a diagram to illustrate the relationship.

Flexible Architecture

The business priority can and will change. The things planned at the beginning can and will change. Having a flexible architecture that adapts to change will affect execution efficiency.

The architecture relationship among each domain should be loosely-coupled and abstracted through interfaces. There are patterns like Strategy Pattern and Abstract Factory that can help create replaceable dependencies. When a new service is not ready or partly ready, having an abstraction will keep the architecture evolving.

Each domain should be flexible, and the lifecycle of each domain should also be replaceable. So when some part of the lifecycle is not actually there, create a simple one with little effort to help the business grow.

At the end of the day, the vision is about thought determination, that the business is doomed to change in an iterative manner. The above things are things to consider, but the key is about determination and bringing people along. Even if things are not there at the beginning, things can still happen.

Transformation a large-scale architecture 2: The first 30 days, and half year.

Transform a large-scale architecture: The first 30 days, and half year.

This blog is part of the series "Transform a Large-Scale Architecture Guide." Please see the links below for the other parts of the series:

• Before the Landing
• Create the Vision, the First 30 Days, and the First Half Year
• Get the First Project Started
• Execution and Delivery
• Promotion and Performance Evaluation

After careful consideration, you have decided to join the team. The first 30 days are key to making a good impression on your new colleagues. People will have expectations and judgments, so take the time to get to know them, create a positive impression, and present yourself as they expect. Keep a low profile and don't rush to make judgments.

First 30 Days

As an architect, people naturally expect you to bring new ideas and directions. Talk to your peers and managers to see what they expect. Your job is to transform, so they expect you to challenge them, but do so in a cooperative manner.

Don't judge. Bringing freshness to the team is good, but not everyone wants to be challenged. Be respectful, kind, and empathetic. Ask questions and help them understand that you are here to help them achieve success.

Try to start with something small. It takes time to create a thinking framework for a grand picture, and lots of research is required. This is the most important job of the transformation, but it takes lots of time. Meanwhile, find a peer who needs something from you and partner with them.

Be mindful of what you say. Don't talk about things that people are worried about. Transformation in architecture also means changes to the business and teams associated with it. If there are team changes, it will be an organizational change initiated by the leadership and driven by the transformation. The architect's role is to enable the organization to do that. Let people see it. It's too early for the first year to even talk about it. The organization will arrange things more appropriately and gently.

Try to make as many friends as possible. Attend the team's typical activities and be a part of them. Identify people who may have a similar background.

First Half Year

The most important thing is to create a workable vision and roadmap. But it takes time, so don't rush. Talk on a weekly basis and set up a routine to talk. Note things down with any notebook software. Talk with ideas, learn the ideas through research, keep track of industry trends, and check the reactions from peers, partners, customers, and other leaders. Identify what is doable and what is not.

While creating a grand vision and workable architecture, the first 30 days have already presented some low-hanging fruit to work on. Get the existing team members to do them. Understand what the team is good at, what the culture is, what the working standards are, what the quality is, what the process is, and so on.

Hiring

In my experience, transformation always comes along with hiring. During the first half year, you have a chance to change the culture by hiring key people.

Hiring is one of the most important things in transformation. People are hard to change. As an HRBP said before, 80% of things are settled once people are hired, and only up to 20% can change. The culture can be transformed by hiring, but you will need to know what kind of people you need to hire. The market is big, and you can always hire many people, but try to hire more experienced people rather than fresh graduates. The fresh graduates will be transformed by the team instead of transforming the existing.

Getting Things Done

Getting small things done creates credit among your peers and is one of the key supports you will need during the journey. Leverage the current team members to help deliver. Work with them, see through the work, see what they are qualified for, and see what they are not.

Coaching people is one of the ways to change. There will always be people who want to learn, especially passionate engineers. One of the best things about transformation is that there are tons of coaching opportunities. Coach through challenges, through various work, give space to everybody, let them do it, and then coach on cases.

Create the Vision

There are always plenty of things to do, but besides hiring and getting things done, spare a big part of the time focusing on the big roadmap.

First, you will need ideas, lots of ideas, lots of workable ideas. The ideas come from learnings on the industry, outside research, notes, talking with various leaders, and research on existing systems. Then, formulate the ideas into a workable solution that needs connections. The more connections, the more complex the whole solution becomes, and there will be a lot of moving pieces.

Second, after getting the ideas and connections, you need to know where to start and how to start. Creating a stable boundary will be critical. You will need to create domains and set up connections with each domain. There are books like Clean Architecture that talk about it and the elegance around it. Creating an architecture can also mean working with legacy systems, so Working Effectively with Legacy Code and Monolith to Microservices teach you how to work on that.

Finally, in my experience, it is often harder than technology itself. The books often tell you how to work from a technical perspective, but not from a product and organizational perspective, which the latter articles will cover.

Large-scale transformation leadership opportunity? Be cautious.

# Introduction

During the past several years, I have had opportunities to act as one of the transforming leaders for a few businesses. The businesses are industry-impacting and impact millions to billions of customers directly or indirectly, so they are mission-critical and can easily be on the news.

The transformation journey can bring immense rewarding happiness, as well as personal and team pains. Those learnings have been with me for a while, and I want to document them for future references.

Looking back, the key to transformation is understanding the current business, culture, and execution procedure. From an outsider, the benefit is not to carry any legacy thinking, so you have an opportunity to define a new picture and realize it.  

I have failed (a company aka River) and successful transformations (a company aka Game). In this document, I will try to share them.

![](https://lh3.googleusercontent.com/6v_ZUqv-ZPhHroAqCYZzHZNdUr1uZrypgQN3L3PlxIYPNRNuPe18nJpxjmNDfzDQSxRho2mDVoJrKFxjhKhX33BXF6a4gMKhlP19ZNvlUXD7mSyGyNqaC568WqomW6eSxBVnOHMEqTWCw-2TxmrvgP5nI1k-6D47avJtp8ZHfXzRX5fTwfkp5SAx9Q =400x400)

The document will divide into several parts:
-   [Large-scale transformation leadership opportunity? Be cautious.](https://www.blogger.com/blog/post/edit/2928180620906399337/6853774654012328719#)
-   [Transformational challenge: Create a vision for everyone](https://www.blogger.com/blog/post/edit/2928180620906399337/6853774654012328719#)
-   More

# Before the landing
During the River journey, my former manager reached out. This is a leadership position to help him transform an industry-impacting business. He needed me there because of my passion, technical skills, and trust.

But I didn't realize, according to reports, that only 16% of the transformation succeeded. Statistically, the change typically fails.

Transformation is hard. It starts from business goals, then ends up in different projects. It begins with a small group and ends with the entire organization.  

In the first transformation journey, It didn't succeed. All the people the manager brought in ended up leaving the organization, including himself.

It is hard. This memo is trying to help and create a thinking framework to deal with all of these in a group. 

You can't do it alone. Nobody can.

# Understand the business
Transformation is about the current and the future of the business. Running a large-scale business is challenging. A change to the business means changes to the products and ultimately impacts the customers.Understanding the business both external and internal can help.

Understanding the products through external helps you walk through the industry impression, rate 1-5, and what the image is. The River transformation we are in is rated probably two according to hacker news and Reddit. Competitors in the industry are growing fast, and the competitor's performance sets the business goals in the planning process. A typical customer question is why the competitor supports A, but you are not.  

Understanding the product internally helps you understand why A is not supported. The River is about technical debt, so a production line could not produce fast enough. When there is an execution problem, It is a typical sign of a problem with the production line.

# Understand the leaders
The business is run by the leaders. Any business change will require agreements from the leaders; understanding and working with them is vital at the early stage.

Talk with people about their concerns about the business and what opportunities you can work with on their business concerns. Identify those concerns and tell them that your job is to address those. It helps you create trust and support for future vision and roadmap definition. In River, by then, most managers worried they would lose their jobs, and several leaders had already switched jobs, so making them stable was my manager's top thing. And that didn't end well.

Paying attention to different leaders' behavior also helps. 80% of the language is through the body instead of wording. Understanding their level of interest is essential for your time spent. People are motivated by self-interest, and people are more interested with politics typically does not have good understand on deep knowledge; people who want to get things done will not care much about politics. Be aware of that.

# Understand the current culture
People's behavior formed the culture. Leaders have a crucial influence on the culture; leaders may not be Managers/Tech Leads, but leaders could be one of the best influencers among the people.

Understand the culture of each business process and how the team behaves. In the River team:
* During the planning, the River team planned more on the minor incremental changes, but it ended up adding more and more technical debts; The bigger changes are bold but disconnected, not ground up;
* The debt is too much during the design session, and only old and experienced engineers can discuss details. We kept * talking about every detail and ended up ruining the design session;
* The legacy tech lead is incapable of designing large-scale elegant systems, reputed badly among engineers, and ended up developing systems that took years to deliver and never achieving the business goals;
* During the execution phase, the technical debts added up, caused deployment issues, and could not go out according to the timeline;

# Understand the legacy architecture
When the systems can not operate well, it becomes a management problem. Ultimately, management wants to achieve autonomy to make everybody's life easier.

However, there is no magic in the engineering world. Everything is code written by engineers. In the river team, the code base is 7 years old, and the engineers have switched jobs to different organizations. Many junior engineers joined the team 1 or 2 years ago. They are hard-working, fast to deliver and cause problems with the old systems.

Understanding legacy architecture also means understanding the thinking along the way, why they started the product, and what happened along the way to the coding, people, and business. The River team started the product to compete for the timeline to get the business, so they did it super quick, like 7 days. They created a business on top of that code base. All the things later keep reapeating the same logic.