Platform Engineering: How does it work

Introduction

Platform engineering is a Top Gartner 2023 IT Trend. There are different concepts in the industry, but how can these concepts and products help an enterprise?

This article will discuss platform engineering from an enterprise perspective.

What does a tech organization need?

Digital transformation has changed enterprises. Every enterprise needs IT investments. As the enterprise grows, the Tech department also grows. To operate the Tech teams efficiently, just like an enterprise requires ERPs and CRMs, the Tech team needs specified tooling to achieve operational efficiency. This is now called DevOps, but there is often a disconnect.

Tech is built by people, for people. People are humans and cannot be 100% available like a machine. To achieve the desired quality, efficiency, and cost, a systematic approach is needed to cover every aspect and work well with PM, Dev, Test, and Ops engineers.

Amazon, for example, uses Dev to cover Dev, Test, and Ops engineers' roles, sometimes the PM role as well, and learns from that. Then, it applies the learning into the product itself to close the loop. Internally, Amazon has a strong infrastructure and Tools platform to cover for the engineers, so the engineers can focus on the actual work for each, rather than environment settings, permission management, host replacement, etc.

Google built Borg to achieve consistency in cluster management and improve cluster utilization to reduce costs. Google also invented the concept of SRE and built SRE engineering to focus more on continuous delivery, monitoring, etc. Based on the Tools and Infrastructure internally, they built and open-sourced Kubernetes. Kubernetes has gained tremendous popularity across the industry and changed Tech.

Amazon and Google are the whales, but you can also see examples from Netflix, Airbnb, Uber, etc.

Why does a tech organization need platform engineering?

The public cloud market share is expected to be 500 Billion in 2022 and 600 Billion in 2023(according to Gartner report). While it has gained tremendous popularity and redefined the IT industry, it has also created a tremendous challenge for DevOps. The infrastructure and tools used before have changed to Multi-cloud. The complexity has caused the developer efficiency to be lower, cost to increase, and quality to vary.

Platform engineering simplifies the build experience of new products. When a company is young and growing at its baby stage, it keeps trying to find a product-market-fit. At the growing stage, it matches multiple use cases and customers adopt it with a similar expectation on latency, performance, scalability, and experience. One way to achieve this is to use the same engineers or the engineers who have done this before. Another way is to bake the engineering experiences into tools and platforms, so products gain the learnings easily.

Platform engineering simplifies internal governance. Software organizations are product-lines with innovation. They innovate constantly and build different product lines. The dynamism from a product perspective is encouraged. The governance requirements typically land specifically in one organization, called platform, compliance, and as of engineering DevOps, it is now called platform engineering, with an emphasis on being platform-driven, but still focusing on the engineering metrics like requirement delivery time, deployment failure rate, software bug count, etc.

Platform engineering Concepts

The software industry is good at innovating new concepts, but they all have a purpose and history.

The Internal Developer Portal (IDP) is to build a standardized portal experience. With the emergence of micro frontend architecture, it has made it easier to build templatized and decoupled frontend architecture. Netflix shared their experience regarding Paved Road, and Spotify open-sourced backstage, which matched the enterprise internal websites demand.

Infrastructure focuses on operations, but the practices vary for company type. For the cloud companies, it means the hardware supply chain, standardization, virtualization, cluster management, etc. For the companies that rely on the cloud, it means to provide simplified, standardized, and secure access. Examples include Kubernetes and their variations, infrastructure-as-code, etc.

Productivity Tools focus on engineering output, with a focus on code lines output, code review count. Sometimes it also provides release tools, for example, Bazel, compiler tools, and software deployment tools. It embeds the software best practices standards within the tool to accelerate the development process.

How does it work for an enterprise

Enterprises focus on the quality, speed, and cost aspects of engineering.

On the quality part, the productivity tools embed the software standards. For example, Google used Code Review Certifier to review the code changes, with the tooling enforcing that checklist. The industry has multiple practices including LORE, SPACE to focus on different metrics, but the goal is the same, and the method will vary.

On the cost front, it is either the people or infrastructure cost. With a centralized infrastructure organization, it has the ability to control the infra cost, but to not impact the businesses, there is typically a negotiation process and a top-down OKR to drive it.

On the speed of innovation, the productivity tools in Google invented Bazel to accelerate the build speed. Netflix invented Spinnaker to simplify the deployment process, and Today there are AI tools to help generate code and tests.

In the end, this space is not new, but the mindset has evolved time-to-time. With the new AI innovation in the space, we are likely to see more.

DevOps Transformation at Tencent Infrastructure Services

 To memorize the two year journey from 2021 to 2023.

Summary

Tencent's Infrastructure Services (TIS) owns a critical part of the Chinese internet (such as WeChat). As time flies, operating the business becomes more painful using the legacy embedded model. To transform the operations, we built the internal DevOps platform (EasyCloud) to solve the challenge using software.

TIS architecture is one of the most technically complex parts across the company. It has built core COS (similar to S3), CBS (similar to EBS), and CDN for the whole Tencent business. TIS's business has both a public and private cloud. The public cloud runs a partnership with Tencent Cloud with a focus on the dataplane side.

Both the technical and business complexity have posed pressure on both engineering and DevOps, leading to the pains of efficiency. The leaders of the business want to make a change.

Over the past two years, TIS built its internal EasyCloud to unify and automate the operations. At the end of the 2-year journey, the VP said, “The things I did not see change in the last 10 years, I see a change in the last 2 years.”

Legacy architecture and challenges 

TIS started as a storage service, and the complexity of architecture, coupled with rapid growth, required Ops engineers to collaborate deeply. To support this growth, the ops teams were embedded into the storage business, a strategy that proved to work well.

Figure 1: Legacy Ops Embedded Model empowered the growth also leads to tooling silos and fragmentation.

During more than 15 years of growth, as TIS scaled its business, the embedded model scaled accordingly. As a result, the overall business faced the following challenges:

• A 6:1 dev to ops ratio: The public cloud business operations demanded a higher release frequency, a larger customer base, and a 10x increase in zone-based geolocations. Due to the current fragmented tooling and human-driven operational model, more business operations required more ops engineers.

• 30% deployment failure rate: The deployment best practices were tribal knowledge held by experienced subject matter experts due to the tooling fragmentation. A new Ops engineer and a complex deployment could easily lead to deployment failure.

• Low deployment standard parity: The continuous deployment platform had been rebuilt four times, and there were three deployment standards released before. During a customer conversation, one customer asked a question: “When did the deployment standard actually land consistently?”

Current Architecture

We achieved a 10:1 engineering to ops ratio by building the EasyCloud platform. This platform allowed us to build a suite of services and an ecosystem to automate deployment, chaos management, policy enforcement, and more. Enabled ops model transformation. 

Figure 2: EasyCloud enables an automated operating standard across businesses, transforming the operating model.

• The Product Catalog and EasyCloud Portal lay the foundation for the EasyCloud ecosystem, facilitating transformation. The EasyCloud Portal serves as a unified entry point, offering insights and tools for daily use.

• The Continuous Deployment Platform introduces a new CD platform with an embedded deployment standard and a pluggable architecture to execute deployment workflows at scale.

• The Ecosystem and other platform services, such as Chaos Engineering, are built from day 1 based on the product catalog. As the platform proves successful, we continue to build more platforms, enhancing business efficiency, including observability and build platform.

Product Catalog & EasyCloud Portal

The Product Catalog (PC) constructs a tag-based Configuration Management Database (CMDB) for approximately 5 million instances worldwide. It establishes a unified and modernized view for all business operations. Engineers with over 5 years of experience in the field within the business have expressed that 'it has achieved what they dreamed about before.

Figure 3: Product catalog synchronizes data from existing systems and builds a foundation to enable an ecosystem.

Millions of instances of data synchronization. The TIS business had integrated the cluster and application launching process deeply into its own systems. The synchronization incrementally syncs the data (application, instance, and tags, etc.) at scale, builds appropriate indexes, and performs anti-entropy for data accuracy.

The CMDB is a tag-based service. It has built the batch and dynamic tag-based query with pagination to support legacy wildcard query use cases. The CMDB separates the primary and backup, that the primary is for write and backup for read.

Continuous Deployment 

Continuous Deployment (CD) deploys to millions of instances globally for both the public cloud and private cloud. A modern-tool-based deployment improvement has accelerated the deployment success rate to one quarter per business, three times faster than AWS.

Figure 4: Continuous deployment empowers the standardization and flexibility at scale.

For the new CD, it enabled 100% deployment standard parity. The Workflow Definition Document (WDD) defines a standardized schema and builds 34 default workflow step execution plugins to support blue/green deployment, approval & notification, staggered deployment etc. It minimizes the deployment standard parity cost, with benefits on ease of use and scalability.

70% to 100% deployment success rate improvement. 

• 6% reduction in system failures. The core of the CD is a workflow execution service that builds workflow allocation, idempotency, isolation, and delay tolerance to dramatically reduce service-related failures.

• 10% reduction in failures. The standardization of the instance deployment template and improved failure handling both reduce and tolerate partial instance failures. The flexible workflow orchestration supports tag-based arrangements to enable use cases like hardware-type-based blue-green deployment.

• 8% reduction in human errors. The new CD supports Subject-Matter-Experts (SME) to embed their experiences into the system. They are empowered to define their own standards, enabling any operator to operate safely without breaking the deployment.

• 4% enforcement of quality checks. The system utilizes the PC to unify the data from CI and checks testing and version information as mandatory. It also enforces a double deployment check process (both the OP Leader and Dev Leader), the streamlined process leads to stable expectation on deployment.

4.8 out of 5 satisfaction. On the ease-of-use side, the WDD enables a flexible UI-driven orchestration, with a 200 ms latency at p99. On the system reliability side, the powerful execution engine provides control to pause and resume reliably. The overall service now boasts a 99.95% availability compared to 95% before. The simplified experience and dedicated on-call schedule contribute to a superior support experience. Requirements and feature requests are managed using our sprint and bi-weekly release process, providing a predictable expectation.

Ecosystem & Other platforms

As the CD proved to be successful, when given a chance, more systems were rewritten to build on top of the platform from day 1. Chaos engineering leverages the PC to perform scoped operations based on instance tags. The Policy engine service leverages PC to control the pace of scanning the operational environment to ensure production is safe.

The observability platform builds a unified metrics and event store to provide an application-centric and unified view to check the application health, handling petabytes of monitoring data per day. The build platform utilizes Bazel to offer an incremental build experience, with customized support for security and integration of the ecosystem.

Transform an existing architecture - 6 - define and transform culture

 Define and transform culture

This blog is part of the series "Transform a large-scale architecture guide", see:

• Before the landing
• The first 30 days, and the first half year
• Create the vision
• Get the first product started
• Execution and delivery
• Define and transform culture

Why Define a Culture?

Culture guides the daily behaviors of a team, recognizes and rewards people, identifies great, good, and bad behavior, creates an environment, and defines autonomy. A good cultural environment is necessary for the team to scale and repeat success. Defining culture guidelines also defines the rewarding and recognition system. The right leaders will naturally emerge based on culture, leading the product, technology, and people to success.

A culture that cannot meet the growth of the industry or customer requirements will eventually harm the company. Complaints like being unable to recognize talented people, reward heroes, or prioritize self-interest over customers will eventually be sorted out by competition.

What is Culture?

Culture is a set of habits that people understand and follow well. For example, customer success is a culture, and the team's focus should be on customer success. 

Good behavior that creates customer success creates a habit/culture. A single habit cannot guide all behaviors. 

Customer success itself defines the guiding principles for work, but other important things may be overlooked. For example, a platform team needs to be simple and consistent. Simple means the platform API should be designed consistency for easy understanding. The concept across the platform should be consistent so that customers don't need to think twice. 

Culture defines the hiring, promotion, rewarding system, and guiding principles. Amazon specifically asks and trains people to use them in various documents. My team uses this for hiring, recognition, and promotion.

Define Culture

The first 20 people hired define culture. Culture is not obvious in the first month of running a business. It becomes more and more obvious when interacting with various stakeholders and customers.

The founder of the team defines the culture by hiring the first 20 people. It wasn't clear to me when I started hiring, but all the people hired reflect the founder's values and instincts.

The right culture makes customers and businesses happy. Identify the early stage successes and formalize them. For example:

• Customer success: Customer success generates NPS, promoting and scaling business success.
• Simple and Consistent: Simple interfaces and consistent behavior and APIs simplify the customer onboarding process and make internal and external team communication easier.
• Open Collaboration: In a 2B business environment with many dependencies, finding opportunities to collaborate helps deliver requirements easier and simplifies communication upward or externally.
• Data-Driven and Geek Spirit: Products are built by engineers. Good quality systems are data-driven. Engineers are hackers and are not afraid of technical issues. They always find ways to improve the systems.

Transform Existing Culture

80% of behavior is already decided when a person is hired. The existing people have already adopted a culture, so culture transformation is hard and needs to be done in phases.

1. Hire the right people. Hire according to the founder's spirit, identify and decide across the process to ensure the right people are hired. It is a hard process. Interviewing 10 people may generate one candidate, and the market is also competitive. Get help from HR, hire through personal leads, and hire through friends.
2. Promote the right people. With the vision and products defined earlier, promote the right people with the right behavior. The old behavior that does not make the business successful should be discussed during 1:1s with serious outcomes. Promote and reward the right people with the right behavior.
3. Find positions for people with inappropriate behavior. Talk with peers and leaders to arrange open positions for them. For example, one Ops Engineer we worked with moved to another Ops team and was promoted to a leader because his behavior matched perfectly as an Ops engineer.
4. Formalize culture and guide through hiring, promotion, and recognition. Communicate with leaders and team members to build a transparent working environment to help everybody be on the same page.

Transforming an existing culture is challenging since there are many connections in between. Start with a commonly understood vision, create as many allies as possible, and then transform with strength.

Transform a large-scale architecture - 5 - execution and delivery

 

Execution and Delivery

This blog is part of the series "Transform a large-scale architecture guide", see:

• Before the landing
• The first 30 days, and the first half year
• Create the vision
• Get the first product started
• Execution and delivery
• Hiring, promotion, and performance evaluation

Ownership

Dividing a product into different milestones and executing them may seem like a straightforward approach, but it is a common pitfall. It is more important to have the right people with ownership to execute than the milestones themselves.

Recognizing the right people with ownership starts with trust in project/feature delivery. If the owner has a track record of on-time delivery and quality attributes, that person can be used as a role model to create an ownership culture. This may be the single most important thing for execution.

The ownership team culture is not easy to build, and project delivery should start from zero trust first. It should improve by tracking details end-to-end, building trust step-by-step, and then building momentum.

Communication

Conceptual and high-level designs set the direction for product delivery, but they need to be divided into different components. Each component will require collaboration among the APIs, UIs, and customer feedback. Things will evolve, and the product and people will need to change. What was designed at the beginning may not be exactly the same as the initial design.

The gap comes from communication. One-way communication is bad. Two-way communication and group communication are important. As an architect, fostering communication is vital to success.

Communication has style and content challenges. Documenting as many details as possible helps to bridge the gap. Take the time to write them down, as clearly and visually as possible. That document can be passed along from the beginning to the end, and it will help bring more context along the way. Sometimes things change, and documents become outdated, but the high-level design should still be the same. Key principles should last. Try to document key changes as appendices to the original design.

Communication is not only about high-level/detailed level design. It is also about code communication, which is the code-review process. Equal communication and bringing people along in the process help people learn and create programmatic programmers.

Fostering communication in a professional way and bringing people along the way creates an equal way of communicating. All the team needs is to create a fantastic product to help customers succeed. So, put personal ego aside and let's collaborate!

Meetings

As in Lean, Plan-Do-Check-Act is a four-step process for execution. For product development, once the initial design is settled, divide it into different iterations for execution.

Plan: Sprint planning is about setting the right expectations on task quality. Design, coding, and testing should meet the expectation. The planning should consider vacation time, holidays, release time, dependencies, etc. The planning should be from bottom-up and top-down. Bottom-up as for time estimation, top-down as for priority and definition.

Check: Review, testing, and feedback from customers can serve as checkpoints. During the development process, different checkpoints can be made, such as design review, code review, testing, and feedback. Finding feedback earlier benefits the product and the customers more.

Act: Retrospective or private communication can serve as action items. Retrospective is for team communication to have an open forum for discussing and improving as a whole. Private communication is for personal advice such as feedback, time management, quality issues, and performance evaluation.

Keeping Focus

Focus is about saying no. Different types of distractions during product development include production issues, bug tracking, escalations, and chaotic communications.

Production issues unfortunately affect the team's performance since customers come first. But capping that into a single person or half person helps. Find an engineer who is good at dealing with those and can also deliver. During sprint planning, consider that. During performance review, also account for that.

Bug tracking comes with a priority. High-risk priority should still need to be considered. During sprint planning, create another lane for that. Account for delivery and timeline schedule.

Escalations come from production issues and bug tracking. If they are not managed properly, they surely come with an escalation. Customer service is the first to calm down the customers. Communicate with direct timeline and schedule. Customers typically can resonate.

Chaotic communications should be delegated to one proper person. That person handles all the private communications and then optimizes by knowledge base, bots, etc.

Transform a large-scale architecture - 4 - get the first product started

 

Get the first product started

This blog is part of the series "Transform a large-scale architecture guide", see:

• Before the landing
• The first 30 days, and the first half year
• Create the vision
• Get the first product started
• Execution and delivery
• Hiring, promotion, and performance evaluation

Finding Priorities

When creating a vision, you need to determine the north star and overall issues in each domain area. However, with limited resources, it can be challenging to tackle all the problems. So, how do you prioritize which issues to address first and which have better return on investment (ROI)?

Conversations with customers can help prioritize concerns, but these conversations may be opinionated. The individuals you speak with are responsible for a business and focus on what's most important to them. While they can provide data and facts, it may be specific to their priorities and not the entire business.

Gathering data and facts from the bottom up can lead to an overwhelming amount of information. However, grouping them together can help identify the most important things for the systems to operate. These may not be the most urgent issues to address, but they are still essential.

Examining how customers and engineers spend their time is another way to gather information on how to improve their happiness. Understanding why they're unhappy and what they're spending time on can provide insights into priorities.

To find the right balance between the most important and most urgent issues, list out all the concerns, examine them with data and conversations, and then use your experience and judgment to prioritize them. List out the most important things and verify with leaders by having a group conversation with facts and data.

Once you've established priorities, allocate your resources in a laser-focused manner. Losing this focus will affect your timeline and patience from funding partners/leadership.

Defining Product Scope

The vision and priorities guide the project's direction, but the leaders have expectations for results within a specific timeframe. While you can define a perfect product that takes years to deliver, business leaders may lose patience and need to see results sooner.

Ultimately, the product should gain a certain amount of market share within a specific time frame. In my experience, for a 2B product, it can be divided into several milestones:

• Proof of Concept/Buy-in from flagship customer (3 months)
• Delivery of the first product and migration from flagship customer (6-9 months)
• Expansion of the product into the market to gain a certain amount of market share (9-12 months)

The Proof of Concept (PoC) can be a prototype or a simple website that showcases the experience. The PoC should define the flow of the experience and how it simplifies customers' experiences.

The PoC should define the scope of each experience and list out the things you will do in the beginning and the things you will NOT do in the first phase. Not defining it well can create high expectations that cannot be delivered later on in the expected timeframe.

The product delivery and migration should correlate well with the PoC. The requirements should be listed out with a risk factor, including the migration. The product delivery will also need to set up the timeline and expectations right, so there is a two-week or monthly reporting. This is the time to create trust and gain support.

Expansion of the product is an iterative process that requires a pipeline of customers, conversations, plans with the customers, and should be added as part of the product roadmap, iteratively.

High-Level Design

High-level design should focus on the domain-driven architecture. Each domain should either have its own logical or physical boundary. There will always be arguments on micro-services or monolithic services. In my experience, each domain or several similar related domains should be a micro-service.

Defining a micro-service will have its boundaries, including security, database, API, CAP, etc. Define each micro-service and what it should and should not be doing.

Defining the key algorithms/user flows that flow through the micro-services ensures major cases will be working well. Along with it, define the major testing cases to verify they are working excellently.

Defining the operating model for the major services, including failure case fallbacks and pre-defined standard-operating-procedures (SOPs) or automations.

Review them with the team all together, brainstorming on each one of them, and making sure everyone understands their responsibilities, scope-of-work, and delivery expectations.

Finally, make a delivery milestone for each one, work as a team, and get started.

Making Trade-Offs

An Amazon quote says "Is this a one-way door decision?" but it depends on the context. For a product owner, the ultimate question for making trade-offs is whether it is a key experience for customers at this phase and whether it correlates with the key concepts.

The key concepts and architecture define the boundary of the ultimate vision and product. The key concepts should be primitive and present from day 1. Adding/changing them will be expensive.

Then there is a domain area for each product to deliver, and there are domain concepts for each. The domain concepts are key experiences that cannot make trade-offs. Then comes the use cases and experiences, which can make trade-offs.

Customers always have ways to get things done, but the key concepts that will affect their core experiences, including security, cost, and simplicity, cannot be sacrificed.

The best way is always that things can be iterated and incremental. Even with small web-page customization, as long as the concepts and flow are there, as more and more customers onboarded, the small customizations will eventually change and it is a low cost.

Transform a large-scale architecture - 3 - create the vision

Transform a large-scale architecture: Create the architecture vision

This blog post is part of the series "Transform a Large-Scale Architecture Guide," which includes the following parts:

• Before the Landing
• The First 30 Days, and the First Half Year
• Create the Vision
• Get the First Project Started
• Execution and Delivery
• Hiring, Promotion, and Performance Evaluation

The Vision

A vision is a mission statement that should be precise, easy to understand, and memorable. Defining a vision is a collective thinking exercise that requires a lot of understanding, time, and effort. There is no one way to define a vision, but the following are necessary:

Be Open-Minded and Help People Solve Their Pain Points

Be open-minded and always think about win-win solutions. Software is created by humans, and any changes to the current software product will involve people. Communication with people requires mindfulness about the conversations. The goal is to have honest conversations, but conversations can easily go nowhere.

Be a multiplier by asking questions to enable things. The current business may already have ten systems running for over ten years and believe changing them would be too disruptive. Redefining them may not bring necessary business value. Instead, think about why you are doing the same thing. Put that thinking aside and ask questions about the challenges of the existing systems and how you can help.

Understand the Current

Research the current systems to understand their relationships, functions, technologies, and business values. Use an Excel sheet to list all the functions and business use cases. Abstract them into different domains and establish the relationships among them.

Understanding the current also means having conversations with people familiar with the systems. Learn about how they are using them, what the patterns are, the current challenges, and any ideas for change. The conversations about the current will be fragmental but are representative of daily life.

Think from the current business perspective and connect them through business metrics data. If there isn't any, try to find something similar or representative.

Think Outside the Box

The current systems are still running, but maybe not very well. There are backgrounds on the existing business, and it's not just the technical reason why it is still here today. Think outside the technical box to understand the organizational, business, and industry growing history part of it.

The industry is evolving every day, and there are well-established technologies and newly innovated solutions or tools. For the domains identified from the current, think about industry trends, innovations, or new ideas. As an experienced engineer, pick the right trends for the current business to invest in, work with, or create new ones.

Keeping track of the latest trends in research areas and the industry can also help to keep the mind fresh and know where to find more detailed information when necessary. Joining open research channels, podcasts, conferences, etc. regularly can be helpful.

Value Proposition

Once you understand the current and think outside the box, there are apparently things the business can do better, either through using new technology or just creating a better product to achieve better results. But what are the better results, and how can you justify them through data?

Collecting data through the current is not easy since, if they have already understood the current metrics so easily, why are they not starting to change? The metrics data may be buried in various issues, notes, emails, or manual works, but not in a systematic way. Finding a representative data that is similar can also work. Maybe through a customer survey, data mining on current notes, issues.

Collecting data from the industry means reading reports and surveys on the industry-leading companies. Market researching companies like Gartner provide insight into who the leaders are in each domain and why they are the leaders. The leaders are usually innovators who educate the market on what are the things and provide insights on how can they use their product to achieve better results.

Use the industry and current data to propose the values, collect feedback, and reach agreement to fund the projects.

Architecture Framework

Architecture is an abstraction of each domain and the relationship among them. There are layers from bottom to top and connectors in between. Defining an architecture can be easy or hard depending on the level:

• Don't talk about too many details, create new concepts;
• The architecture boundary should be stable, not changed by details;
• The architecture should be flexible so that each domain can be prioritized differently.

Create New Concepts

There are always many details. Try to scope them into different domains and name them with different concepts. Verify the concepts through different conversations, document them, and communicate over-and-over to see people's reactions.

The concept name should not be completely new that nobody understands at first hearing. Try to find the concepts through people's talkings so that they can match their expectations more easily. Keep explaining new concepts over-and-over means education costs at latter phases.

The concepts should not leave out the technical details, including protocols, access and identity, databases, security considerations. Don't talk about them. Instead, focus on the concepts and what the use cases are.

Stable Architecture Boundary

Architecture is for guidance on technology. It is a high-level abstraction of the system. It should cover all the cases researched before and will cover the cases according to the industry research. Each domain architecture is an abstraction of the current business and technology and should not move in the many years to come.

The architecture is also forward-looking. Any new changes coming should be easily extendable to the current architecture, which means that the core concepts are unified, stable, and also extendable. Change to core concepts is a disaster to the whole system, so the concept and life-cycle around it need to be well-communicated and stabilized.

Each business domain is composed of the business cases around it. The core concepts need to be loosely-coupled. Each interface for each business domain should be stable and backward-compatible. It should have its business perspective, and the design should focus on its perspective. Even if there is a problem around each domain, it should be easily replaceable without affecting other domain services.

The book "Clean Architecture" has a diagram to illustrate the relationship.

Flexible Architecture

The business priority can and will change. The things planned at the beginning can and will change. Having a flexible architecture that adapts to change will affect execution efficiency.

The architecture relationship among each domain should be loosely-coupled and abstracted through interfaces. There are patterns like Strategy Pattern and Abstract Factory that can help create replaceable dependencies. When a new service is not ready or partly ready, having an abstraction will keep the architecture evolving.

Each domain should be flexible, and the lifecycle of each domain should also be replaceable. So when some part of the lifecycle is not actually there, create a simple one with little effort to help the business grow.

At the end of the day, the vision is about thought determination, that the business is doomed to change in an iterative manner. The above things are things to consider, but the key is about determination and bringing people along. Even if things are not there at the beginning, things can still happen.

Transformation a large-scale architecture 2: The first 30 days, and half year.

Transform a large-scale architecture: The first 30 days, and half year.

This blog is part of the series "Transform a Large-Scale Architecture Guide." Please see the links below for the other parts of the series:

• Before the Landing
• Create the Vision, the First 30 Days, and the First Half Year
• Get the First Project Started
• Execution and Delivery
• Promotion and Performance Evaluation

After careful consideration, you have decided to join the team. The first 30 days are key to making a good impression on your new colleagues. People will have expectations and judgments, so take the time to get to know them, create a positive impression, and present yourself as they expect. Keep a low profile and don't rush to make judgments.

First 30 Days

As an architect, people naturally expect you to bring new ideas and directions. Talk to your peers and managers to see what they expect. Your job is to transform, so they expect you to challenge them, but do so in a cooperative manner.

Don't judge. Bringing freshness to the team is good, but not everyone wants to be challenged. Be respectful, kind, and empathetic. Ask questions and help them understand that you are here to help them achieve success.

Try to start with something small. It takes time to create a thinking framework for a grand picture, and lots of research is required. This is the most important job of the transformation, but it takes lots of time. Meanwhile, find a peer who needs something from you and partner with them.

Be mindful of what you say. Don't talk about things that people are worried about. Transformation in architecture also means changes to the business and teams associated with it. If there are team changes, it will be an organizational change initiated by the leadership and driven by the transformation. The architect's role is to enable the organization to do that. Let people see it. It's too early for the first year to even talk about it. The organization will arrange things more appropriately and gently.

Try to make as many friends as possible. Attend the team's typical activities and be a part of them. Identify people who may have a similar background.

First Half Year

The most important thing is to create a workable vision and roadmap. But it takes time, so don't rush. Talk on a weekly basis and set up a routine to talk. Note things down with any notebook software. Talk with ideas, learn the ideas through research, keep track of industry trends, and check the reactions from peers, partners, customers, and other leaders. Identify what is doable and what is not.

While creating a grand vision and workable architecture, the first 30 days have already presented some low-hanging fruit to work on. Get the existing team members to do them. Understand what the team is good at, what the culture is, what the working standards are, what the quality is, what the process is, and so on.

Hiring

In my experience, transformation always comes along with hiring. During the first half year, you have a chance to change the culture by hiring key people.

Hiring is one of the most important things in transformation. People are hard to change. As an HRBP said before, 80% of things are settled once people are hired, and only up to 20% can change. The culture can be transformed by hiring, but you will need to know what kind of people you need to hire. The market is big, and you can always hire many people, but try to hire more experienced people rather than fresh graduates. The fresh graduates will be transformed by the team instead of transforming the existing.

Getting Things Done

Getting small things done creates credit among your peers and is one of the key supports you will need during the journey. Leverage the current team members to help deliver. Work with them, see through the work, see what they are qualified for, and see what they are not.

Coaching people is one of the ways to change. There will always be people who want to learn, especially passionate engineers. One of the best things about transformation is that there are tons of coaching opportunities. Coach through challenges, through various work, give space to everybody, let them do it, and then coach on cases.

Create the Vision

There are always plenty of things to do, but besides hiring and getting things done, spare a big part of the time focusing on the big roadmap.

First, you will need ideas, lots of ideas, lots of workable ideas. The ideas come from learnings on the industry, outside research, notes, talking with various leaders, and research on existing systems. Then, formulate the ideas into a workable solution that needs connections. The more connections, the more complex the whole solution becomes, and there will be a lot of moving pieces.

Second, after getting the ideas and connections, you need to know where to start and how to start. Creating a stable boundary will be critical. You will need to create domains and set up connections with each domain. There are books like Clean Architecture that talk about it and the elegance around it. Creating an architecture can also mean working with legacy systems, so Working Effectively with Legacy Code and Monolith to Microservices teach you how to work on that.

Finally, in my experience, it is often harder than technology itself. The books often tell you how to work from a technical perspective, but not from a product and organizational perspective, which the latter articles will cover.