How Apex Financial Migrated 400 Legacy APIs to Event-Driven Architecture Without Impacting Uptime

Overview

Apex Financial is a mid-size digital payments processor handling more than twelve million financial transactions every day across mobile, web, and POS channels. Founded in 2007, the company built its core settlement engine as a tightly coupled monolith composed of four hundred synchronous REST APIs. By 2023, the system was struggling to keep pace with business growth, incident response complexity, and changing regulatory requirements. Apex engaged Webskyne to lead a full-stack architecture modernisation programme focused on delivering higher availability, scalable throughput, and shorter release cycles while preserving functional correctness and zero customer-facing downtime.

The migration programme lasted eight months and touched every layer of the technology stack: API gateways, service boundaries, data models, deployment pipelines, and operational tooling. Webskyne deployed a senior architect, two platform engineers, and a developer-experience specialist to embed with the Apex engineering team and ensure knowledge transfer throughout the engagement.

Apex operates in a highly regulated environment. Every transaction must generate an immutable audit trail that is available for scrutiny by the Monetary Authority and by internal compliance officers. Any change to the settlement path required careful alignment between legal, risk, and engineering leadership. Webskyne structured weekly architecture review forums that included representatives from all three groups, ensuring that technical decisions were validated against regulatory and business constraints in real time.

Challenge

The existing system exhibited several critical failure modes. Dependency chains between services meant that a slow downstream response could cascade into wide-area latency spikes. Database contention on a single write-optimised cluster capped throughput at approximately two hundred transactions per second in peak windows, forcing queue buffers and manual throttles. Observability was fragmented: log aggregation existed, but tracing was absent, so incident investigators routinely spent hours correlating request IDs across services before pinpointing root cause. The test-suite coverage for integration contracts between APIs hovered near thirty-two percent, making large-scale refactoring high risk. Finally, release coordination required a full-system regression test lasting eight hours because useful integration tests were rare and most services were still deployed onto bare metal without containerisation.

Beyond these technical concerns, business stakeholders demanded that the migration be unnoticeable to customers. Regulatory teams insisted on precise audit semantics for every transaction, including replay guarantees and event-level immutability, which were difficult to express in the current request-response model. In total, the programme faced four major constraints: maintain ninety-nine point nine percent availability, reduce end-to-end latency from a median of eight hundred thirty milliseconds to under two hundred milliseconds, improve developer deployment frequency from monthly to daily, and implement immutable audit logs compatible with both Monetary Authority and PCI-DSS requirements.

Technical Debt Inventory

Before proposing a solution, Webskyne conducted a two-week technical-debt inventory to quantify exactly where risk lived. The team profiled every external API contract, catalogued database dependencies, and mapped the call graph of the settlement engine. The findings were sobering: forty percent of endpoints had no integration tests, twenty-three services shared a single schema without versioning, and the mean time to detect a degraded dependency exceeded forty minutes.

Scalability Roadblock

Peak transaction periods, including government disbursement days and festive shopping weekends, routinely pushed the system beyond capacity. On-call engineers implemented manual throttles and rate caps that slowed legitimate traffic and degraded user experience. The root cause was the shared database write path: every debit, credit, and reconciliation hit the same tablespace, causing lock contention and I/O saturation that cascaded across unrelated services. Scaling vertically was feasible only by provisioning ever-larger instances, a path that was both expensive and operationally complex.

Observability Gaps

Incident response was slow because of missing context. Engineers could see that API gateway response times had spiked, but they could not tell whether the bottleneck was the database, a third-party payment rail, or an internal retry storm. Logs were aggregated in a central repository but lacked correlation identifiers that tied requests across service boundaries. Without distributed tracing, each microsecond of mystery translated into minutes of investigation.

Goals

• Decompose the monolithic settlement engine into bounded contexts communicating through asynchronous events.
• Introduce an event log with replay, offset tracking, and schema registry to satisfy audit and compliance requirements.
• Build a strangler-fig migration layer that gradually shifted traffic without cutover windows or downtime.
• Reduce median p95 latency from 830 ms to below 200 ms.
• Achieve 99.95% availability target and 24×7 incident isolation per domain.
• Increase deployment frequency from monthly to daily per service.
• Enable product teams to ship new payment methods and payout workflows with minimal release coordination.

Approach

Webskyne proposed a three-phase migration anchored on domain-driven design and event-sourcing patterns. Phase zero focused on observability and safety. The team deployed distributed tracing using OpenTelemetry, instrumented all existing APIs, and established SLO dashboards with clear error-budget policies. Before any production code path changed, the engineering team could measure system health and react to issues before customers noticed them.

Phase one introduced the event backbone. Apache Kafka was selected as the primary transport and durable log because of its proven throughput at scale, exactly-once semantics, and strong ecosystem support for schema management and replay. The schema registry enforced Avro schemas so that producers and consumers could evolve independently without breaking downstream contracts. During this phase, the team also modernised the data layer by replacing the shared relational cluster with bounded-context stores: one event journal, two read models (PostgreSQL for transactional queries and ClickHouse for analytics), and Redis for session and rate-limit state.

Phase two executed the strangler-fig migration. An API gateway sat in front of the monolith and progressively routed individual endpoints to new microservices after each endpoint passed contract and chaos tests. The runtime traffic mirroring capability copied live transactions into the new event log so that the replacement services could be warmed by real customer data long before they accepted production traffic. Deployment automation using GitOps and progressive rollouts minimised blast radius and allowed rollback in under ninety seconds.

Domain-Driven Decomposition

A distinguishing feature of the engagement was the depth of domain analysis performed prior to writing any migration code. Over three weeks, engineers, product owners, and compliance officers participated in event storming sessions that surfaced fifty-six aggregate roots. The team identified four natural bounded contexts: authorisation, which validates credentials, checks risk rules, and approves or declines transactions; clearing, which aggregates approved transactions into settlement batches; settlement, which applies liquidity rules and generates instructions for external banks; and reporting, which builds regulatory statements and merchant reconciliation files.

Each bounded context maintained its own data model and published events that other contexts consumed. This arrangement eliminated the database-sharing anti-pattern that had previously made independent deployment impossible. Services within a context still communicated via REST where synchronous interaction was appropriate, but cross-context communication preferred asynchronous events to preserve loose coupling.

Event Sourcing and Schema Governance

The decision to treat the event log as the primary source of truth was driven by compliance requirements and operational flexibility. Because regulators demanded immutable records, a write-optimised event store was easier to defend than a set of mutable relational tables. Schema governance used Confluent Schema Registry with backward and forward compatibility checks enforced at build time. Any breaking schema change required explicit consumer migration plans reviewed by architects and product owners, preventing the accidental contract drift that had caused past incidents.

Event replay capability was critical for recovery scenarios. Operations teams could instruct the platform to rebuild any read model from a known offset, often in under twenty minutes. This made disaster recovery drills faster and gave auditors confidence that historical data could be reconstructed on demand.

Strangler-Fig Migration Layer

Rather than attempt a big-bang replacement, the engineering team implemented a strangler-fig pattern using an API gateway at the edge. The gateway maintained a routing table that assigned each endpoint to either the legacy monolith or a new microservice. For each migrated endpoint, the team used canary releases that directed a small percentage of production traffic to the replacement. If error rates or latency exceeded defined thresholds, the gateway would instantly revert traffic to the stable path. This mechanism allowed the team to measure behaviour under real load with minimal risk.

Traffic mirroring further increased confidence. Live requests were duplicated into the new event log so that the microservice could process production-grade volumes long before it received real customer traffic. Business stakeholders reviewed mirrored performance dashboards and gave explicit approval before any endpoint was promoted to full production traffic.

Resilience and Fault Isolation

Resilience patterns were applied consistently across all new services. Circuit breakers prevented cascading failures by tripping after a configurable ratio of errors occurred within a sliding window. Bulkheads isolated payment processing workloads from administrative and analytics workloads so that a batch reporting job could not interfere with live transactions. Retry budgets used exponential backoff with jitter to smooth retry storms and reduce thundering-herd effects.

Service boundaries coincided with organisational boundaries, reducing cognitive load and coordination overhead. Each team owned the full lifecycle of its services, from infrastructure provisioning to on-call rotation. This alignment clarified ownership and accountability, which in turn improved mean time to recovery.

Observability and SLO Management

The observability stack was designed around the RED method for services (Rate, Errors, Duration) and the USE method for resources (Utilisation, Saturation, Errors). Alerts were based on SLO burn rate rather than absolute thresholds, meaning engineers were paged only when the probability of breaching the availability target within a given window exceeded a defined threshold. This change reduced nocturnal alert fatigue by more than sixty percent, according to post-mortem surveys.

Context propagation standards required that every external call carry a trace identifier, a span identifier, and a baggage field containing a request-level correlation ID. These identifiers allowed the observability platform to reconstruct end-to-end traces across asynchronous message boundaries, dramatically reducing the time required to diagnose incidents.

Results

After eight months of phased execution, the new event-driven platform achieved the original performance and reliability targets. Median latency dropped to one hundred seventy-eight milliseconds, representing a 78% improvement over the baseline. Peak throughput more than tripled to approximately seven hundred transactions per second with headroom for seasonal peaks. Availability improved to 99.97%, driven by domain isolation and faster failure detection through distributed tracing.

Developer productivity improved dramatically. Deployment frequency increased from monthly to several times per day per service. Feature-flag driven releases allowed product teams to ship safely during business hours. Mean time to recovery fell from three hours to twenty-two minutes, largely because scoped observability and reversible traffic routing removed the need for high-stakes cutovers.

Operational costs decreased by an estimated forty percent over two years due to the elimination of manual coordination work, over-provisioned database capacity, and incident response overhead. The audit-ready event log became a new source of truth for compliance, eliminating the need for separate reconciliation workflows that had previously consumed six engineer-weeks each quarter.

Business Impact

The migration unlocked new revenue opportunities that had been blocked by the previous architecture. Payment gateway providers, which required sub-two-hundred-millisecond response times, became viable partners for the first time. The settlement engine could now process cross-border transactions in real time rather than overnight batches, reducing foreign-exchange losses and improving customer cash flow. Product teams reported a shorter time-to-market for new features, accelerating the launch of installment plans and merchant cash-advance products.

Team Transformation

Equally significant was the cultural shift within the engineering organisation. Engineers moved from a reactive, hero-culture mindset to a proactive, platform-engineering mindset. They embraced GitOps practices, automated testing, and blameless post-mortems. The platform team embedded in the engagement documented all runbooks and architecture decision records so that future engineers could understand the rationale behind every major design choice, preserving organisational memory beyond any single individual.

Key Metrics

• Median transaction latency: 830 ms → 178 ms (-78.5%)
• Peak throughput: ~200 TPS → ~720 TPS (+260%)
• System availability: 99.90% → 99.97%
• Deployment frequency: ~1/month → 5×/day per service
• Mean time to recovery: 180 min → 22 min
• Test coverage for integration contracts: 32% → 87%
• Operational cost reduction: ~40% over 24 months
• Reconciliation manual effort: 6 engineer-weeks/quarter → 0

Timeline

The migration followed a predictable cadence. During months one and two, the team established baselines, deployed observability tooling, and conducted domain analysis. Months three and four were dedicated to building the event backbone, schema registry, and read models. Months five through seven implemented the strangler-fig gateway, performed canary migrations of the highest-traffic endpoints, and validated resilience under load. Month eight focused on final migration of remaining endpoints, hypercare monitoring, and knowledge-transfer workshops with the Apex engineering team.

Lessons Learned

Several lessons shaped subsequent engagements at Webskyne. First, investing in observability before touching production architecture paid for itself within the first two months by reducing investigation time and preventing regression incidents. Second, schema governance was not a bureaucratic afterthought but a critical safety mechanism; the compatibility review process became a standard practice across all event-sourcing projects. Third, strangler-fig migration reduced anxiety across stakeholders because business continuity was visible and measurable from day one. Fourth, bounded-context alignment required ongoing facilitation; it was not enough to draw boxes on an architecture diagram; teams needed shared language practices and explicit context maps to prevent drift.

Finally, the importance of gradual traffic migration cannot be overstated. Live traffic mirroring and canary routing gave product owners and security teams the confidence to approve changes that would once have required months of negotiation. The result was a transformative improvement in capability without a single customer-visible outage during the entire migration window.