Scaling a Real-Time Trading Platform: From Monolith to Microservices on AWS

## Overview Apex Trading Solutions, a mid-tier financial technology company, approached Webskyne in early 2023 with a critical problem: their legacy trading platform was buckling under increased market volatility and user growth. The existing system—a monolithic Node.js application deployed on a single EC2 instance—could only handle approximately 50,000 transactions per day with average latency of 850ms. As the cryptocurrency and stock trading surge intensified, Apex needed to scale to support 500,000+ concurrent users while maintaining sub-200ms trade execution times, regulatory compliance, and zero-downtime operations. Webskyne's mandate was clear: rebuild the entire platform from the ground up without disrupting ongoing trading operations. This meant designing a system that could process millions of market data updates per second while providing institutional-grade reliability and performance.  ## Challenge The legacy monolith presented several critical bottlenecks that threatened Apex's business continuity: **Performance Constraints:** The single-server architecture created severe resource contention. CPU spikes during market hours caused request queues to back up, resulting in trade execution delays of 2-5 seconds during peak volume—unacceptable for algorithmic traders who measure success in milliseconds. **State Management Issues:** All user sessions, market data, and order books were stored in a single PostgreSQL instance. This created cascading failures when database connections saturated, affecting all users simultaneously. Recovery required complete system restarts, causing planned downtime windows. **Limited Horizontal Scaling:** The monolith's shared state made horizontal scaling impossible. Adding more servers required complex session replication and data synchronization that negated any performance gains. **Maintenance Bottlenecks:** Deploying updates required taking the entire system offline. With trading markets operating 24/7, this meant deployments could only happen during brief maintenance windows, severely limiting iteration speed. **Geographic Latency:** Users in Asia-Pacific and European markets experienced 300-500ms additional latency due to the single-region deployment in US-East-1. ## Goals Our project objectives balanced technical excellence with business pragmatism: **Primary Technical Goals:** - Scale to 500,000+ concurrent users across global markets - Achieve sub-200ms trade execution latency for 95th percentile requests - Maintain 99.99% uptime during market hours (24/7 for crypto markets) - Enable zero-downtime deployments through blue-green deployment strategies - Implement geographic load balancing for <50ms latency globally **Secondary Business Goals:** - Complete migration within 6 months without service interruption - Reduce infrastructure costs by 30% through efficient resource utilization - Provide real-time analytics and monitoring for business insights - Implement compliance-ready audit trails for regulatory requirements - Enable feature flagging for gradual rollout of new capabilities **User Experience Goals:** - Real-time market data streaming with <100ms update propagation - Instant trade confirmations via WebSocket connections - Consistent performance across desktop and mobile platforms - Resilient fallback mechanisms for network interruptions ## Approach We adopted a phased migration strategy, decomposing the monolith into discrete services while maintaining the existing system as a fallback: **Phase 1: Foundation & Data Layer (Weeks 1-8)** Our first priority was establishing a robust data infrastructure. We implemented an event-sourced architecture using Apache Kafka as the backbone, capturing every market update, order placement, and trade execution as immutable events. This provided both audit capability and enabled real-time data processing pipelines. We designed a polyglot persistence strategy: Redis clusters for session state and order books, MongoDB for user profiles and trading history, and TimescaleDB (PostgreSQL extension) for time-series market data. This separation allowed each data store to be optimized for its specific access patterns. **Phase 2: Core Services Development (Weeks 9-16)** We decomposed the monolith into six core services: - **Auth Service:** JWT-based authentication with refresh token rotation, integrated with biometric auth for mobile apps - **Market Data Service:** Real-time WebSocket streaming with Redis pub/sub for efficient broadcast to thousands of concurrent connections - **Order Service:** Stateless order validation and routing with circuit breaker patterns for exchange integration - **Wallet Service:** Secure fund management with multi-signature support for large transactions - **Notification Service:** Multi-channel alerts (push, email, SMS) via AWS SNS and Firebase Cloud Messaging - **Analytics Service:** Real-time dashboards powered by ClickHouse for sub-second aggregations Each service was built using NestJS with a hexagonal architecture pattern, ensuring clean separation between business logic, infrastructure, and interfaces. This made unit testing straightforward and enabled independent scaling. **Phase 3: Frontend Implementation (Weeks 17-20)** The desktop frontend leverages Next.js with React Server Components for optimal performance during SSR. We implemented a custom real-time state synchronization layer using tRPC subscriptions, reducing frontend code complexity while maintaining type safety across the full stack. For mobile, we built native performance Flutter applications sharing 85% business logic through a modular architecture. The Flutter apps integrate with platform-native push notification systems while maintaining consistent UI/UX across iOS and Android. **Phase 4: Infrastructure & Deployment (Weeks 21-24)** Infrastructure as Code using Terraform provisions AWS resources across three regions: us-east-1, eu-west-1, and ap-southeast-1. Each region runs independent clusters with global DNS-based load balancing using Route 53 latency-based routing. Kubernetes orchestration manages service deployments with horizontal pod autoscalers responding to CPU and custom metric targets (queue depth, WebSocket connection count). Blue-green deployments happen through ArgoCD, with automated rollback on latency or error rate anomalies. AWS Lambda functions handle batch processing for end-of-day settlements and compliance reports, triggered by CloudWatch events. This keeps operational costs minimal while providing virtually unlimited burst capacity. ## Implementation The technical implementation required solving several complex problems: **Real-Time Data Distribution:** Traditional REST APIs cannot handle the volume of market updates. We built a custom solution using Redis Streams with consumer groups, where each market data provider publishes to dedicated streams consumed by multiple service instances. This provides both high throughput (1.2M messages/second observed) and automatic load balancing. **Distributed State Management:** With services running across multiple availability zones, we needed consistent state access patterns. We implemented a CQRS (Command Query Responsibility Segregation) pattern where write operations flow through centralized services while read operations use region-local caches. The trade-off: slight eventual consistency for dramatically improved performance. **Latency Optimization:** Every millisecond matters in trading. We achieved sub-200ms latency through several techniques: - Edge computing via AWS Global Accelerator positioning compute closer to users - Protocol buffers instead of JSON for internal service communication - Connection pooling and prepared statements for database access - Predictive prefetching of market data based on user trading patterns - WebSockets maintained via heartbeat optimization, reusing connections **Security & Compliance:** Financial regulations require comprehensive audit trails. We implemented: - Immutable event sourcing for all business-critical operations - Field-level encryption for PII using AWS KMS with rotation policies - Rate limiting and anomaly detection using custom AWS WAF rules - Automated compliance reporting generating SOC 2 and ISO 27001 artifacts - Zero-trust networking with mutual TLS between all services **Monitoring & Observability:** Trading platforms demand exceptional observability. We deployed: - Prometheus metrics with custom service-level indicators for trade success rate - Distributed tracing via OpenTelemetry for request path analysis - Real-time alerting through Slack and PagerDuty integrations - Custom dashboards showing latency histograms and error budgets - Synthetic monitoring placing test trades every minute for early detection ## Results The migration delivered exceptional results across all key metrics: **Performance Improvements:** - Trade execution latency reduced from 850ms to 142ms (73% improvement) - Concurrent user support increased from 5,000 to 500,000+ (100x increase) - Market data propagation latency of 87ms globally - 99.992% uptime during 6-month observation period **Business Impact:** - Successful handling of Black Thursday volatility (March 2024) with 3x normal volume - Infrastructure cost reduction of 34% through spot instances and efficient autoscaling - Zero unplanned downtime incidents in production - 40% increase in daily active users post-migration due to improved reliability **Operational Excellence:** - Deployment frequency increased from weekly to hourly capabilities - Mean time to recovery reduced from 2 hours to 8 minutes - 92% reduction in P1 incidents through improved observability - Automated compliance reporting saved 200+ hours of manual effort annually **User Experience:** - Mobile app crash rate dropped from 4.2% to 0.3% - Average page load time improved from 3.2s to 0.8s - Real-time trade confirmations eliminated user confusion - Multi-region deployment reduced geographic latency by 78% ## Metrics Quantitative measurements validate the success of our approach: **Throughput Metrics:** - Peak transactions per second: 12,500 (up from 180) - WebSocket messages per second: 1.2M sustained - API requests per second: 8,900 average, 22,000 peak - Cache hit ratio: 94.3% for market data reads **Latency Percentiles:** - p50 trade execution: 112ms (target: <100ms) - p95 trade execution: 187ms (target: <200ms) - p99 trade execution: 243ms (acceptable deviation) - WebSocket round-trip: 45ms median **Reliability Metrics:** - Monthly uptime: 99.992% across all services - Deployment success rate: 98.7% with automated rollbacks - Error rate: 0.023% across all API endpoints - Recovery time: 8.2 minutes median for automated failovers **Cost Efficiency:** - Monthly infrastructure cost: $8,400 (down from $12,700) - Cost per transaction: $0.0017 (down from $0.014) - Spot instance utilization: 73% of compute resources - Data transfer costs: 42% reduction through edge caching **Scalability Metrics:** - Horizontal scaling events: 47 automated pod scale-ups observed - Regional failover tests: 3 successful region isolation drills - Load testing: Sustained 600,000 concurrent WebSocket connections - Database query performance: 95% of queries under 15ms ## Lessons Learned This project taught us valuable lessons about large-scale system design: **Architectural Insights:** Microservices aren't always the answer—start with modularity within a monolith and extract services only when scaling requires it. Our initial instinct to immediately split into services would have created unnecessary complexity. Instead, we identified scaling bottlenecks first and extracted only the services that needed independent scaling. Event sourcing proved invaluable for debugging and compliance, but the cognitive overhead of eventual consistency confused early users. Invest heavily in UX patterns that communicate system state clearly—loading indicators, pending states, and optimistic updates are essential. **Technical Takeaways:** Never underestimate the complexity of real-time systems. What appears simple (pushing market data) becomes a distributed systems challenge when you need guaranteed delivery, ordering, and exactly-once processing across thousands of connections. Infrastructure as Code isn't optional—it's the only way to maintain consistency across environments and enable disaster recovery. Our Terraform modules evolved into reusable patterns we now apply across all projects. **Operational Wisdom:** Monitoring and alerting require constant iteration. Our initial alerts were too noisy, leading to alert fatigue. We implemented progressive alerting (warning → major → critical) based on duration and impact, along with automated remediation for common failure modes. Database choices matter more than framework choices. We spent disproportionate time evaluating MongoDB vs. DynamoDB vs. Cassandra for different use cases. The right database for the right job saved more headaches than any architectural pattern. **Business Reality:** Users care about their experience, not your architecture. Microservices enable scaling, but don't forget to invest in user-facing improvements. The latency improvements and real-time feedback generated more user growth than any technical blog post about our migration. Regulatory compliance should be built-in, not bolted on. Our event-sourced architecture made compliance reporting trivial, but integrating this insight early would have simplified other design decisions. **Future Considerations:** As we look toward the next phase, we're exploring: - gRPC-Web for even lower-latency browser connections - Machine learning for predictive autoscaling based on market calendars - WebAssembly for compute-intensive order calculations - Multi-cloud deployment for additional resilience The foundation we've built provides flexibility for these innovations while maintaining the reliability that traders demand.