Hi, I'm Rushil

Most of my previous projects were about building features. This project was about building a system from the ground up. I wanted to move past monolithic architecture and actually make decisions regarding perfomance tradeoffs and system design. I built a platform to track NBA player stats and predict future performance, but the real challenge was how these services talked to each other.

The "Why" Behind the Tech

I went with a Polyglot Architecture because I wanted the right tool for the job, not just the tool I knew best. I used Java (Spring Boot) for the Security and Stats services because Spring boot is the de facto for microservices. Also, tools like Netflix Eureka, Spring Cloud were very helpful for service discovery, load balancing, and fault tolerance. But for the nba-fetcher and prediction services, I used Python. Trying to do data science or heavy API scraping in Java felt like fighting the language, while Python’s ecosystem made it seamless.

One of the biggest trade-offs I made was implementing gRPC for internal communication between my Security and Stats services. I could have used standard REST, but I wanted to practice low-latency communication. Using Protobufs felt like a win because it forced me to define a strict contract between services, instead of guessing what an API response looks like.

Engineering Maturity & Habits

When you have five services running at once, you can’t just keep it all in your head. Documentation as a Tool: I started using ADRs (Architectural Decision Records). For example, I documented why I chose Redis Pub/Sub as the event bus for my AI Agent. I wrote down the consequences of that decision (services are decoupled), and the tradeoffs (No message persistence).

Visualizing Complexity: I adopted the C4 Model for my diagrams. I realized that if I couldn't draw the data flow at a Container and Component level, I didn't actually understand my own system.

Testing & Resilience: I moved away from just testing happy paths. I spent a lot of time writing unit tests in JUnit and Mockito to handle what happens when a service goes down. I used Docker Compose to manage the environment, ensuring that my PostgreSQL and Redis instances were always configured correctly across dev and prod.

Takeaway

Microservices are a double-edged sword. They give you amazing flexibility, but they tax you with complexity in service discovery and data consistency. Navigating that tax made me a much better developer. I stopped just writing code and started thinking about how systems survive in the real world.

View README on GitHub →

Why FastAPI Over Flask or Django?

I chose FastAPI because I needed native async/await support for concurrent API calls to Finnhub. Flask's sync nature would have required workarounds like Celery, adding unnecessary complexity. Django felt too heavy for a focused API service. FastAPI's automatic OpenAPI documentation and Pydantic integration meant I could enforce data contracts at compile-time rather than discovering issues at runtime. This was important when handling financial data where type safety matters.

AWS SES and S3: Why Not SendGrid or Local Storage?

I evaluated SendGrid and Mailgun for email delivery, but AWS SES made more sense for three reasons: (1) significantly lower cost at scale, (2) tight integration with other AWS services I planned to use (Lambda for future expansion), and (3) better deliverability reputation for transactional emails. For logging, I chose S3 over local filesystem or CloudWatch because I wanted immutable audit trails that survive container restarts, and S3's lifecycle policies let me automatically archive old logs without manual intervention.

Layered Architecture: Preventing Service-ORM Coupling

I strictly separated Routers → Services → Repositories to avoid the common mistake of embedding SQLAlchemy queries directly in business logic. This meant I could swap PostgreSQL for another database (or add caching) by only touching the repository layer. The Service layer handles password hashing and ETL orchestration without knowing how User objects are stored. This separation also made testing easier, since I could mock repositories without spinning up a database.

Async Patterns for ETL Performance

The ETL pipeline needed to fetch quotes and profiles for potentially dozens of tickers. Running these sequentially would have caused unacceptable latency. Using asyncio.gather let me fire all requests concurrently, cutting execution time by. I used asyncio.to_thread for the Finnhub SDK (which is synchronous) to prevent blocking FastAPI's event loop (a mistake I made in early iterations that caused the entire API to hang during data fetches)

Why SQL and Data Warehousing Matter for Backend Engineers

Most backend engineers interact with databases through ORMs, which is fine for OLTP workloads but fails for analytics. I built this project to understand how data warehouses differ fundamentally from transactional databases. Different query patterns (aggregations vs point lookups), Different schemas (star schema vs normalized), and Different optimization strategies (columnar storage, materialized views). This knowledge directly applies when designing APIs that need to serve analytical queries efficiently.

Medallion Architecture: Why Three Layers?

I chose Medallion (Bronze/Silver/Gold) over alternatives like just loading directly into a star schema because it provides clear boundaries for data quality improvements. Bronze preserves raw source data (critical for auditing), Silver handles cleansing without business logic creep, and Gold serves analytics without impacting transformation pipelines. When data quality issues emerged in the Silver layer, I could fix them without re-ingesting from source or breaking Gold-layer reports.

ELT vs ETL: Processing In-Database

I used ELT (Extract-Load-Transform) instead of ETL because SQL Server can process transformations faster than Python. Loading raw CSVs into Bronze, then transforming in-database using SQL, leverages the database's query optimizer and parallelism. ETL (transforming before loading) makes sense when the source system is too slow or you need complex logic that SQL can't express, but for structured CSV data, in-database transformations were 5x faster in my tests.

React Over Raw HTML/CSS or Other Frameworks

I evaluated three options: vanilla HTML/CSS/JS, React, and Vue. Raw HTML would have meant duplicating features and components across the page and manually managing DOM updates for the appointment form. Vue was tempting for its simplicity, but React's ecosystem (more libraries, better TypeScript support, larger community) made it the pragmatic choice. The component-based model meant I could build Employee.js once and render it with different props—impossible with static HTML without templating engines.