Apache kafka is highly scalable and distributed platform for creating and processing the streams in RealTime.
Messaging system having 3 components :
Producer
Consumer
Broker
Kafka Works as a pub-sub messaging system.
Kafka consists of 5 components :
Kafka Broker
Kafka Client API
Kafka Connect
Kafka Stream
Kafka KSQL
Broker : Using broker we can exchange the data between the producer and consumer
Cluster of Brokers : Each cluster runs in one instance of the each kafka broker.
Topic : Unique name for a data stream.
Topic Partitions : We can break the topics into smaller partitions and store those partitions into multiple cluster.
Partition Offset : Offset number starts with 0 and continues.
Consumer Group : Multiple consumer can form a group and share the work together.
Kafka Connect :
Kafka connect is a internal system in kafka which is using to connecting and moving the data into external systems.
Source Connector
Sink Connector
Kafka streams vs SparkStreaming, Nifi, Flink :
Kafka No cluster required but in spark and others required.
Kafka Streaming is per data streaming, but others is micro batch streaming
Kafka Scaling is easy by just adding a java process
Kafka Streams :
Core conepts of Kafka Streams :
Terminology :
Kafka streams : is a sequence of immutable data records, that fully ordered, can be replaced, and is fault tolerant.
Kafka Stream processor : transforms the incoming streams, record by record and create a new stream from it.
Topology : Nothing but full graph
Source processor : is a specical processor that takes directly its data directly from topic. It has no predecessors in a topology, and doesn't transform the data.
Sink Processor : It doesn't have children. It sends stream data directly to the kafka topic.
KStream vs KTable :
Kafka Schema Registry :
Normally Kafka consumers will not do the validations. It will consume in byte code and produces in byte codes.
To validate the data it should be seperate and it should be communicate with consumer and producer.
So Schema Registry came into the picture and its seperated and producer and consumer can communicate with that and should be light weight to imporve the performance.
Common data format must be agreed up on :
==> it should be supports schema
==> it should be supports evolution
==> it must be light weight
So confluent schema registry for Schema related issues
Apache AVRO to supports all data formats issues.
Avro Schemas & Avro in java :
JD of Kafka :
Experience
in development of Event based architecture, messaging frameworks and
stream processing solutions using Kafka Messaging framework
Strong
knowledge and experience with Kafka Streams API, Kafka Connect,
Kafka brokers, zookeepers, API frameworks, Pub/Sub patterns, schema
registry, KSQL, Rest proxy, Replicator, ADB, Operator and Kafka
Control centre
Hands
on experience on Kafka connectors such as MQ connectors, Elastic
search connectors, JDBC connectors, File stream connector. Provide
expertise and hands on experience in custom connectors using the
Kafka core concepts and API
Create
topics, setup redundancy cluster, deploy monitoring tools, alerts
and has good knowledge of best practices. Experience in building
Kafka producer and consumer applications using Spring Boot
Java 8 : Lamda expressions, Functional Interfaces, Default and Static interfaces, Streams, Completable features and New Date and time functions are introduced
Java 11 : Local Variable syntax changes in Lamda, Enhanced Streams and Collections concepts and HttpClient is introducted
Java 21 : Virtual Threads, Pattern Matching in swith, Record Patterns and Sequence Collections are introduced
Immutable Class : A class which is not having setters and who's instance cannot be change after they are created.
Declaring a final class
Make all fields are private
Make all fields are final.
Donot provide the setters.
Advantages of Immutable class :
Thread safety
Security & Consistency
Reliable Hash keys
Microservices Design Patterns :
Circuit Breaker DP :
Circuit breaker implemented using Resillance4J and it has 3 components : CLOSED, OPEN and HALF-OPEN.
CLOSED : When failure rate threshold is below
OPEN : When failture rate threshold is above
HALF-OPEN : After wait durtaion it will go to HALF-OPEN
Circuit breaker uses two types of sliding windows to store and aggregate the outcome of calls.
1. Count based sliding window
2. Time-based sliding window
Bulk Head Pattern : 2 types of SemaphoreBulkhead and FixedThreadPoolBulkhead
Rate Limtter Design Pattern : Rate limiting is an imperative technique to prepare your API for scale and establish high availability and reliability of your service.
Retry Design Pattern : Just like the CircuitBreaker module, this module provides an in-memory RetryRegistry which you can use to manage (create and retrieve) Retry instances.
Saga Design Pattern :
Event Driven Approach Design Pattern :
Kafka based event driven approach
Database Design Pattern :
Indentity Design Pattern : Security Identity Management (verifying who is making requests) and Domain Data Identity (how data entities maintain their identifiers across service boundaries)
Feign Client vs Rest Client :
The primary difference is that Feign Client is declarative (you write an interface and let the framework generate the HTTP code), while a Rest Client is programmatic/fluent (you manually write the steps to build and execute the request).
Microservice vs Monolithic :
A monolithic architecture consolidates all software components into a single program, whereas a microservices architecture divides the application into separate, self-contained services.
When to Use Microservices
Microservices are advantageous for certain types of projects:
Complex Systems
Scalability
Technology Diversification
Autonomous Teams: For bigger organizations with multiple teams that need to work independently.
Challenges while using Microservices :
Database per service
Data inconsistency
Integrate Testing
How Microservies communicate each other :
Synchronous
Asynchronous
Restful api's
Event Based communication
Database per service
API-Gateway
How would you decompose a monolithic application into microservices?
Identify Domains
Service Boundaries
Data Segrigation
Decouple services
Kafka Based Interview Questions :
Kafka consumer vs Consumer Group :
Kafka consumer reads the data from the topic
Consumer group is a set of consumers work together and reads one or more topics.
OffSet :
Offset is a unique sequential identifier record with in a partition.
Kafka tracks the offset per partition, per consumer group. So each group can consumes its own position.
How does kafka handles data retention :
Retention can be time based, once it reaches to limit old message will be discarded.
Retention limit will be provided while creating the kafka clusters.
How Kafka ensure the data consistency :
Replication : Each partition is replicated across the mulitple brokers
Acknowlegements : Producers can wait for leaders only
Atomic, orders writes to a partition
Idempotent producers to prevent duplicate writes on retry.
KRaft :
To manage the metadata management kafka introduced the KRaft by removing the dependency on Apache ZooKeeper.
MCP helps to build Modular, Secure and Future Proof AI Integrations. As AI systems connect more deeply with enterprise tools, data, and workflows, one requirement is becoming increasingly important: The integration layer needs standardization. That is where MCP (Model Context Protocol) becomes valuable. MCP gives AI applications a structured way to connect with tools, APIs, data sources, file systems, and enterprise systems through a common protocol pattern. From an architecture perspective, this matters because one of the biggest sources of complexity in AI projects is custom integration logic spread everywhere. What makes MCP useful is the separation of concerns it brings: → Hosts and clients manage the AI application side → MCP servers expose tools, data, resources, and prompts → Backend systems continue to hold business logic and enterprise data → Model providers remain more interchangeable behind the interaction layer This opens up multiple implementation patterns: Basic client-server for simple integrations, multi-server for modularity, gateway patterns for centralized policy control, chained tools for multi-step automation, dynamic discovery for extensibility, and scoped context for stronger isolation and multi-tenant security. My view is simple: MCP is not just another protocol. It is an architectural pattern for building modular, secure, and future-ready AI integrations. That is why I believe MCP will become an important part of enterprise AI design over the next few years.
Interview Questions :
Production issue in kubernetes :
=====================
Core Design Pattersn :
================
Http Status Codes :
=============
EKS + AKS + GKE CICD Pipeline
=========================
CI CD Architecture :
======================
Continuous integration: code is built and tested before merging to the main branch, and the artifacts are created, which will be used for staging and production. That artifact is the exact thing that moves forward, so nothing gets rebuilt later.
Continuous delivery: the build is prepared for the release but won't be deployed to production yet. Every good build still goes through staging and the readiness checks first. It will require a human approval or a planned step before it goes live. This is continuous delivery. You'd keep that gate when a release needs sign-off or a heads-up for customers.
Continuous deployment: If you remove the human or process check and the build is directly deployed on the production, then this is called continuous deployment.
OAuth2 + JWT
===========
N+1 Problem
==========
HLD Vs LLD
==========
Docker Vs Kubernetes :
=================
Docker Architecture :
===============
Kubernetes Commands :
=================
Kafka
=====
A simplified architecture looks like this:
📍 1. Delivery Partner App The rider's app continuously sends GPS coordinates (latitude, longitude, timestamp, order ID, etc.) every few seconds. ⬇️ ⚡ 2. Kafka Instead of sending updates directly to every service, the app publishes them to a Kafka topic (for example, location-updates). Kafka acts as a high-throughput event streaming platform capable of handling millions of events reliably. ⬇️ 🔄 3. Stream Processing Consumer services read these events and: • Validate incoming locations • Filter invalid or duplicate updates • Enrich events with additional data • Execute business logic ⬇️ ⚡ 4. Fast Storage The latest location is stored in a low-latency datastore such as Redis. Keeping only the latest location in memory allows extremely fast reads. ⬇️ 📱 5. Customer Application The customer app continuously fetches (or subscribes to) the latest location and updates the rider's position on the map in real time.
💡 Why Kafka?
Kafka isn't used just because it's popular.
It's chosen because it solves real engineering problems. ✅ Handles millions of location updates per second ✅ Preserves event ordering within partitions ✅ Decouples producers from consumers ✅ Enables multiple downstream services to consume the same event stream independently ✅ Provides fault tolerance and horizontal scalability
🎯 Interview Insight
One thing I've learned is that interviewers rarely want to hear: "Kafka is used for live tracking." Instead, they want to understand why Kafka fits the problem.
They're looking for answers to questions like: • Why stream events instead of making synchronous API calls? • Why introduce a message broker? • How are events processed after Kafka? • Why use Redis for live location storage? • How does the customer finally see the moving rider?
That's what separates knowing a technology from understanding a system. System Design interviews aren't about memorizing tools.
They're about understanding why each component exists and how they work together to build scalable systems.
Securing the API's using JWT and Spring Security :
=====================================
Building Secure APIs using Spring Security & JWT
A secure API is much more than checking a username and password. It's about ensuring every request is authenticated, authorized, traceable, and protected against common security threats. Here are the key building blocks of a production-ready secure API:
1. Authentication – Verify Who the User Is
2. Authorization – Verify What the User Can Access Authorization ensures users only access resources they are permitted to. Examples: 👤 USER -View Profile -Update Own Profile
3. JWT – Stateless Authentication - Instead of maintaining server-side sessions, JWT carries user information inside a signed token. A JWT typically contains: - User ID - Roles / Authorities - Expiration Time - Issued Time - Digital Signature
4. Secure Every Request For each incoming request: a. Extract JWT from the Authorization header b. Validate the token signature c. Check token expiration d. Load user details e. Verify user permissions f. Store authentication in the Security Context g. Allow access to protected APIs, Otherwise → Return 401 Unauthorized or 403 Forbidden
