Apache kafka is highly scalable and distributed platform for creating and processing the streams in RealTime.
Messaging system having 3 components :
Producer
Consumer
Broker
Kafka Works as a pub-sub messaging system.
Kafka consists of 5 components :
Kafka Broker
Kafka Client API
Kafka Connect
Kafka Stream
Kafka KSQL
Broker : Using broker we can exchange the data between the producer and consumer
Cluster of Brokers : Each cluster runs in one instance of the each kafka broker.
Topic : Unique name for a data stream.
Topic Partitions : We can break the topics into smaller partitions and store those partitions into multiple cluster.
Partition Offset : Offset number starts with 0 and continues.
Consumer Group : Multiple consumer can form a group and share the work together.
Kafka Connect :
Kafka connect is a internal system in kafka which is using to connecting and moving the data into external systems.
Source Connector
Sink Connector
Kafka streams vs SparkStreaming, Nifi, Flink :
Kafka No cluster required but in spark and others required.
Kafka Streaming is per data streaming, but others is micro batch streaming
Kafka Scaling is easy by just adding a java process
Kafka Streams :
Core conepts of Kafka Streams :
Terminology :
Kafka streams : is a sequence of immutable data records, that fully ordered, can be replaced, and is fault tolerant.
Kafka Stream processor : transforms the incoming streams, record by record and create a new stream from it.
Topology : Nothing but full graph
Source processor : is a specical processor that takes directly its data directly from topic. It has no predecessors in a topology, and doesn't transform the data.
Sink Processor : It doesn't have children. It sends stream data directly to the kafka topic.
KStream vs KTable :
Kafka Schema Registry :
Normally Kafka consumers will not do the validations. It will consume in byte code and produces in byte codes.
To validate the data it should be seperate and it should be communicate with consumer and producer.
So Schema Registry came into the picture and its seperated and producer and consumer can communicate with that and should be light weight to imporve the performance.
Common data format must be agreed up on :
==> it should be supports schema
==> it should be supports evolution
==> it must be light weight
So confluent schema registry for Schema related issues
Apache AVRO to supports all data formats issues.
Avro Schemas & Avro in java :
JD of Kafka :
Experience
in development of Event based architecture, messaging frameworks and
stream processing solutions using Kafka Messaging framework
Strong
knowledge and experience with Kafka Streams API, Kafka Connect,
Kafka brokers, zookeepers, API frameworks, Pub/Sub patterns, schema
registry, KSQL, Rest proxy, Replicator, ADB, Operator and Kafka
Control centre
Hands
on experience on Kafka connectors such as MQ connectors, Elastic
search connectors, JDBC connectors, File stream connector. Provide
expertise and hands on experience in custom connectors using the
Kafka core concepts and API
Create
topics, setup redundancy cluster, deploy monitoring tools, alerts
and has good knowledge of best practices. Experience in building
Kafka producer and consumer applications using Spring Boot
Java 8 : Lamda expressions, Functional Interfaces, Default and Static interfaces, Streams, Completable features and New Date and time functions are introduced
Java 11 : Local Variable syntax changes in Lamda, Enhanced Streams and Collections concepts and HttpClient is introducted
Java 17 :
Java 21 : Virtual Threads, Pattern Matching in switch, Record Patterns and SequencedCollection are introduced
Immutable Class : A class which is not having setters and who's instance cannot be change after they are created.
Declaring a final class
Make all fields are private
Make all fields are final.
Donot provide the setters.
Advantages of Immutable class :
Thread safety
Security & Consistency
Reliable Hash keys
Multithreading :
CompletableFuture vs Future :
supplyAsync() — starting async tasks the right way (Java 8 CompletableFuture supplyAsync example)
thenApply() & thenAccept() — transforming and consuming results (Multiple chained Futures cannot combined together using Future)
thenCombine() — combining two futures into one (Multiple futures cannot combined together)
exceptionally() — exception handling in CompletableFuture (Poor exception handling in Future)
Chaining of futures vs combining futures — when to use each
Executor Service :
Java supports Thread, Runnable, Callable and ExecutorService. In production, prefer managed executors instead of creating raw threads per task. In Java 21, virtual threads are a strong option for high-concurrency blocking I/O workloads.
Virtual Threads :
Its a light weight threads which are managed by the JVM rather than OS.
Its java 21 and Springboot 3.2+ onwards.
Platform Threads : Traditional OS Managed threads
Virtual Threads : JVM managed threads.
Virtual threads are useful at Async Processing by replacing the traditional thread pool executors with virtual thread exectuors in SpringBoot Async configuration.
Advantages :
Higher Concurrency
Lower Memory Overhead
Better scalability for I/O bound operations
Use Cases :
API Gateway Parllel processing of larger datasets.
Real time Systems like Chat applicatoins, Live Updates, Trackers
I/O Bound operations like Database calls, HTTP requests, file Operations
Handles millions of Concurrent Request.
Microservices Design Patterns :
Circuit Breaker DP :
Circuit breaker implemented using Resillance4J and it has 3 components : CLOSED, OPEN and HALF-OPEN.
CLOSED : When failure rate threshold is below
OPEN : When failture rate threshold is above
HALF-OPEN : After wait durtaion it will go to HALF-OPEN
Circuit breaker uses two types of sliding windows to store and aggregate the outcome of calls.
1. Count based sliding window
2. Time-based sliding window
Bulk Head Pattern : 2 types of SemaphoreBulkhead and FixedThreadPoolBulkhead
Rate Limtter Design Pattern : Rate limiting is an imperative technique to prepare your API for scale and establish high availability and reliability of your service.
Retry Design Pattern : Just like the CircuitBreaker module, this module provides an in-memory RetryRegistry which you can use to manage (create and retrieve) Retry instances.
Saga Design Pattern :
Event Driven Approach Design Pattern :
Kafka based event driven approach
Database Design Pattern :
Indentity Design Pattern : Security Identity Management (verifying who is making requests) and Domain Data Identity (how data entities maintain their identifiers across service boundaries)
Feign Client vs Rest Client :
The primary difference is that Feign Client is declarative (you write an interface and let the framework generate the HTTP code), while a Rest Client is programmatic/fluent (you manually write the steps to build and execute the request).
Microservice vs Monolithic :
A monolithic architecture consolidates all software components into a single program, whereas a microservices architecture divides the application into separate, self-contained services.
When to Use Microservices
Microservices are advantageous for certain types of projects:
Complex Systems
Scalability
Technology Diversification
Autonomous Teams: For bigger organizations with multiple teams that need to work independently.
Challenges while using Microservices :
Database per service
Data inconsistency
Integrate Testing
How Microservies communicate each other :
Synchronous
Asynchronous
Restful api's
Event Based communication
Database per service
API-Gateway
How would you decompose a monolithic application into microservices?
Identify Domains
Service Boundaries
Data Segrigation
Decouple services
Kafka Based Interview Questions :
Kafka consumer vs Consumer Group :
Kafka consumer reads the data from the topic
Consumer group is a set of consumers work together and reads one or more topics.
OffSet :
Offset is a unique sequential identifier record with in a partition.
Kafka tracks the offset per partition, per consumer group. So each group can consumes its own position.
How does kafka handles data retention :
Retention can be time based, once it reaches to limit old message will be discarded.
Retention limit will be provided while creating the kafka clusters.
How Kafka ensure the data consistency :
Replication : Each partition is replicated across the mulitple brokers
Acknowlegements : Producers can wait for leaders only
Atomic, orders writes to a partition
Idempotent producers to prevent duplicate writes on retry.
KRaft :
To manage the metadata management kafka introduced the KRaft by removing the dependency on Apache ZooKeeper.
How to fix the lag issues in Kafka :
Increase the Partition count
Add more consumers instances :
Ensure 1:1 Ratio : Align the no of active consumers with no of topic partitions.
Streams
Parllel Stream vs Stream :
Streams :
Run's single thread and results are predictable
Sequential
Low overheaded
Parllel Streams :
Run's on multi threaded and results are unpredictable.
Parllel
High Overheaded due to thread management.
Map vs Flat Map :
Map :
One to one mapping
To use basic data transformation
FlatMap :
One to Many (zero)mapping
To handling collection of collections
Tweleve-Factor App Concepts in java :
CodeBase
Dependencies
Configuration
Backing services
Build, Release, Run
Processes
Port Binding
Concurrency
Disposability
Dev/Prod Parity
Logs
Admin Process
AWS Interview Questions :
What is Cloud Computing :
Cloud computing provides on-demand access to IT resources like compute, storage, and databases over the internet.
3 Types of Cloud Computing :
SAAS (Software as a service) : Aws email service etc services by AWS.
PAAS (Platform as a service) : Elastic Beanstalk, Heroko.
IAAS (Infrastructure as a service) : EC2 instance, S3 Storage, VPC.
EC2 Instance : Elasic Cloud Computing
Scalable virtual servers called instances in AWS
EC2 instances are used to host websites
Run batch job process to acheive scalability
S3 Storage :
Simple Storage Service
Stores the objects in secure way
IAM :
Identity Access Management
Helps you to securely access to AWS services
IAM allows us to manage users & Roles.
RDS :
Relational Database Service
To Manage the database service
VPC :
Virtual Private Network
To create a virtual network in AWS.
AWS Cloud Watch :
Uses for monitoring purposes
Metrics, Alarms, Logs, Events
AWS Lamda :
Server less Compute service
ELB : Elastic Load Balancer
EBS : Elasic Bean Stalk
Blue Green Deployment :
Blue means current version deployment
Green means new version deployment
ECS : Elastic Container System
Replacement to kubernetes, this simplifier to run EC2 Instances.
AWS Code Build : Compile the code, run tests, produce deployable artificats
AWS Code Deploy : Deploy into AWS Environment
AWS Code Pipeline : Automate the deployment process using CICD workflow.
Amazon ECS : Deploying the Docker based applications into ECS Containers.
AWS Secrets : Maintain the sensitive information into secret manager service.
Migrating the Old Application into New Applicaiton in AWS :
Must follow the 7 Rs Framework
Rehost
Replatform
Repurchase
Refactor
Retire
Retain
Relocate
Used EC2 and RDS to migrate also important.
DataStructures :
Array
String
Linked List
Queue
Stack
Tree
Graph
Hashing
SpringBoot ::
Spring Boot Scopes::
Singleton
Prototype
Request
Session
Application
Websocket
Exceptino Handling :
CustomException ::
A custom exception is a class created by the developer to represent a specific error in the application.
Examples: Invalid account number, Insufficient balance, Payment failed, Flight not found
Serilization vs De-Serialization ::
HashMap Internally working technique ::
When we will do the HashCode and Equals() :
when ever both the string object values are same then we will use hashcode and equals().
What is a functional interface?
A functional interface has exactly one abstract method and can be used as the target of a lambda or method reference.
why are Streams usually faster than loops for large data?
Streams use internal optimizations like: Stream Pipeline
• Lazy ovaluation
• Mothod chaining
•Built-in parallel proconcing
Syncrnonization ::
Bad practise of synchronization : synchronizing on the boxed type Integer:
private int count = 0;
private final Integer intLock = count; // the solution for synchronization on the boxed primitive is to create a new instance. private final Integer intLock = new Integer(count);
public void boxedPrimitiveBadPractice() {
synchronized (intLock) {
count++;
// ...
}
}
Why does HashMap allow one null key and multiple null values ?
HashMap is docignod to handle null gracofully.
It treats null key's hashCode as O and stores it in bucket O. Multiple null values are allowed because values are not used
in hashing or key comparison.
