Apache kafka is highly scalable and distributed platform for creating and processing the streams in RealTime.
Messaging system having 3 components :
Producer
Consumer
Broker
Kafka Works as a pub-sub messaging system.
Kafka consists of 5 components :
Kafka Broker
Kafka Client API
Kafka Connect
Kafka Stream
Kafka KSQL
Broker : Using broker we can exchange the data between the producer and consumer
Cluster of Brokers : Each cluster runs in one instance of the each kafka broker.
Topic : Unique name for a data stream.
Topic Partitions : We can break the topics into smaller partitions and store those partitions into multiple cluster.
Partition Offset : Offset number starts with 0 and continues.
Consumer Group : Multiple consumer can form a group and share the work together.
Kafka Connect :
Kafka connect is a internal system in kafka which is using to connecting and moving the data into external systems.
Source Connector
Sink Connector
Kafka streams vs SparkStreaming, Nifi, Flink :
Kafka No cluster required but in spark and others required.
Kafka Streaming is per data streaming, but others is micro batch streaming
Kafka Scaling is easy by just adding a java process
Kafka Streams :
Core conepts of Kafka Streams :
Terminology :
Kafka streams : is a sequence of immutable data records, that fully ordered, can be replaced, and is fault tolerant.
Kafka Stream processor : transforms the incoming streams, record by record and create a new stream from it.
Topology : Nothing but full graph
Source processor : is a specical processor that takes directly its data directly from topic. It has no predecessors in a topology, and doesn't transform the data.
Sink Processor : It doesn't have children. It sends stream data directly to the kafka topic.
KStream vs KTable :
Kafka Schema Registry :
Normally Kafka consumers will not do the validations. It will consume in byte code and produces in byte codes.
To validate the data it should be seperate and it should be communicate with consumer and producer.
So Schema Registry came into the picture and its seperated and producer and consumer can communicate with that and should be light weight to imporve the performance.
Common data format must be agreed up on :
==> it should be supports schema
==> it should be supports evolution
==> it must be light weight
So confluent schema registry for Schema related issues
Apache AVRO to supports all data formats issues.
Avro Schemas & Avro in java :
JD of Kafka :
Experience
in development of Event based architecture, messaging frameworks and
stream processing solutions using Kafka Messaging framework
Strong
knowledge and experience with Kafka Streams API, Kafka Connect,
Kafka brokers, zookeepers, API frameworks, Pub/Sub patterns, schema
registry, KSQL, Rest proxy, Replicator, ADB, Operator and Kafka
Control centre
Hands
on experience on Kafka connectors such as MQ connectors, Elastic
search connectors, JDBC connectors, File stream connector. Provide
expertise and hands on experience in custom connectors using the
Kafka core concepts and API
Create
topics, setup redundancy cluster, deploy monitoring tools, alerts
and has good knowledge of best practices. Experience in building
Kafka producer and consumer applications using Spring Boot
Java 8 : Lamda expressions, Functional Interfaces, Default and Static interfaces, Streams, Completable features and New Date and time functions are introduced
Java 11 : Local Variable syntax changes in Lamda, Enhanced Streams and Collections concepts and HttpClient is introducted
Java 21 : Virtual Threads, Pattern Matching in swith, Record Patterns and Sequence Collections are introduced
Immutable Class : A class which is not having setters and who's instance cannot be change after they are created.
Declaring a final class
Make all fields are private
Make all fields are final.
Donot provide the setters.
Advantages of Immutable class :
Thread safety
Security & Consistency
Reliable Hash keys
Microservices Design Patterns :
Circuit Breaker DP :
Circuit breaker implemented using Resillance4J and it has 3 components : CLOSED, OPEN and HALF-OPEN.
CLOSED : When failure rate threshold is below
OPEN : When failture rate threshold is above
HALF-OPEN : After wait durtaion it will go to HALF-OPEN
Circuit breaker uses two types of sliding windows to store and aggregate the outcome of calls.
1. Count based sliding window
2. Time-based sliding window
Bulk Head Pattern : 2 types of SemaphoreBulkhead and FixedThreadPoolBulkhead
Rate Limtter Design Pattern : Rate limiting is an imperative technique to prepare your API for scale and establish high availability and reliability of your service.
Retry Design Pattern : Just like the CircuitBreaker module, this module provides an in-memory RetryRegistry which you can use to manage (create and retrieve) Retry instances.
Saga Design Pattern :
Event Driven Approach Design Pattern :
Kafka based event driven approach
Database Design Pattern :
Indentity Design Pattern : Security Identity Management (verifying who is making requests) and Domain Data Identity (how data entities maintain their identifiers across service boundaries)
Feign Client vs Rest Client :
The primary difference is that Feign Client is declarative (you write an interface and let the framework generate the HTTP code), while a Rest Client is programmatic/fluent (you manually write the steps to build and execute the request).
Microservice vs Monolithic :
A monolithic architecture consolidates all software components into a single program, whereas a microservices architecture divides the application into separate, self-contained services.
When to Use Microservices
Microservices are advantageous for certain types of projects:
Complex Systems
Scalability
Technology Diversification
Autonomous Teams: For bigger organizations with multiple teams that need to work independently.
Challenges while using Microservices :
Database per service
Data inconsistency
Integrate Testing
How Microservies communicate each other :
Synchronous
Asynchronous
Restful api's
Event Based communication
Database per service
API-Gateway
How would you decompose a monolithic application into microservices?
Identify Domains
Service Boundaries
Data Segrigation
Decouple services
Kafka Based Interview Questions :
Kafka consumer vs Consumer Group :
Kafka consumer reads the data from the topic
Consumer group is a set of consumers work together and reads one or more topics.
OffSet :
Offset is a unique sequential identifier record with in a partition.
Kafka tracks the offset per partition, per consumer group. So each group can consumes its own position.
How does kafka handles data retention :
Retention can be time based, once it reaches to limit old message will be discarded.
Retention limit will be provided while creating the kafka clusters.
How Kafka ensure the data consistency :
Replication : Each partition is replicated across the mulitple brokers
Acknowlegements : Producers can wait for leaders only
Atomic, orders writes to a partition
Idempotent producers to prevent duplicate writes on retry.
KRaft :
To manage the metadata management kafka introduced the KRaft by removing the dependency on Apache ZooKeeper.
Streams
Parllel Stream vs Stream :
Streams :
Run's single thread and results are predictable
Sequential
Low overheaded
Parllel Streams :
Run's on multi threaded and results are unpredictable.
Parllel
High Overheaded due to thread management.
Map vs Flat Map :
Map :
One to one mapping
To use basic data transformation
FlatMap :
One to Many (zero)mapping
To handling collection of collections
Tweleve-Factor App Concepts in java :
CodeBase
Dependencies
Configuration
Backing services
Build, Release, Run
Processes
Port Binding
Concurrency
Disposability
Dev/Prod Parity
Logs
Admin Process
AWS Interview Questions :
What is Cloud Computing :
Cloud computing provides on-demand access to IT resources like compute, storage, and databases over the internet.
3 Types of Cloud Computing :
SAAS (Software as a service) : Aws email service etc services by AWS.
PAAS (Platform as a service) : Elastic Beanstalk, Heroko.
IAAS (Infrastructure as a service) : EC2 instance, S3 Storage, VPC.
EC2 Instance : Elasic Cloud Computing
Scalable virtual servers called instances in AWS
EC2 instances are used to host websites
Run batch job process to acheive scalability
S3 Storage :
Simple Storage Service
Stores the objects in secure way
IAM :
Identity Access Management
Helps you to securely access to AWS services
IAM allows us to manage users & Roles.
RDS :
Relational Database Service
To Manage the database service
VPC :
Virtual Private Network
To create a virtual network in AWS.
AWS Cloud Watch :
Uses for monitoring purposes
Metrics, Alarms, Logs, Events
AWS Lamda :
Server less Compute service
ELB : Elastic Load Balancer
EBS : Elasic Bean Stalk
Blue Green Deployment :
Blue means current version deployment
Green means new version deployment
ECS : Elastic Container System
Replacement to kubernetes, this simplifier to run EC2 Instances.
AWS Code Build : Compile the code, run tests, produce deployable artificats
AWS Code Deploy : Deploy into AWS Environment
AWS Code Pipeline : Automate the deployment process using CICD workflow.
Amazon ECS : Deploying the Docker based applications into ECS Containers.
AWS Secrets : Maintain the sensitive information into secret manager service.
Migrating the Old Application into New Applicaiton in AWS :
