Kafka

 KAFKA

    Apache kafka is highly scalable and distributed platform for creating and processing the streams in RealTime.

 

Messaging system having 3 components : 

•     Producer 
•     Consumer
•     Broker 

Kafka Works as a pub-sub messaging system.

Kafka consists of 5 components : 

•     Kafka Broker
•     Kafka Client API
•     Kafka Connect
•     Kafka Stream
•     Kafka KSQL

Broker : Using broker we can exchange the data between the producer and consumer

Cluster of Brokers : Each cluster runs in one instance of the each kafka broker.

Topic : Unique name for a data stream. 

Topic Partitions : We can break the topics into smaller partitions  and store those partitions into multiple cluster.

Partition Offset : Offset number starts with 0 and continues.

Consumer Group : Multiple consumer can form a group and share the work together.

Kafka Connect :

    Kafka connect is a internal system in kafka which is using to connecting and moving the data into external systems.

•     Source Connector 
•     Sink Connector

Kafka streams vs SparkStreaming, Nifi, Flink : 

1. Kafka No cluster required but in spark and others required.
2. Kafka Streaming is per data streaming, but others is micro batch streaming 
3. Kafka Scaling is easy by just adding a java process

Kafka Streams : 

    Core conepts of Kafka Streams : 

Terminology : 

    Kafka streams : is a sequence of immutable data records, that fully ordered, can be replaced, and is fault tolerant.

    Kafka Stream processor : transforms the incoming streams, record by record and create a new stream from it.

Topology : Nothing but full graph 

    

    Source processor : is a specical processor that takes directly its data directly from topic. It has no predecessors in a topology, and doesn't transform the data.

    Sink Processor : It doesn't have children. It sends stream data directly to the kafka topic.

KStream vs KTable : 

Kafka Schema Registry : 

    Normally Kafka consumers will not do the validations. It will consume in byte code and produces in byte codes. 

    To validate the data it should be seperate and it should be communicate with consumer and producer. 

    So Schema Registry came into the picture and its seperated and producer and consumer can communicate with that and should be light weight to imporve the performance. 

Common data format must be agreed up on : 

            ==> it should be supports schema 

            ==> it should be supports evolution

            ==> it must be light weight

So confluent schema registry for Schema related issues 

Apache AVRO to supports all data formats issues.

Avro Schemas & Avro in java :  

JD of Kafka :     

1. Experience in development of Event based architecture, messaging frameworks and stream processing solutions using Kafka Messaging framework

2. Strong knowledge and experience with Kafka Streams API, Kafka Connect, Kafka brokers, zookeepers, API frameworks, Pub/Sub patterns, schema registry, KSQL, Rest proxy, Replicator, ADB, Operator and Kafka Control centre  

3. Hands on experience on Kafka connectors such as MQ connectors, Elastic search connectors, JDBC connectors, File stream connector. Provide expertise and hands on experience in custom connectors using the Kafka core concepts and API

4. Create topics, setup redundancy cluster, deploy monitoring tools, alerts and has good knowledge of best practices. Experience in building Kafka producer and consumer applications using Spring Boot