Kafka - Master Avro, the Confluent Schema Registry and Kafka REST Proxy. Build Avro Producers/Consumers, Evolve Schemas
About This Video
Kafka - Master Avro, the Confluent Schema Registry and Kafka REST Proxy. Build Avro Producers/Consumers, Evolve Schemas
About This Video
In Detail
This is the new volume in the Apache Kafka Series! Learn Apache Avro, the confluent schema registry for Apache Kafka and the confluent REST proxy for Apache Kafka. Join hundreds of knowledge savvy students in learning some of the most important components in a typical Apache Kafka stack. Apache Avro is one of the most powerful and most popular fast data serialisation mechanism with Apache Kafka: Schema is a first class citizen of the data; data serialisation format that is lightweight; documentation is embedded in the schema; easy to read and write using many languages; fast and big data de-facto data serialisation format. Confluent schema registry for Apache Kafka is the de-facto standard way of storing Avro schemas for your Apache Kafka topics: Stores a versioned history of all your schemas in Apache Kafka; supports and enforces conditions for schema evolution (backward, forward, full compatibility): Kafka Avro serialiser and deserialiser automatically integrate with Apache Kafka and confluent schema registry; make your messages much smaller and lighter! Perfect for increasing Apache Kafka throughput and performance. Confluent REST Proxy is the perfect way to communicate for sending Avro data using non Java languages to Apache Kafka: Write and read binary, JSON and Avro data to Apache Kafka using an HTTP REST API; interact with Apache Kafka using any programming language (not just Java); consult topic list and topic metadata in Apache Kafka
Downloading the example code for this course: You can download the example code files for all Packt video courses you have purchased from your account at http://www.PacktPub.com. If you purchased this course elsewhere, you can visit http://www.PacktPub.com/support and register to have the files e-mailed directly to you.
Chapter 1 : Course Introduction
The need of schemas in Kafka 00:04:11
Course Structure & Objectives 00:05:05
About your instructor 00:00:47
Architecture for Kafka with the Schema Registry and REST Proxy 00:02:26
Chapter 2 : Avro Schemas
What is Avro? 00:08:20
Avro Primitive Types 00:02:17
Avro Record Schema Definition 00:06:41
Avro Complex Types 00:05:47
Practice Exercise: Customer & CustomerAddress 00:04:52
Avro Logical Types 00:02:47
The complex case of Decimals 00:04:01
Avro Section Summary 00:00:37
Chapter 3 : Avro in Java
Section Objective 00:01:16
Generic Record in Avro - Hands On 00:20:10
Specific Record in Avro - Hands On 00:13:34
Check-in on now vs later in Kafka 00:01:44
Avro Tools - Hands On 00:04:16
Reflection in Avro - Hands On 00:04:44
Schema Evolution - Theory 00:07:42
Schema Evolution - Hands On 00:07:34
Avro Java Section Summary 00:01:51
Chapter 4 : Setup and Launch Kafka
Docker on Mac (recent versions) 00:03:54
Docker Toolbox on Mac (older versions) 00:05:17
Docker on Linux (Ubuntu as an example) 00:04:56
Docker on Windows 10 64bit 00:03:43
Docker Toolbox on Windows (older versions) 00:05:37
Starting Kafka using Docker Compose 00:04:36
Chapter 5 : Confluent Schema Registry and Kafka
Confluent Schema Registry 00:07:45
Kafka Avro Console Producer & Consumer 00:09:29
Writing a Kafka Avro Producer in Java 00:12:59
Writing a Kafka Avro Consumer in Java 00:07:40
Reminder on Schema Evolution 00:01:04
Writing a V2 Kafka Producer 00:06:59
Writing a V2 Kafka Consumer 00:03:45
Summary on Compatibility Changes 00:01:15
Kafka Schema Registry Deep Dive 00:04:32
Managing Schemas Efficiently & Section Summary 00:02:13
Chapter 6 : Confluent REST Proxy
Kafka REST Proxy Introduction and Purpose 00:03:54
V1 vs V2 APIs 00:02:18
Insomnia Setup (REST Client) 00:01:27
Topic Operations 00:03:26
Producing in Binary with the Kafka REST Proxy 00:07:15
Consuming in Binary with the Kafka REST Proxy 00:05:08
Producing in JSON with the Kafka REST Proxy 00:03:18
Consuming in JSON with the Kafka REST Proxy 00:03:26
Producing in Avro with the Kafka REST Proxy 00:05:36
Consuming in Avro with the Kafka REST Proxy 00:03:42
Chapter 7 : Annexes
Full Avro End to End: Kafka Producer + Kafka Connect + Kafka Streams 00:19:00
Kafka REST Proxy Installation and Scaling - Overview 00:03:39