ftisiot ideas about food and life
who talks suggestions soft json-pg

Breathe in Breathe out: get Kafka Connect settings right!

Presented September 20, 2022 at Current 2022

Slides Venue

A deep dive into Kafka connect settings and pitfalls, analysing several tips to have a better experience when setting up a connector.

Intro image

Abstract

Kafka Connect is the spell book for creating magical data streaming setups. It allows us to integrate Apache Kafka with the rest of our data ecosystem and get all the data flowing to the right place. However you need some rather dark magic to configure all the weird and wonderful connectors recently, and this talk will teach you the tricks you need.

We’ll talk about streaming data into topics, the data formats to use and what to look out for when Kafka Connect is plugging data from another platform into your setup. Since we don’t live in a perfect world, we’ll also cover configurations like error tolerance, dead letter queues and single message transforms that can make things more robust.

You’ll see some examples of good practices, and hear some stories about how I learned a few of these things the hard way. Finally we’ll shed light on some of the options, like auto evolution, that seem like a great idea when you are prototyping a new solution but which can store up problems for the longer term.

If you are ready to make magic with Kafka Connect and the Apache Kafka ecosystem, this is the talk for you!

Useful Links

  • Aiven for Apache Kafka®
  • Aiven for Apache Kafka® Connect
  • Deep dive into Single Message Transforms
  • JDBC Source connector: what could go wrong?
  • JDBC Sink connector not working with schemaless topics
  • Works on JDBC sink to parse schemaless topics by inferencing the JSON structure
  • Dead Letter Queue - Twitter thread by Gunnar Morling

Check out the Aiven Rolling Challenge


Francesco Tisiot

Mastodon Francesco comes from Verona, Italy and works as a Senior Developer Advocate at Aiven. With his many years of experience as a data analyst, he has stories to tell and advice for data-wranglers everywhere. Francesco loves sharing knowledge with others as a speaker and writer, and is on a mission to defend the world from bad Italian food!

Story logo

© 2023