Lesson 9. Kafka Connect Basics

Mission link: This is the final lesson of the arc: a real pipeline moves data in and out of Kafka through connectors, not hand-written producers and consumers, and this lesson is how Connect's own parallelism and fault-tolerance model relates to everything this workspace already derived, rather than granting anything new.
Primary source: Docs: "Kafka Connect", Apache Kafka
Prerequisites: Lesson 8, Partition

Warm-up

1. ▢ What does a message carry instead of the full schema when Schema Registry is used, and why does that matter for overhead?

2. ▢ Contrast backward and forward compatibility. Which supports upgrading consumers first, and which supports upgrading producers first?

Know this

Source and sink connectors move data in opposite directions

Kafka Connect is a framework for moving data into and out of Kafka without hand-writing a producer or consumer for every integration. A source connector pulls data from an external system, a database, a file, an API, into a Kafka topic. A sink connector pushes data from a Kafka topic into an external system. Most common integrations already have existing connector plugins (a JDBC source or sink, an S3 sink, a change-data-capture source like Debezium) that only need configuring, not writing from scratch.

Tasks are Connect's own unit of parallelism

A connector splits its work across one or more tasks. A source connector might split by table or by partition of the external system; a sink connector's tasks map onto a subset of the topic's partitions, the same way a consumer group's members do, since a sink connector's tasks are, under the hood, essentially a consumer group. This is the same parallelism idea lesson 1 introduced for partitions and lesson 2 introduced for consumer group members, now expressed at Connect's own layer: tasks are the unit that determines how much of a connector's work can run concurrently.

Distributed mode gets its fault tolerance from the same rebalancing idea

Standalone mode runs Connect as a single process with its configuration in local files: simple, fine for development, with no fault tolerance if that process dies. Distributed mode runs multiple worker processes that coordinate with each other using Kafka itself, storing connector configurations, task assignments, and offsets in internal topics, the same pattern lesson 2's __consumer_offsets topic used, reusing Kafka's own log as the coordination substrate. If a worker process dies in distributed mode, its tasks get reassigned to the remaining workers, directly analogous to lesson 2's consumer group rebalancing when a member's heartbeats stop arriving; this is exactly what makes distributed mode the production-appropriate choice over standalone.

Connect grants no guarantee the underlying API didn't already provide

A sink connector consuming from a topic still inherits whatever delivery guarantee (lessons 4 and 5) and ordering guarantee (lesson 6) that topic and its consumer-side configuration actually provide; Connect doesn't add a new guarantee beyond what the producer and consumer APIs it wraps already give. Exactly-once behavior for a source connector still depends on the same idempotent-producer and transaction machinery lesson 5 derived, not on something Connect invents independently. A source connector's output commonly integrates with Schema Registry directly, converting the external system's native schema into a registered schema automatically, which is where lesson 8's compatibility-mode concerns meet whatever schema evolution the external system does over time. Connect is an operational convenience, not a different set of underlying rules.

Practice

1. ▢ Distinguish a source connector from a sink connector. Which direction does each move data?

2. ▢ Describe how a connector's tasks provide parallelism, and how that's conceptually similar to lesson 1's partition-as-parallelism-unit and lesson 2's consumer group parallelism.

3. ▢ Contrast standalone and distributed mode. Which provides fault tolerance if a worker process dies, and how does that recovery relate to lesson 2's rebalancing?

4. ▢ Does using Kafka Connect grant a new delivery guarantee beyond what the underlying producer and consumer APIs already provide? Why or why not?

5. ▢ Which claim is true of Kafka Connect?

   • a) Connect invents its own delivery guarantees, independent of the idempotent-producer and transaction mechanisms this workspace already covered
   • b) A connector's tasks are its own unit of parallelism, and distributed mode reassigns a dead worker's tasks the same way a consumer group rebalances a dead member's partitions
   • c) Standalone mode is the production-appropriate choice, since it requires no coordination overhead
   • d) A sink connector's tasks have no relationship to the topic's own partitions

Real-world reps

• [ ] Find an existing connector plugin (JDBC, S3, Debezium, or similar) for an integration you have or are considering, and read its configuration options for task count and any delivery-guarantee-related settings.
• [ ] For a Connect deployment you have access to, check whether it's running in standalone or distributed mode, and what that implies for fault tolerance if a worker process fails.
• [ ] Tomorrow: revisit this workspace's mission in README.md and confirm, in your own words, that you can design a topic, partition, and consumer-group layout for a stated workload, diagnose a lagging or rebalancing consumer, and state what exactly-once actually costs.

Going further

• Docs: "Kafka Connect", Apache Kafka
• Docs: "Schema Registry", Confluent
• Resources

Not landing? Reread the primary source at the top, since this lesson compresses it and compression is where understanding leaks. Check the glossary for any term that felt slippery.

If the lesson itself is unclear rather than the material, that is a defect: open an issue.