Apache Kafka Explained: The Pipe Carrying the Internet

The second you hit play on a song, a message fires across the internet faster than you can blink — and most of the time, Kafka is carrying it. So, what even is Kafka? Apache Kafka is an open-source platform built at LinkedIn around 2011 because they had a real problem. Dozens of different systems needed to talk to each other at the same time, and doing that without a central hub was chaos. They built Kafka, then gave it to the world through the Apache Software Foundation. Here is how it works. Apps that send data are called producers. They drop messages into labeled channels called topics — think of a topic like a mailbox with a name on it. Apps that read that data are called consumers. They pick up messages whenever they are ready. And the machines running all of this are called brokers. Add more brokers, and Kafka automatically spreads the work across them. Now here is what makes Kafka genuinely different. A regular message system deletes a message the second someone reads it — gone. Kafka keeps every single event, saved in order, with a timestamp, like a logbook that never gets erased. That means any system can go back and replay everything from the beginning. Topics are also split into partitions — separate lanes that let many consumers read at the same time without bumping into each other. More lanes, more speed. LinkedIn was processing over a trillion messages a day by the mid-2010s using exactly this. Netflix uses it. Uber uses it. Banks use it to catch fraud before you even notice something is wrong. Every click, every payment, every song play happening on the internet right now is probably a Kafka event — because the internet does not just store data, it streams it, and Kafka is most of the pipe carrying all of it.