AI Agents Need More Than Kafka to See Your Data
By 2026, publishing events to Kafka is table stakes. Your services emit domain events, a schema registry enforces the contracts, dead-letter queues catch the poison messages, and Kafka Connect shoves the rest into some warehouse. Almost none of that can answer the question leadership actually asks: what is happening right now?
The events sit in a topic. The state they describe is not queryable. That gap is where the real 2026 architecture work lives.
Kafka is doing three different jobs
Kafka quietly handles three very different jobs, and teams blur them together. The first is point-to-point messaging: service A publishes, service B consumes and acts. Consumer groups, offsets, partition assignment. Kafka does this well and you need nothing more. The second is aggregation: join streams, window over time, flag anomalies, where a SQL-based stream processor earns its keep. The third is new. An agent wants to ask a question and get an answer, and it cannot do that by tailing a topic. It needs the current state kept queryable as events arrive.
The classic stack has no clean answer for that third job. You build a consumer application that keeps its own state store, which is now Flink plus Postgres plus a cache, or you run a batch job that refreshes a table every fifteen minutes and hand the agent a snapshot that is already stale.
A streaming SQL layer keeps the state fresh
RisingWave is the tool most teams land on here, and it is open source under Apache 2.0 with roughly nine thousand stars on GitHub. The idea is a database that ingests continuously and serves continuously. It reads from Kafka, Pulsar, Kinesis, database CDC, webhooks, and even historical S3 data. It computes materialized views incrementally: when an upstream row changes, only the affected results recompute, not the whole view. End-to-end freshness sits under a hundred milliseconds, and queries serve at 10 to 20 milliseconds p99.
The practical effect is that a materialized view stores the computed answer. "Total revenue by product this week" is one row the agent reads, not a GROUP BY over millions of events at query time. Because it speaks the PostgreSQL wire protocol, you connect with psycopg2, pgx, node-postgres, or any JDBC driver. LangChain and LlamaIndex point at it with a normal postgres connection string and never know it is a stream.
In most shops this collapses a Debezium plus Kafka plus Flink plus a separate serving database into a single system. That is a meaningful reduction in the number of moving parts you are on call for.
CREATE MATERIALIZED VIEW order_sla AS
SELECT order_id,
(paid_at - created_at) AS time_to_pay,
shipped_at IS NOT NULL AS fulfilled
FROM orders
WHERE created_at > now() - interval '24 hours';Where the agents fit in
An agent is built for a query-and-reason loop, not event consumption. Hand it a Kafka topic and it has to replay and aggregate on every decision. Hand it a materialized view and it just runs a SELECT. The loop is observe, think, act: read a fresh view, reason, act. The whole thing finishes in milliseconds.
This is not a fight with your vector database. They do different work:
- A streaming SQL database answers exact questions with fresh, incremental data. "What is product 4's price right now?" is a row, not a scan.
- A vector database answers similar questions with minutes-to-hours freshness. "Which docs mention late events?"
The combined setup gives the agent both: exact, live facts from the streaming database, semantic retrieval from the vector store. RisingWave can even sink a prepared document view, product data joined with descriptions, back to a Kafka topic your embedding pipeline consumes, so the vector store stays current too.
Exposing the views without a wrapper
This is the part that made me stop. You do not hand-write a tool per view. Because RisingWave is Postgres-compatible, any PostgreSQL MCP server connects, discovers the materialized views through information_schema, and exposes them as tools automatically. Each view's columns and types become the tool's self-describing parameters, and the agent turns a natural-language question into SQL against the discovered views. One connection now serves dashboards, apps, and agents.
How the pieces fit together
A layout I have seen settle: PostgreSQL takes the high-frequency writes, your agent_events table, where ACID matters. RisingWave maintains the read views incrementally, so current state per agent is a plain SELECT. Write and read load scale independently. The views sink to Kafka for downstream services and to managed Iceberg for audit. An MCP server sits in front, and the agents write their own audit events back to Kafka, closing the observability loop.
It is also where distributed transactions go quiet. A view that joins order_created, payment_processed, and inventory_reserved surfaces the sagas stuck at one step, payment went through but the reservation never arrived. That is observability, not orchestration. The database does not run your saga; it shows you the ones that stalled. And schema evolution stays boring: the producer adds an optional field, the registry enforces compatibility, the new column appears in the view on the next event.
Where this is the wrong tool
Do not reach for it if your problem is point-to-point messaging; consumer groups are enough and the extra layer is dead weight. If your transformations need heavy custom stateful logic that SQL will not express, Flink or a purpose-built consumer fits better. And the streaming mental model, incremental computation, watermarks, event time versus processing time, has a real learning curve. It is a database for continuous reads, not a high-frequency OLTP write store.
None of this means rip out the pipeline. It means adding one layer that has been missing: a queryable view of the state your events already describe, kept current by the stream itself. When agents, dashboards, and auditors all read the same fresh row, the topic finally becomes an asset instead of a log nobody can read.
Comments