Blog · Performance
Why a Kafka client takes a minute to open a cluster with thousands of topics
You open your Kafka client on the work cluster. Two thousand topics, twenty-eight thousand partitions. Click "connect" and then you wait: ten seconds, thirty, sometimes minutes, with a spinner turning on "Retrieving topics…".
Yet the cluster answers quickly. Where does the time go?
One request or thirty thousand
To show the list of topics, one request to the broker is enough: Metadata. The answer contains every topic, its partitions, leaders and replicas. Even with tens of thousands of partitions it is a reply of a few megabytes that arrives in a fraction of a second.
On a real corporate cluster with 7 brokers, 2,050 topics and 28,319 partitions, with SASL_SSL and certificate verification, we measured 0.77 seconds for the first complete Metadata request, including connection, TLS and authentication.
The problem starts when the client also wants to show how many messages each topic holds. Metadata does not say. For that you need the start and end offsets of every partition, with the ListOffsets request. Twenty-eight thousand partitions, two offsets each, often on different brokers: even with batching, that is a lot of round trips and a lot of work for the brokers.
In our comparison on a networked cluster with 10,003 topics and 27,026 partitions, a client that loads every count took 51 seconds before showing the list. The same list, with names only, arrived in 0.88 seconds. It is not the same work, and that is exactly the point: hardly anyone looks at the counts of every topic at once.
The tricks that make the difference
Ask only for what you show
The topic list does not need counts. When you open a topic, ask for the offsets of its partitions: one batched request, a few tenths of a second.
Draw only the visible rows
A tree with two thousand nodes, or ten thousand, should not be built all at once. A virtual list draws only the rows on screen, fifty or so, and recycles elements as you scroll. That way the filter keeps up as you type, even with tens of thousands of entries.
Keep a local copy
Topic names change rarely. Saving them in an on-disk cache, per connection, lets you show the last known list immediately when you come back to a cluster, and refresh it when you ask.
Reuse the connection
Connection, TLS and SASL authentication cost more than the request itself: in our measurement the first Metadata took 0.77 s, the following ones on the open connection 0.39 and 0.37 s. A client that opens a new connection for every screen pays that price every time.
What to check if your client is slow
- Is the time in the connection or in the list? Try the same cluster with a command-line tool: if it is fast there, the bottleneck is the client.
- Does the client compute counts for every topic on opening? There is often an option to turn it off.
- Does the slowdown grow with the number of topics linearly or worse? A non-virtual tree shows it at once: even scrolling freezes.
Measurement conditions
Metadata: real corporate cluster, SASL_SSL and SCRAM, from the VDI, 16 October 2026, Topicup version 0.1 engine from the command line, no cache. List with and without counts: networked Redpanda cluster with SASL_SSL, median of 3 runs, 21 October 2026; the full method and raw data are on the comparison page.