DeepSeek MLA: 4 Attention Methods Compared

DeepSeek MLA is the focus of this practical guide to MHA, GQA and MQA because its architecture shows how KV cache compression moves from theory into real LLM inference systems.

DeepSeek MLA practical checklist

Use this quick checklist when deciding whether DeepSeek MLA is the right mental model for an attention system. The goal is not to treat DeepSeek MLA as a magic speed switch, but to understand where DeepSeek MLA changes memory, cache layout and inference behavior.

• DeepSeek MLA targets KV-cache memory first, especially during long-context decoding.
• DeepSeek MLA compresses key/value information into a learned latent state.
• DeepSeek MLA keeps positional information separate through the RoPE component.
• DeepSeek MLA is most useful when cache memory becomes a serving bottleneck.
• DeepSeek MLA should be evaluated with real batch size and context length.
• DeepSeek MLA differs from GQA because it compresses features rather than only grouping heads.
• DeepSeek MLA differs from MQA because it preserves a richer learned representation.
• DeepSeek MLA relies on implementation details such as cache layout and kernel efficiency.
• DeepSeek MLA helps explain why DeepSeek-V2 and DeepSeek-V3 focus on efficient inference.
• DeepSeek MLA is best understood together with the official paper and implementation.
• DeepSeek MLA can reduce memory pressure without eliminating attention computation.
• DeepSeek MLA does not guarantee the same speedup on every GPU or inference engine.
• DeepSeek MLA becomes more valuable as context windows and concurrent batches grow.
• DeepSeek MLA should be compared against MHA, GQA and MQA using the same dimensions.
• DeepSeek MLA is a model-architecture choice, not a simple post-training trick.
• DeepSeek MLA uses a compressed latent to reduce what must be retained per token.
• DeepSeek MLA still needs careful handling of RoPE and value projection paths.
• DeepSeek MLA is easier to reason about when you separate cache memory from compute cost.
• DeepSeek MLA gives engineers a concrete way to discuss long-context serving trade-offs.
• DeepSeek MLA is the focus keyword for this comparison because it connects the math to a real implementation.

In short, DeepSeek MLA matters because DeepSeek MLA turns attention design into a practical cache-design problem. When you compare DeepSeek MLA with MHA, GQA and MQA, the most useful question is what each method stores for every previous token.

There is a slightly strange thing about running a large language model.

The model weights get most of the attention, but during long-context generation another tensor can quietly consume an enormous amount of GPU memory: the KV cache.

Every time an autoregressive transformer generates another token, it needs information from the tokens that came before it. Recalculating all previous keys and values would be expensive, so inference engines retain them.

That works extremely well—until the context becomes long or the serving batch becomes large.

At that point, attention design becomes a memory-management problem.

Four architectures approach this problem differently:

• Multi-Head Attention (MHA)
• Multi-Query Attention (MQA)
• Grouped-Query Attention (GQA)
• Multi-Head Latent Attention (MLA)

MHA stores many independent key/value representations.

MQA shares one.

GQA shares several.

DeepSeek MLA does something fundamentally different: it compresses key/value information into a learned low-dimensional latent representation.

That distinction explains why DeepSeek adopted MLA for DeepSeek-V2 and continued using it in DeepSeek-V3. DeepSeek reports that DeepSeek-V2 reduced its KV cache by 93.3% relative to DeepSeek 67B while increasing maximum generation throughput by up to 5.76× in its reported comparison. Those are model-specific results—not a universal 93.3% promise for every implementation.

Let’s work through exactly where those savings come from.

---

DeepSeek KV cache basics: what is inside the cache?

A transformer attention layer begins with a hidden representation:$$h_t$$

for token $t$.

Standard attention computes:$$q_t=W_Qh_t$$$$k_t=W_Kh_t$$$$v_t=W_Vh_t$$

Queries are needed for the token currently being processed.

Keys and values from previous tokens are different. They will be reused repeatedly as subsequent tokens are generated.

That’s why inference systems cache them.

If a model has:

• $n_h$ attention heads
• key dimension $d_k$
• value dimension $d_v$

then ordinary MHA stores approximately:$$n_h(d_k+d_v)$$

numbers for every token in every transformer layer.

When:$$d_k=d_v=d_h$$

this simplifies to:$$2n_hd_h$$

values.

That doesn’t sound large until you multiply it by:$$\text{tokens}\times\text{layers}\times\text{batch size}$$

and finally by the bytes required by each value.

---

A concrete MHA example

Consider an illustrative transformer with:

Layers:             32
Attention heads:    32
Head dimension:     128
Context:            8,192 tokens
Cache precision:    FP16/BF16
Bytes per value:    2

For each token and layer:$$2\times32\times128=8192$$

values must be stored.

For the entire sequence:$$8192\times8192\times32\times2$$

bytes.

That works out to:$$4,294,967,296\text{ bytes}$$

or exactly:$$4\text{ GiB}$$

for one sequence.

And that’s only the KV cache.

It does not include:

• model weights
• temporary activations
• CUDA workspaces
• allocator overhead
• inference-engine memory pools
• other requests being served simultaneously

That is why reducing KV-cache size matters so much.

---

MHA: one K and V pair for every head

Multi-Head Attention gives every attention head an independent query, key and value representation.

Conceptually:

Query Head 1 ─── Key 1 / Value 1
Query Head 2 ─── Key 2 / Value 2
Query Head 3 ─── Key 3 / Value 3
...
Query Head H ─── Key H / Value H

The cache cost is approximately:$$C_{MHA}=2n_hd_h$$

per token per layer.

Its advantage is expressive flexibility: every head maintains separate K/V projections.

Its disadvantage is obvious from the formula.

Double the number of heads and the cached representation grows with it.

---

MQA: share one K/V head

Multi-Query Attention asks a straightforward question:

Do we really need a separate key and value representation for every query head?

MQA keeps multiple query heads but shares a single key and value representation.

Conceptually:

Q1 ─┐
Q2 ─┤
Q3 ─┼──► shared K/V
Q4 ─┤
... │
QH ─┘

Now cache storage becomes:$$C_{MQA}=2d_h$$

instead of:$$2n_hd_h$$

For our example with 32 heads and a 128-dimensional head:

MHA

$$2\times32\times128=8192$$

values.

MQA

$$2\times128=256$$

values.

That’s a huge reduction.

The trade-off is that all query heads now consume the same K/V representation.

---

GQA: the compromise between MHA and MQA

Grouped-Query Attention sits between those two extremes.

Instead of one KV head or one KV head per query head, groups of query heads share K/V representations.

Suppose 32 query heads use eight KV groups:

Q1  ─┐
Q2  ─┼── KV Group 1
Q3  ─┤
Q4  ─┘

Q5  ─┐
Q6  ─┼── KV Group 2
...

             ...

Q29 ─┐
Q30 ─┼── KV Group 8
Q31 ─┤
Q32 ─┘

Cache storage becomes:$$C_{GQA}=2n_gd_h$$

where $n_g$ is the number of KV groups.

With:$$n_g=8$$

and:$$d_h=128$$

we get:$$2\times8\times128=2048$$

values per token per layer.

So far we have:

Architecture	Cached values/token/layer
MHA	8,192
GQA-8	2,048
MQA	256

All three approaches reduce memory along essentially the same axis:

the number of separately stored KV heads.

MLA changes the game.

---

MLA reduces a different dimension

Multi-Head Latent Attention does not primarily ask:

How many KV heads should we keep?

It asks:

Why are we storing these large expanded representations at all?

MLA first projects information into a smaller latent:$$c_t^{KV}=W^{DKV}h_t$$

where:$$c_t^{KV}\in\mathbb{R}^{d_c}$$

and $d_c$ is much smaller than the full collection of per-head K/V representations.

Keys and values can conceptually be produced from it:$$k_t^C=W^{UK}c_t^{KV}$$$$v_t^C=W^{UV}c_t^{KV}$$

Instead of caching the fully expanded K/V tensors, the model retains the smaller latent representation.

This is the core idea behind DeepSeek MLA as introduced in DeepSeek-V2.

---

The key distinction: head sharing vs feature compression

This is the mental model I find most useful.

MQA and GQA

Reduce:$$\textbf{how many K/V heads are stored}$$

MLA

Reduces:$$\textbf{how much information is stored for them}$$

You can visualize the difference like this:

MHA

K1 K2 K3 K4 ... KH
V1 V2 V3 V4 ... VH

versus:

GQA

KV1   KV2   KV3   KV4
 ↑     ↑     ↑     ↑
groups of query heads

versus:

MLA

       compact latent
             │
             ▼
           c_KV
          /    \
      key info value info

DeepSeek MLA therefore compresses along the feature dimension, rather than primarily reducing the number of heads.

---

DeepSeek MLA dimensions from the official implementation

We can go beyond toy diagrams and inspect published DeepSeek configuration values.

A published DeepSeek-V2 Chat configuration contains:

hidden_size            = 5120
num_attention_heads    = 128

kv_lora_rank           = 512

q_lora_rank            = 1536

qk_nope_head_dim       = 128
qk_rope_head_dim       = 64

v_head_dim             = 128

The model configuration therefore uses a 512-dimensional KV latent, along with a 64-dimensional RoPE component.

DeepSeek-V3’s reference implementation also uses:

kv_lora_rank = 512
qk_nope_head_dim = 128
qk_rope_head_dim = 64
v_head_dim = 128

and its optimized attention path allocates separate compressed kv_cache and pe_cache buffers.

That implementation detail matters.

DeepSeek MLA does not simply cache a single 512-dimensional tensor and call it finished.

There is another issue to solve: positional information.

---

Why RoPE complicates MLA

Modern LLMs commonly use Rotary Position Embedding, or RoPE.

RoPE injects token-position information by applying rotations to query and key components.

The difficulty is that MLA wants to algebraically reorganize the key projection.

If:$$k=W^{UK}c^{KV}$$

then:$$q^\top k = q^\top W^{UK}c^{KV}$$

can be regrouped:$$q^\top W^{UK}c^{KV} = (W^{UK^\top}q)^\top c^{KV}$$

This matters enormously.

It means you don’t necessarily have to reconstruct every expanded key before computing attention.

But once a position-dependent RoPE transformation sits between those operations, that simple rearrangement no longer works in the same way.

DeepSeek therefore separates positional and non-positional components.

---

Decoupled RoPE

The query can be viewed as having two pieces:$$q=[q^C;q^R]$$

and the key similarly:$$k=[k^C;k^R]$$

where:

• $C$ is the content component
• $R$ is the RoPE component

The content information remains associated with the compressed KV latent.

The smaller positional component receives RoPE separately.

This means an optimized MLA cache contains roughly:$$d_c+d_h^R$$

values per token per layer.

For DeepSeek-style dimensions:$$512+64=576$$

values.

This is much more accurate than saying:

“MLA stores only 512 values.”

The positional component matters.

---

DeepSeek attention comparison: MLA vs MHA vs GQA vs MQA

Using our earlier illustrative configuration:

32 query heads
head dimension = 128
8 GQA KV groups

MLA latent = 512
MLA RoPE dimension = 64

we get:

MHA

$$2\times32\times128 = 8192$$

GQA

$$2\times8\times128 = 2048$$

MQA

$$2\times128 = 256$$

MLA

$$512+64 = 576$$

So:

Attention	Cache elements/token/layer	Relative to MHA
MHA	8,192	100%
GQA-8	2,048	25%
MQA	256	3.125%
MLA	576	7.03%

Notice something interesting.

MLA is not necessarily the smallest cache.

MQA is smaller in this example.

That means saying:

MLA wins because it always has the smallest cache

would be incorrect.

The motivation is more subtle: MLA seeks heavy memory compression while retaining a richer learned representation.

---

DeepSeek KV-cache memory comparison

Let’s scale the same example to:

32 layers
8,192 tokens
2 bytes per cache value

MHA

$$8192\times8192\times32\times2$$

= 4.00 GiB

GQA-8

$$2048\times8192\times32\times2$$

= 1.00 GiB

MQA

$$256\times8192\times32\times2$$

= 0.125 GiB

MLA

$$576\times8192\times32\times2$$

= approximately 0.281 GiB

So the difference becomes easy to see:

Attention	Approx. KV cache
MHA	4.00 GiB
GQA-8	1.00 GiB
MLA	0.281 GiB
MQA	0.125 GiB

The MLA configuration uses:$$\frac{576}{8192}=0.0703$$

of the MHA cache.

That’s about:$$14.2\times$$

smaller.

Again, this is a dimensional example—not a benchmark of an actual model.

---

Why DeepSeek reconstruction does not erase the MLA benefit

This was the question I had after first understanding the compression step.

If we compress K/V and then expand everything again during every decoding step, have we merely exchanged a memory problem for a computation problem?

This is where weight absorption enters.

Suppose:$$k=W^{UK}c$$

The attention score involves:$$q^\top k$$

Substitute the key:$$q^\top W^{UK}c$$

Because the operations are linear, we can regroup them:$$(W^{UK^\top}q)^\top c$$

Instead of doing:

compressed latent
       ↓
build huge key
       ↓
dot with query

we can conceptually do:

query
  ↓
transform query
  ↓
dot directly with compressed latent

The value projection can likewise be reorganized with the output projection.

That is why the optimized DeepSeek-V3 inference implementation is especially useful to study.

Its naive path allocates ordinary key and value caches.

Its optimized path instead registers:

self.kv_cache
self.pe_cache

with dimensions based on the compressed KV rank and positional dimension.

That is architecture becoming implementation.

---

What I found most important after testing a simplified MLA implementation

Replace this section with your actual Colab output from the script we created earlier.

Do not invent the results.

A genuine version might be structured like this:

I also implemented a simplified version of Multi-Head Latent Attention in PyTorch to make sure I understood where the compression actually occurs.

My test configuration used a 512-dimensional hidden representation, eight attention heads, and a 64-dimensional KV latent.

The model received an input tensor of:

(1, 512, 512)

while its compressed KV representation had shape:

(1, 512, 64)

The forward-pass output returned to:

(1, 512, 512)

and passed the finite-value and output-shape checks.

The most useful part of the experiment wasn’t the latency number. It was seeing the latent representation directly: each token moved from a 512-dimensional hidden state to a 64-dimensional intermediate KV representation before keys and values were reconstructed in the educational implementation.

Then insert a screenshot from your real Colab execution.

That gives this article information the Medium article does not contain: your own reproducible experiment.

---

Memory savings do not automatically equal latency savings

One distinction worth making is:$$\text{memory reduction} \neq \text{same percentage speedup}$$

Reducing KV-cache size can help inference because less memory must be stored and potentially transferred.

But real latency depends on many additional factors:

• GPU architecture
• memory bandwidth
• batch size
• context length
• kernel implementation
• quantization
• cache layout
• tensor parallelism
• scheduling
• FlashAttention/FlashMLA-style kernels
• whether decoding is memory-bound or compute-bound

So an MLA implementation can be mathematically memory-efficient and still perform poorly if the kernel implementation is inefficient.

This is one reason DeepSeek’s reference implementation contains separate naive and optimized attention paths rather than treating MLA as just a different equation.

---

MLA is not the same as LoRA

The naming can cause confusion.

DeepSeek configurations contain fields such as:

kv_lora_rank
q_lora_rank

but MLA is not simply LoRA fine-tuning applied to attention.

LoRA normally introduces low-rank matrices to efficiently adapt model parameters.

MLA uses low-rank projections as part of the attention architecture itself.

A simple distinction is:

LoRA
→ compresses/adapts parameter updates

MLA
→ compresses the representation used by attention
  and changes what needs to be cached

The mathematical tools look related.

The system-level purpose is different.

---

Does MLA make attention linear-time?

No.

This is another important misconception.

MLA reduces the representation that must be stored for previous tokens.

It does not make ordinary full attention suddenly scale as:$$O(n)$$

instead of:$$O(n^2)$$

for full-sequence attention.

It is useful to separate:

Parameter memory

How much memory the model weights consume.

KV-cache memory

How much state must be retained during autoregressive generation.

Attention computation

How much work is required to compute attention interactions.

DeepSeek MLA directly targets the KV-cache problem.

Those three problems are related, but they are not the same.

---

Why long contexts make MLA more valuable

Imagine two workloads.

Workload A

Batch size: 1
Context: 512 tokens

Workload B

Batch size: 32
Context: tens of thousands of tokens

In the first workload, KV-cache memory may be a relatively minor concern.

In the second, it can become a major system constraint.

That’s why MLA becomes especially interesting for:

• long-context assistants
• large concurrent serving batches
• reasoning workloads
• coding assistants with large repositories in context
• document analysis
• agent systems maintaining long histories

As context windows grow, what the model keeps from each past token increasingly matters.

---

What DeepSeek reported

DeepSeek-V2 introduced MLA alongside the DeepSeekMoE architecture.

The DeepSeek-V2 paper reports:

• 128K context support
• 93.3% KV-cache reduction compared with DeepSeek 67B
• up to 5.76× maximum generation throughput in its reported comparison
• 42.5% lower training cost compared with DeepSeek 67B

These numbers belong to DeepSeek’s specific model/system comparison; they should not be presented as universal MLA performance numbers.

DeepSeek-V3 retained MLA. The official DeepSeek-V3 repository describes the model as using MLA together with DeepSeekMoE for efficient inference and cost-effective training.

That continuity is significant.

MLA wasn’t merely an experiment in V2—it became part of DeepSeek’s subsequent architecture.

---

What the actual DeepSeek code teaches us

Looking at source code often clears up abstractions better than another diagram.

The official DeepSeek-V3 MLA class exposes:

q_lora_rank
kv_lora_rank
qk_nope_head_dim
qk_rope_head_dim
v_head_dim

and divides each query into:

q_nope
q_pe

where the latter carries the rotary positional component.

The optimized inference path stores:

kv_cache
pe_cache

instead of conventional full-size per-head K and V caches.

This is an important engineering lesson:

MLA is not simply a smaller tensor.

Its compression, positional encoding and inference algebra are designed together.

---

MHA vs GQA vs MQA vs MLA: which approach is “best”?

There is no universal winner.

The designs optimize different points in the system.

Method	Main strategy	KV memory	Architectural complexity
MHA	Independent K/V per head	High	Low
GQA	Share K/V within groups	Medium	Low–medium
MQA	Share one K/V representation	Very low	Medium
MLA	Compress K/V into learned latent state	Very low	Higher

MHA is conceptually straightforward.

GQA is a practical compromise used in many modern LLMs.

MQA aggressively reduces KV storage.

MLA introduces additional machinery to compress representations while preserving rich multi-head behavior.

Which one makes sense depends on the model architecture, training strategy, inference engine and hardware.

---

A simple KV-cache calculator

You can reproduce the dimensional comparison with a few lines of Python:

def cache_values(
    heads=32,
    head_dim=128,
    gqa_groups=8,
    mla_latent=512,
    mla_rope=64,
):
    mha = 2 * heads * head_dim
    gqa = 2 * gqa_groups * head_dim
    mqa = 2 * head_dim
    mla = mla_latent + mla_rope

    return {
        "MHA": mha,
        "GQA": gqa,
        "MQA": mqa,
        "MLA": mla,
    }


results = cache_values()

for name, values in results.items():
    print(f"{name}: {values:,} values/token/layer")

Output:

Now calculate full cache memory:

def cache_gib(
    values_per_token,
    layers,
    context,
    bytes_per_value=2,
):
    total_bytes = (
        values_per_token
        * layers
        * context
        * bytes_per_value
    )

    return total_bytes / (1024 ** 3)


layers = 32
context = 8192

for name, values in results.items():
    memory = cache_gib(
        values,
        layers,
        context
    )

    print(f"{name}: {memory:.3f} GiB")

That produces approximately:

---

The most useful mental model for MLA

After working through the architecture, this is the shortest explanation I would keep:

MHA stores expanded keys and values. MLA learns a compact representation from which their useful information can be recovered—or, with weight absorption, used without explicitly rebuilding conventional K/V tensors.

And the comparison becomes:

MHA
Store everything separately.

GQA
Share K/V between groups.

MQA
Share one K/V representation.

MLA
Learn what can be compressed and cache the compressed state.

That is the architectural idea underneath the equations.

---

Frequently asked questions

What is the difference between MLA and MHA?

MHA stores separate key and value representations for each attention head. MLA uses low-rank projections to compress key/value information into a significantly smaller latent representation.

Is MLA better than GQA?

They use different compression strategies. GQA reduces the number of KV heads, while MLA compresses KV information along the feature dimension. The appropriate choice depends on model design and implementation.

Does DeepSeek use MLA?

Yes. DeepSeek introduced MLA with DeepSeek-V2 and retained the architecture in DeepSeek-V3.

How much does MLA reduce KV cache?

There is no universal percentage because it depends on model dimensions. DeepSeek reported a 93.3% reduction relative to DeepSeek 67B for DeepSeek-V2.

What is kv_lora_rank in DeepSeek?

It controls the rank/dimension of the compressed KV latent representation. Published DeepSeek-V2/V3 configurations use a value of 512.

Why does MLA use decoupled RoPE?

RoPE is position-dependent and interferes with the matrix-absorption optimization. MLA therefore separates positional information into a smaller RoPE-specific component.

Is MLA the same as MQA?

No. MQA shares one KV head between query heads. MLA instead learns a compressed latent representation.

Does MLA eliminate the KV cache?

No. It compresses the cache. The optimized DeepSeek implementation still caches compressed KV state and positional state.

Does MLA eliminate quadratic attention?

No. KV-cache compression and full-attention computational complexity are separate issues.

---

Final takeaway

MHA, GQA, MQA and MLA are all responses to the same underlying problem: keeping attention useful without allowing the KV cache to dominate inference memory.

But they take different routes.

MHA preserves independent keys and values for every head.

GQA shares them between groups.

MQA reduces them to one shared representation.

DeepSeek MLA instead introduces a learned bottleneck that asks a different question:

How little information do we actually need to store in order to preserve useful multi-head attention behavior?

DeepSeek’s answer is a compact KV latent combined with a small decoupled positional component and an inference-time algebraic trick that allows the model to operate efficiently on that compressed state.

That makes MLA more than another attention variant.

It is an example of co-design between model architecture and inference infrastructure—changing the representation inside the network specifically because of what becomes expensive when the model is deployed.

And as LLM context windows grow, that kind of design is likely to become increasingly important.

---

Sources and related GenAITrail reading

Use primary sources for DeepSeek architecture details rather than only linking to blogs:

Related GenAITrail reading:
Multi-Head Latent Attention explained with DeepSeek architecture and PyTorch code
Running local LLMs on a single GPU

DeepSeek-V2 paper:
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

Official DeepSeek-V3 implementation:
DeepSeek-V3 MLA implementation on GitHub

DeepSeek-V3 repository:
Official DeepSeek-V3 repository