ReCAT: Remember, Count, And Time

Structured Recurrent Memory for Robot Manipulation

Pankhuri Vanjani^1*, Mostafa Hatab^1*, Can Mizrakli^1*, Vaisakh Shaj², Zhuoyue Li¹,
Moritz Reuss³, Rudolf Lioutikov^1,4

¹Intuitive Robots Lab (IRL), Karlsruhe Institute of Technology (KIT), ²University of Edinburgh,
³NVIDIA, ⁴Robotics Institute Germany (RIG)
^*Core contributors

Paper arXiv Videos Code (soon)

66.7%

Real-robot memory tasks

vs. 8.3% for the strongest short-history baseline

62.4%

RMBench

best or tied-best on 6 of 9 tasks

95.3%

LIBERO

at 59 ms per policy step (16.9 Hz)

Same view, different action

Problem

Memory-dependent manipulation requires robots to make decisions using information that is no longer available to their current sensors: recalling an earlier visual cue, tracking task progress, counting repeated events, or estimating elapsed time.

After one scoop and after two, the robot sees the same scene, yet it must either scoop again or put the shovel back. Only memory of what already happened can decide.

Method

We present ReCAT, a language-conditioned policy with structured recurrent memory. An instruction-conditioned encoder forms features from the current observation. A recurrent memory integrates the observation stream through Mamba-2 layers and one causal attention layer. A flow-matching Transformer decoder reads the current and the historical representation through separate cross-attention in every block.

Observation encoder

Compresses images, instruction and proprioception into one feature z_t per step.

Recurrent memory

Mamba-2 layers plus one causal attention layer carry a compact state m_t through the episode.

Flow-matching decoder

Reads z_t and m_t through separate cross-attention in every block.

Step-by-step walkthrough of the architecture.

Interchangeable update rules

Mamba-2 (default)

diag(a_t) h_t−1 + v_tk_t^⊤

Accumulate: repeated events build up in the state.

Mamba-3

diag(a_te^iθ_t) h_t−1 + v_tk_t^⊤

Accumulate + rotate: adds a phase that can track time.

Gated DeltaNet-2

α_t(I − β_tk_tk_t^⊤) h_t−1 + β_tv_tk_t^⊤

Replace: a repeated cue overwrites its old content.

Three real-robot memory tasks

Franka Emika Panda (DROID setup), 20 rollouts per task. Each video compares GMP, X-VLA-H and ReCAT.

Count Add exactly two scoops of soil, put the shovel down, then plant the flower. 45 demos · ~1,200 steps

Plant: frame sequence with the memory-dependent decision point outlined.

Time Put the pot on the stove, wait ~30 s while the scene stays still, then add the pepper. 45 demos · ~980 steps

Pot Timer: frame sequence with the memory-dependent decision point outlined.

Remember Move the sponge to the plate, then return it to its unmarked start position. 48 demos · ~480 steps

Sponge: frame sequence with the memory-dependent decision point outlined.

Baselines fail exactly where memory begins

Result

ReCAT reaches 95.3% average success on LIBERO and 62.4% on RMBench, with the best or tied-best result on six of nine tasks. On three real-robot tasks probing spatial recall, event counting and interval timing, the best ReCAT variant reaches 66.7% average success, against 8.3% for the strongest short-history baseline. Controlled comparisons show that the observation encoder and every-block memory conditioning are needed for this performance, and that update rules behave differently as robot memory: additive updates have the highest observed success on counting and timing, and delta-rule updates on spatial recall.

Policies without persistent memory succeed in 0–8.3% of rollouts. ReCAT variants reach 50.0–66.7%. The 95% Wilson intervals for X-VLA + history ([3.6%, 18.1%]) and ReCAT-Mamba-2 ([54.1%, 77.3%]) do not overlap. Stage-wise columns show where each policy fails: the baselines usually finish the early manipulation and then fail at the decision that needs memory.

Method	Plant				Pot Timer						Sponge			Avg.
Method	1 scoop	2 scoops	3 or more	Flower planted Success	Pot on stove	No wait	Wrong wait	≈30 s wait	Pepper in the pot	Success	Pick sponge, put on plate	Returned to wrong position	Returned to same position Success	Avg.
Baselines
Diffusion Policy	–	–	–	0	–	–	–	–	–	0	–	–	0	0.0
Diffusion Policy + history	15	10	0	0	100	25	0	0	25	0	15	–	0	0.0
X-VLA	–	–	–	0	–	–	–	–	–	0	–	–	0	0.0
X-VLA + history	0	10	85	10	100	0	35	5	40	5	75	65	10	8.3
Gated Memory Policy (GMP)	65	20	10	15	95	20	10	5	35	5	50	50	0	6.7
Ours
ReCAT-Mamba-2	0	100	0	85	100	0	0	100	95	95	20	0	20	66.7
ReCAT-Mamba-3	0	95	0	90	100	0	0	100	100	100	0	0	0	63.3
ReCAT-Gated DeltaNet-2	0	70	0	20	100	0	0	100	90	90	40	0	40	50.0

All values are percentages of 20 rollouts with randomized start positions. Red columns are failure modes, green columns are progress stages, and the darkest green is final success. Plant scoop columns are mutually exclusive. “No wait” and “Wrong wait” left the waiting state early or outside the 29–31 s window. Avg. is the mean success over the three tasks.

Simulation benchmarks

LIBERO

LIBERO is mostly Markovian, so it tests whether full-history recurrence keeps performance when the current observation is enough.

Method	Spatial	Object	Goal	LIBERO-10	Avg.	LIBERO-Plus
Diffusion Policy	78.3	92.5	68.3	50.5	72.4	–
X-VLA (0.9B, pretrained)	98.2	98.6	97.8	97.6	98.1	71.4
ReCAT	96.4	97.2	97.2	90.4	95.3	64.2

RMBench

Nine long-horizon bimanual tasks. In M(n) tasks, actions depend on several past events that are no longer observable. ReCAT has the best or tied-best score on six tasks, including three of the four M(n) tasks.

Task	Type	DP	ACT	π0.5*	X-VLA*	MEM-0	EventVLA*	MemoryVLA*	MemER	ReCAT
		90M	80M	3B	0.9B	10B	4B	7.3B	10B	374M
Observe and Pick Up	M(1)	1	1	9	9	4	21	2	7	4
Rearrange Blocks	M(1)	0	29	13	13	89	96	53	17	96
Put Back Block	M(1)	0	0	11	18	90	95	81	0	100
Swap Blocks	M(1)	11	2	24	16	67	96	76	14	100
Swap T	M(1)	20	2	15	3	14	87	9	7	64
Average M(1)		6.4	6.8	14.4	11.8	52.8	79.0	44.2	9.0	72.8
Battery Try	M(n)	10	19	16	26	28	35	33	27	43
Blocks Ranking Try	M(n)	10	0	6	1	18	81	53	12	99
Cover Blocks	M(n)	0	0	0	2	68	97	69	40	50
Press Button	M(n)	0	0	0	0	0	3	0	0	6
Average M(n)		5.0	4.8	5.5	7.3	28.5	54.0	38.8	19.8	49.5
Overall average		5.8	5.9	10.4	9.8	42.0	67.9	41.8	13.8	62.4

* Uses large-scale pretraining. EventVLA has a higher overall average; it builds on a 4B backbone, about ten times larger than ReCAT, with an extra pretraining phase.

Efficiency

All policies measured on the robot's inference GPU (RTX 4060 Ti) with the same settings. Higher SPARC means smoother trajectories.

Metric	ReCAT	DP-H	X-VLA-H	GMP
Total / trainable params	374M / 142M	267.9M / 267.9M	881.9M / 881.9M	282.2M / 188.7M
Peak inference VRAM	1.44 GB	0.65 GB	2.55 GB	1.09 GB
Inference latency	59 ms (16.9 Hz)	110 ms (9 Hz)	288 ms (3.5 Hz)	93 ms (10.8 Hz)
Smoothness (SPARC)	−5.16	−6.19	−6.76	−13.06

Failure modes and limitations

Most remaining ReCAT failures happen before memory is needed. In Sponge, the start positions vary widely, and failed rollouts usually miss the grasp or placement; whenever the sponge reaches the plate, ReCAT returns it to the right place. In Plant, the failures come from inserting the flower stem.

Every decoder block must read the memory

Memory → decoder

Avg. real-robot success (%)

Cross-attn every block

66.7

Late fusion

38.3

AdaLN

31.7

Scale

28.3

Dropout

Gate

8.3

Encoder & components

Avg. real-robot success (%)

Hybrid Mamba–attn

66.7

Full Transformer

No multimodal enc.

No memory

State width (Plant)

Plant success (%)

d_state 32

d_state 64

d_state 128

d_state 256

Every-block memory cross-attention and the hybrid encoder matter most; removing memory drops all tasks to 0%.

BibTeX

@article{vanjani2026recat,
  title         = {ReCAT: Remember, Count, And Time: Structured Recurrent Memory for Robot Manipulation},
  author        = {Vanjani, Pankhuri and Hatab, Mostafa and Mizrakli, Can and Shaj, Vaisakh and
                   Li, Zhuoyue and Reuss, Moritz and Lioutikov, Rudolf},
  journal       = {arXiv preprint arXiv:2609.35200},
  year          = {2026},
  eprint        = {2609.35200},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO}
}