A Visual Guide to Real-Time Action Chunking in Robotics

Prerequisites

We first review several ideas that appear throughout the paper.

Action Chunking

I first learned about action chunking from Tony Z. Zhao et al.’s paper, 📄Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware.

In a classic MDP, an agent outputs one action $\mathbf a_t$ from the current state $\mathbf{s}_t$ . In a POMDP, it usually acts from observation $\mathbf o_t$ instead.

An action-chunking policy

$\pi(\mathbf{A}_t \mid \mathbf{o}_t)$

outputs a sequence of future actions

$\mathbf{A}_t=[\mathbf{a}_{t}, \mathbf{a}_{t+1},\dots,\mathbf{a}_{t+H-1}]$

rather than only one action $\mathbf{a}_t$ .

Tip

This is the dominant approach by 2026-07-08. Since the inference costs quite a lot of time. Therefore, outputs a chunk of action saves the GPU inference time.

We call the $H$ as the prediction chunk and $K$ as the execution chunk, meaning the number of actions sent to the controller. Usually, the execution chunk is often shorter, i.e., $K \leq H$ .

Remark

An empirical ratio is $K\approx H/2$ which is used in 📄Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, pi0, and pi05.

Warning

The original paper uses $s$ for the execution horizon. But I prefer $K$ to avoid confusion to the state $\mathbf{s}_t$ .

Flow Matching

Many continuous-action robot policies use Flow Matching to generate action chunks.

To generate a chunk, we first sample Gaussian noise

\mathbf A_t^0\sim\mathcal N(\mathbf 0,\mathbf I).

A useful mental picture is:

A trained neural network $\mathbf{v}^\pi$ predicts a velocity field.

Its inputs are

the current noisy or partially denoised chunk $\mathbf A_t^\tau$ ,
the observation $\mathbf o_t$ , and
the flow time $\tau$ , Its output is a velocity field which has the same shape as the action chunk (make it flow🍃).

With $n$ Euler steps, one update is

\mathbf A_t^{\tau+\frac1n} = \mathbf A_t^\tau + \frac1n\mathbf v_\pi(\mathbf A_t^\tau,\mathbf o_t,\tau).

The $\tau\in[0,1)$ is the flow-matching time, and $n$ is the number of integration steps.

Intuitively, the learned velocity field moves the initial random noise toward a meaningful action chunk.

Remark

For a more detailed introduction, see A Beginner’s Guide to Flow Matching.

Tip

Robot policies often use a small number of flow steps $n$ , such as 4, 5, or 8. The exact value depends on the model.

What does real-time mean?

As you might notice, there is a word “real-time” in the title. How to define it then🤔?

Let

$\Delta t$ be the sampling period of the learned policy or controller, and
$\delta$ be the time required to generate an action chunk.

Then a system is real-time only if it can generate an action chunk ( $\mathbf{A}_t$ ) with respect to the input (received $\mathbf{o}_t$ ) within the fixed time constraint ( $\Delta t$ ), i.e., $\delta \leq \Delta t$ .

Example

Suppose the learned policy runs at 50 Hz, so $\Delta t=20\text{ ms}.$ If my measured OpenPI inference time is about $\delta = 60\text{ms},$ then the model cannot finish within one policy period. It is not real-time🚫 since 60ms > 20ms.
By contrast, the control rate of Franka Research 3 robot is 1KHz and therefore the $\Delta t = 1\text{ms}$ . The ruckig: Motion Generation for Robots and Machines can manage to generate action chunks around 250 $\mu s$ .
Therefore, we know that real-time is is possible✅ in ruckig since 250µs < 1ms.

Remark

The frequency of the learned policy could be considered as the dataset action sampling rate during teleoperation. In practice, the robot might run in low-level controller with 1KHz but the sampled dataset could be 50Hz or even smaller 20Hz.
Because these are different layers of the control stack, comparisons should be interpreted carefully‼️.

Synchronous Inference

This is the simplest way to deploy an action-chunking policy on a robot. The robot executes the current chunk, reaches the end of its execution horizon, starts the next inference call, and waits for the new chunk to finish.

When $\delta > \Delta t$ , this creates visible pauses between chunks. The pauses have 2 main effects:

The robot takes longer to complete the task.
The robot dynamics during evaluation differ from the dynamics in the training data.

Remark

Synchronous inference is the a common baseline for policies such as π0, π0.5, 📄OpenVLA: An Open-Source Vision-Language-Action Model, 📄Octo: An Open-Source Generalist Robot Policy, 📄RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control, etc.

The second effect is especially problematic. Consider the following comparison:

Training	Evaluation

Suppose a demonstration contains a smooth motion.

During training, the joint velocity $\dot{q}$ changes naturally along the trajectory.
During synchronous evaluation, however, the robot may be forced to hold its position while waiting for the next chunk, i.e., $\dot{q}\neq 0$ .

Therefore,

$\dot{q}_{\text{train}} \not\approx \dot{q}_{\text{eval}},$

and a distribution shift is introduced👎️. The policy may never have seen this stop-and-restart behavior in its training data.

Asynchronous Inference

Synchronous inference wastes time by making the robot wait. A natural next step is asynchronous inference: generate the next chunk in the background while the current chunk is still being executed.

✒Notation 1: Define the inference delay $d$ as the number of controller timesteps between when $\mathbf{o}_t$ is received and when $\mathbf{A}_t$ is available:

d := \lfloor \delta / \Delta t \rfloor,

where $\lfloor x \rfloor$ denotes floor function.

✒Notation 2: Let $a_{t'\mid t}$ denote the action for world time $t'$ inside the chunk $\mathbf{A}_t$ generated from observation $\mathbf{o}_t$ . It’s position inside that chunk is $(t' - t)$ .

When Should the Next Inference Start?

Suppose $\mathbf{A}_0$ is currently executing and we want to switch chunks after $K$ controller steps. Since inference takes $d$ steps, the next inference call must start at

$K-d.$

Then the new chunk is ready at the desired switch time $K$ which ensures real-time constraint.

For a repeating asynchronous schedule, the useful feasibility condition is

\boxed{d\leq K\leq H-d.}

$d \leq K$ ensures that inference does not need to start before the current cycle begin.
$K \leq H-d$ ensures the previous chunk contains enough future actions to cover the delay while the next chunk is being generated.

Tip

😮‍💨Sounds confusing? No worries! The example will clear them out!

Warning

However, asynchronous execution alone does not guarantee a smooth transition. While generating $\mathbf{A}_{K-d}$ , the policy does not know exactly what will happen during the next $d$ controller steps. Therefore, the transition $a_{K-1 | 0} \longrightarrow a_{K | K-d}$ may be discontinuous and out of distribution.
See more in the coming section.

Alright, enough formulas, let’s use a concrete example. Suppose:

\Delta t = 20\text{ ms},\qquad \delta = 80\text{ ms},\qquad d = \delta/\Delta t = 4,

and choose:

H=10,\qquad K=6.

We can digest everything with this diagram.

1️⃣ The First Chunk

The first chunk is

\mathbf A_0= [\mathbf a_{0\mid0},\mathbf a_{1\mid0},\ldots,\mathbf a_{9\mid0}].

It is conditioned on the observation at time $t=0$ . For example, $\mathbf a_{4\mid0}$ is the action intended for world time $4$ , predicted from $\mathbf o_0$ .

2️⃣ When should asynchronous inference start?

Since the inference takes $d=4$ controller steps and we want to switch at $K=6$ , therefore we need to start the next inference call at

K-d=6-4=2.

After the controller has executed $\mathbf a_{1\mid0}$ and received $\mathbf o_2$ , it starts generating

\mathbf A_2= [\mathbf a_{2\mid2},\mathbf a_{3\mid2},\ldots,\mathbf a_{11\mid2}].

The new chunk becomes available at world time $t=6$ .

3️⃣ Which actions are not used?

There are 2 groups of unused actions which are marked in yellow.

From $\mathbf A_0$ , the actions

[\mathbf a_{6\mid0},\mathbf a_{7\mid0},\mathbf a_{8\mid0},\mathbf a_{9\mid0}]

are not executed because we switch after $K=6$ actions.

From $\mathbf A_2$ , the actions

[\mathbf a_{2\mid2},\mathbf a_{3\mid2},\mathbf a_{4\mid2},\mathbf a_{5\mid2}]

arrive too late. By the time $\mathbf A_2$ is available, world time has already reached $t=6$ .

4️⃣ Which actions does the controller execute?

From the global world-time viewpoint, the controller executes

\mathbf a_{0\mid0}, \mathbf a_{1\mid0}, \ldots, \mathbf a_{5\mid0}, \mathbf a_{6\mid2}, \mathbf a_{7\mid2}, \ldots

Notice that $\mathbf a_{6\mid2}$ is the fifth entry of $\mathbf A_2$ when using zero-based indexing.

5️⃣ Why can this sequence be problematic?

The transition

\mathbf a_{5\mid0} \longrightarrow \mathbf a_{6\mid2}

might be discontinuous. The two chunks were generated from different observations and may represent different valid strategies.

The author provides a nice diagram to illustrate this:

Sometimes, it is coincident that the policy generates action chunks diverged at the inference timestep😮‍💨. In the diagram, the old chunk plans to move above the obstacle, while the new chunk plans to move below it.

Both chunks may be valid on their own, but switching between them in the middle of the motion can create a large jump in velocity or acceleration. This mismatch leads to jerky and unpredictable behavior⚠️.

The Big Picture of Real-time Chunking

Real-time chunking aims to keep the benefits of asynchronous inference while removing discontinuities at chunk boundaries.

No formulas yet. Just take a look at the big picture:

inference-time-realtime-action-chunking-big-picture.svg

The transition $\mathbf A_0\rightarrow\mathbf A_2$ is a useful startup example. After that, inference calls start every $K=6$ steps. The call after $\mathbf A_2$ therefore starts at $t=8$ and generates $\mathbf A_8$ .

📌Gap

Naive asynchronous inference may cause a discontinuous transition at the chunk boundary, e.g. $\mathbf{a}_{11|2}$ to $\mathbf{a}_{17|8}$ as indicated with red arrow.

📌Objectives

The objectives for the inference-time real-time action chunking algorithm are twofold:

Give the first $d$ overlapping entries full guidance as indicated with blue arrows.
Leave the non-overlapping tail generated freely.

That’s it! That is the big picture and we will explain piece by piece in the remaining sections.

Real-Time Chunking as an Inpainting Problem

The novelty of this paper is to conceive of chunk generation as inpainting. Here’s the mindmap:

For the steady-state RTC example, let’s define the current chunk as

\mathbf{C}=[C_0,C_1,\ldots,C_9],

and the next chunk as

\mathbf{N}=[N_0,N_1,\ldots,N_9].

Intuitively, the first $d$ -steps in the $\mathbf{N}$ chunk can be thought of the white region in the diagram which is expected to consistent with its surrounding.

A Review of Ordinary Flow-Matching Generation

Continue to use the $H=10$ as a simple example. Let

\mathbf{A}^\tau=[A^\tau_0,\ldots,A^\tau_9]

be the partially denoised action chunk.

Here:

$\tau=0$ : Gaussian noise;
$\tau=1$ : the final action chunk.

A good mindmap is the following. The $\tau$ could be at anytime in the following diagram, i.e. not finished its denoising procedure.

Warning

The symbol $\tau$ is the flow-generation time. It is not the controller time $t$ used to execute actions on the robot.

The policy predicts a vector field

\mathbf v_\pi(\mathbf A^\tau,\mathbf o,\tau).

Remark

During inference, the learned parameters of $\mathbf v_\pi$ are fixed. We only update the sampled action chunk.

One Euler step is

\mathbf{A}^{\tau+\Delta\tau} = \mathbf{A}^\tau+\Delta\tau\,\mathbf{v}_\pi(\mathbf{A}^\tau,\mathbf{o},\tau), \qquad \Delta\tau=\frac1n.

The Target

Recall that our objective

inference-time-realtime-action-chunking-big-picture.svg

is to make

N_0\approx C_6,\quad N_1\approx C_7,\quad N_2\approx C_8,\quad N_3\approx C_9.

This is exactly what the target variable $\mathbf{Y}$ represents in the paper! The current chunk started $K$ steps before the next chunk, meaning they overlap for exactly $H-K$ steps.

Y_i= \begin{cases} C_{K+i}, & 0\leq i<H-K,\\[4pt] 0, & H-K\leq i<H. \end{cases}

For $H=10$ and $K=6$ , the target looks like this:

\mathbf{Y}= [C_6,C_7,C_8,C_9,0,0,0,0,0,0].

Great! We have our target $\mathbf{Y}$ . Now we just need a mask $\mathbf{W}$ to decide how strongly we enforce this target at each timestep:

$1$ : strongly constrain this timestep;
$0$ : let the model generate freely

Estimating the Final Denoised Chunk

To guide our generation toward the target $\mathbf{Y}$ , we need to calculate an “error.”

Here is the problem: $\mathbf{A}^\tau$ is just a noisy intermediate chunk. We want our final chunk $\mathbf{A}^1$ to match $\mathbf{Y}$ . But while we are at step $\tau$ , we don’t know what $\mathbf{A}^1$ will actually look like yet🫣!

Therefore, we must mathematically approximate a value $\widehat{\mathbf{A}}^1 \approx \mathbf{A}^1$ .

We start from the underlying flow ODE:

\begin{align} \frac{d\mathbf{A}^u}{du} &= \mathbf{v}_\pi (\mathbf{A}^u, 0, u) \\ d\mathbf{A}^u &= \mathbf{v}_\pi(\mathbf{A}^u, \mathbf{o}, u) \,du. \end{align}

Integrating this from the current flow time $\tau$ up to the final flow time $1$ :

\int_\tau^1 d\mathbf{A}^u = \int_\tau^1 \mathbf{v}_\pi(\mathbf{A}^u, \mathbf{o}, u) \,du

Apply the second fundamental theorem of calculus:

\begin{align} \mathbf{A}^1 - \mathbf{A}^\tau &= \int_\tau^1 \mathbf{v}_\pi(\mathbf{A}^u, \mathbf{o}, u) \,du \\ \mathbf{A}^1 &= \mathbf{A}^\tau + \int_\tau^1 \mathbf{v}_\pi(\mathbf{A}^u, \mathbf{o}, u) \,du \\ \end{align}

To approximate this integral without running the full neural network, we “freeze” the velocity at time $\tau$ :

\mathbf{v}_\pi(\mathbf{A}^u, \mathbf{o}, u) \approx \mathbf{v}_\pi(\mathbf{A}^\tau, \mathbf{o}, \tau) \qquad \text{for } u \in [\tau, 1]

Remark

Notice that $\tau$ is our current, fixed denoising timestep, while $u$ is the dummy integration variable moving toward $1$ .

Substituting this approximation, we get:

\begin{align} \mathbf{A}^1 &= \mathbf{A}^\tau + \int_\tau^1 \mathbf{v}_\pi(\mathbf{A}^u, \mathbf{o}, \textcolor{blue}{u}) \,du\\ &\approx \mathbf{A}^\tau + \int_\tau^1 \underbrace{\mathbf{v}_\pi(\mathbf{A}^\tau, \mathbf{o}, \textcolor{red}{\tau})}_{\text{constant w.r.t. } u} \,du \end{align}

Because the integrand no longer depends on $u$ , we can evaluate it directly:

\begin{aligned} \int_\tau^1 \mathbf{v}_\pi(\mathbf{A}^\tau, \mathbf{o}, \tau) \,du &= \mathbf{v}_\pi(\mathbf{A}^\tau, \mathbf{o}, \tau) \int_\tau^1 du \\ &= \mathbf{v}_\pi(\mathbf{A}^\tau, \mathbf{o}, \tau) [u]_\tau^1 \\ &= (1-\tau)\mathbf{v}_\pi(\mathbf{A}^\tau, \mathbf{o}, \tau) \end{aligned}

Therefore, we have

\boxed{\widehat{\mathbf{A}}^1 = \mathbf{A}^\tau + (1-\tau)\mathbf{v}_\pi(\mathbf{A}^\tau, \mathbf{o}, \tau)}

Remark

This is mathematically identical to taking one giant forward Euler step to jump straight to the end of the ODE.

Example

For example, if $\tau=0.6$ and the current predicted velocity is $\mathbf{v}_\pi(\mathbf{A}^{0.6}, \mathbf{o}, 0.6) = \begin{bmatrix} 2\\-1 \end{bmatrix}$ then the remaining time is $1-\tau=0.4.$ so $\widehat{\mathbf{A}}^1 = \mathbf{A}^{0.6} + 0.4 \begin{bmatrix} 2\\-1 \end{bmatrix} = \mathbf{A}^{0.6} + \begin{bmatrix} 0.8\\-0.4 \end{bmatrix}$

Formulating the Guide

Now that we have our target $\mathbf{Y}$ and our estimated final chunk $\widehat{\mathbf{A}}^1$ , we can create a guidance signal $\mathbf{g}$ .

Let $\mathbf{J}$ denotes the Jacobian matrix:

\mathbf{J} = \frac{\partial \widehat{\mathbf{A}}^1}{\partial \mathbf{A}^\tau}.

To simplify this visually:

The columns represent inputs: one column per component of $\mathbf{A}^\tau$ .
The rows represent outputs: one row per component of $\widehat{\mathbf{A}}^1$ .

An individual entry is $\mathbf{J}_{ij} = \frac{\partial \widehat{\mathbf{A}}^1_i}{\partial \mathbf{A}^\tau_j}$ , which answers the question: “If I slightly change the noisy input $\mathbf{A}^\tau_j$ , how much does the predicted clean action $\widehat{\mathbf{A}}^1_i$ change?”

Using this Jacobian, the guidance vector $\mathbf{g}$ is defined as:

\mathbf{g} = \mathbf{J}^\top \Big( \mathbf{W} \odot (\mathbf{Y} - \widehat{\mathbf{A}}^1) \Big)

where $\odot$ refers to element-wise product.

The Soft Mask

Earlier, we treated the mask $\mathbf{W}$ as a hard cutoff like $[1, 1, \dots, 0, 0]$ . In the paper, it is defined much more elegantly as a soft decay:

\mathbf{W}_i = \begin{cases} 1, & i < d \\ c_i\frac{e^{c_i}-1}{e-1}, & d \leq i < H-K \\ 0, & i \ge H-K \end{cases}

where $c_i=\frac{H-K-i}{H-K-d+1}$ A great visual for this is the blue curve below. It smoothly drops off so the network gently ignores the target $\mathbf{Y}$ toward the end of the chunk.

The Final Equation of RTC

Pulling it all together, the final modified velocity field used for execution is:

\mathbf{v}_{\mathrm{RTC}} = \mathbf{v}_\pi + \min\left( \beta, \frac{1-\tau}{\tau r_\tau^2} \right) \mathbf{J}^\top \Big( \mathbf{W} \odot (\mathbf{Y} - \widehat{\mathbf{A}}^1) \Big)

The coefficient out front:

$\min\left( \beta, \frac{1-\tau}{\tau r_\tau^2} \right)$

acts as a dynamic guidance strength $\lambda_\tau$ .

Remark

$\beta$ is a safety limit chosen empirically during experiments to prevent massive spikes.

The $\displaystyle \frac{1-\tau}{\tau r_\tau^2}$ is a theoretically derived, time-dependent guidance schedule originating from the paper 📄Training-free Linear Image Inverses via Flows.

In short, Inference-Time Real-Time Action Chunking boils down to evaluating the original velocity and adding a guide:

\boxed{ \mathbf{v}_{\mathrm{RTC}} = \mathbf{v}_\pi + \lambda_\tau \mathbf{g} }

Tip

If you want to try this out hands-on, don’t miss the implementation of RTC in LeRobot.

A Visual Guide to Real-Time Action Chunking in Robotics

Prerequisites

Action Chunking

Flow Matching

What does real-time mean?

Synchronous Inference

Asynchronous Inference

When Should the Next Inference Start?

1️⃣ The First Chunk

2️⃣ When should asynchronous inference start?

3️⃣ Which actions are not used?

4️⃣ Which actions does the controller execute?

5️⃣ Why can this sequence be problematic?

The Big Picture of Real-time Chunking

Real-Time Chunking as an Inpainting Problem

A Review of Ordinary Flow-Matching Generation

The Target

Estimating the Final Denoised Chunk

Formulating the Guide

The Soft Mask

The Final Equation of RTC

See also...