Master DROID Action Spaces to Use Pre-Trained Policies

Defining the “Left”

We must be clear about what “left” means in the dataset schema. The schema includes:

• exterior_image_1_left
• exterior_image_2_left
• wrist_image_left

It is a common pitfall to assume “left” means the camera is positioned on the left side of the robot’s base frame. Instead, “left” refers exclusively to the left lens of the Zed camera, which serves as the canonical camera feed.

A ZED camera has two lenses.

How Policies Use the Camera Views

Because DROID provides 3 camera views, you might assume all VLA models use all of them. The reality is a bit more complex😵‍💫.

Remark

Below, I refer to exterior_1 as the left-hand side camera and exterior_2 as the right-hand side camera relative to the robot base frame.

In the robot base frame, the left side has $y>0$ and the right side has $y<0$.

• $x$ axis is red
• $y$ axis is green
• $z$ axis is blue

📌 OpenPI $\pi_{0.5}$-DROID

In its fine-tuning config, $\pi_{0.5}$ only allocates slots for 2 camera views (one wrist, one exterior). However, it does not permanently drop either the left or right third-person camera.

During training, OpenPI randomly chooses between the left and right exterior cameras and feeds it into the policy’s single third-person input slot. The policy learns from both viewpoints but only processes one at a time. During inference, the operator must explicitly choose to roll out with either the left or right camera. See droid_rlds_dataset.py.

📌 NVIDIA GR00T N1.7-DROID

Conversely, nvidia/GR00T-N1.7-DROID explicitly drops the right-hand side exterior_2_left camera and only includes the left-hand side camera in its fine-tuning script.

📌 MolmoAct2-DROID

The final approach uses everything. The observation space for allenai/MolmoAct2-DROID includes all 3 cameras during both training and inference.

These examples show why we need to check a checkpoint’s actual inputs🔎, even when all 3 checkpoints are described as “DROID models”.

Notation

Inspired by Russ Tedrake and Introduction to Robotics: Mechanics and Control, I will define the notation first, then apply it to the DROID pipeline.

📌 Position vector: $p$

$p$ denotes a position vector. A position requires a reference point and axes. We write:

${}^A p^C_F,$

which reads as “the position of point $C$, measured from point $A$, with components expressed in frame $F$‘s basis.”

📌 Omitting the subscript

${}^F p^A \equiv {}^F p^A_F$

When the “measured from” and “expressed in” frames are identical, we drop the subscript. Every position in this pipeline is expressed in the same frame it is measured from.

📌 Orientation: $R$; pose: $X$

$R$ denotes rotation: ${}^B R^A$ is the orientation of frame $A$ measured from frame $B$. $X$ denotes a rigid transformation (a pose), which packages both:

${}^B X^A = \left({}^B R^A, {}^B p^A\right).$

Remark

Concretely, ${}^B X^A$ is a $4\times4$ homogeneous transformation matrix: ${}^B R^A$ is the top-left $3\times3$ block, and ${}^B p^A$ is the first 3 entries of the 4th column. ${}^B X^A=\begin{bmatrix}{}^B R^A & {}^B p^A\\\mathbf 0^\top & 1\end{bmatrix}\in SE(3).$

📌 Symbols in DROID teleoperation pipeline

Let:

• $k$ denote the control-step index.
• $\{S\}$ denote the stationary Franka robot base frame (the world frame).
• $\{E\}$ denote the end-effector (flange) frame.
• $\{C\}$ denote the Oculus controller frame.
• $\mathbf q[k],\dot{\mathbf q}[k]\in\mathbb R^7$ denote the measured joint positions and velocities.
• $g[k]\in[0,1]$ denote measured gripper closedness.
• ${}^S X^E[k]$ denote the pose of $\{E\}$ measured from $\{S\}$.

Applying the $X = (R, p)$ decomposition to the end-effector gives:

For an ideal rigid arm with a fixed base, its mechanical state can be written as

Remark

To connect this to hardware, you can read these directly from libfranka: C++ library for Franka Robotics research robots:

• $\mathbf{q}[k]$ corresponds to std::array franka::RobotState::q
• $\dot{\mathbf{q}}[k]$ corresponds to std::array franka::RobotState::dq

The flange pose is determined by forward kinematics:

Check the end-effector frame in libfranka

RobotState::O_T_EE is the pose of configured end-effector frame in Desk, which need not be the flange. Its F_T_EE field describes the transform from flange to end effector. In matrix form, the flange pose is O_T_EE multiplied by the inverse of F_T_EE. DROID instead computes the pose of its configured model link from joint angles. See the libfranka field definitions.

• O_T_EE reads as “origin to end-effector”.
• F_T_EE reads as “flange to end-effector”

TODO: add a diagram of Franka Desk

The proprioceptive part of the released RLDS observation

can therefore be written as

Defining the Control Boundaries

In control theory, we often write $\mathbf u[k]$ for “the” control input. In robot learning, what we call an “action” depends entirely on where we draw the system boundary. Consider 3 distinct levels of control in this pipeline from top to bottom:

1. The DROID conversion layer: The input is a processed teleoperation command.
2. The low-level controllers: The input consists of absolute joint and gripper position targets $(\mathbf q^d[k], g^d[k])$.
3. The physical arm: The input is joint torques $\boldsymbol\tau(t)$.

1. The Human-VR Outer Loop: cartesian_velocity

The cartesian_velocity action can be defined as:

$\mathbf a[k]= \begin{bmatrix} \mathbf a_p[k]\\ \mathbf a_\theta[k]\\ a_g[k] \end{bmatrix}\in[-1,1]^7.$

The fun fact is that the cartesian_velocity has nothing to do with standard physical velocity (like m/s or rad/s). What?🤯 Yes, you read that right. The literal “velocity” label tripped me up at first, too.

So what it is?🧐 Let’s break it down.

Tip

The big picture is: the recorded action action_dict/cartesian_velocity has two parts:

• Indices 0-2 control translation.
• Indices 3-5 control rotation.

The gripper has a separate scalar command.

📌 Tracking error

DROID’s VR controller uses proportional feedback. When we start teleoperation, we reset the motion reference and store the controller pose and robot pose. Call this reference step $k_0$📸.

For translation, positions are measured in meters, so each displacement below is also in meters:

Remark

Intuitively,

• $\Delta p_C[k]$ measures the displacement(translation) of the Oculus Quest 2 controller from the latched pose.
• $\Delta p_E[k]$ measures the translation between the current end-effector frame and when it was at the time $k_0$.

The translation error, also in meters, is

Example

If your hand has moved forward by 0.10 m and the robot has moved forward by 0.06 m, the translation error is 0.04 m.

For rotation, we have a similar concept. However, rather than simple subtraction, we need relative rotations:

DROID forms the rotation error and converts it to Euler coordinates in radians:

Remark

The code computes these products with quaternions, which is equivalent to the rotation-matrix calculation above. Here, $\operatorname{Euler}_{xyz}$ refers to X-Y-Z fixed angles (SciPy’s lowercase xyz convention). This is an Euler representation of the relative rotation, not a rotation-vector logarithm, therefore it still has Euler-angle branch cuts and singularities.

📌 Gains

Recall that we usually need to set $K_p$ and $K_d$ for a PD controller. The same logic applies to DROID error tracking. With the current defaults, DROID scales the errors as follows:

Remark

Here, the equations follow the code’s numerical convention: $\mathbf e_p$ contains position errors expressed in meters, and $\mathbf e_\theta$ contains Euler-angle errors expressed in radians.

📌 The $L_2$ norm

Translation and rotation are independently rescaled so that each Euclidean norm is at most 1:

Example

Suppose $\mathbf u_p[k]=[1.2,0.9,0]^\top$. Its norm is $\sqrt{1.2^2+0.9^2+0^2}=1.5$, so DROID divides the vector by 1.5. The result is $\mathbf a_p[k]=[0.8,0.6,0]^\top$: the direction stays the same, but the magnitude is capped.

📌 Scalar clipping

The gripper command is clipped to the interval $[-1,1]$:

The arm command is

This is action_dict/cartesian_velocity, while $a_g[k]$ is action_dict/gripper_velocity.

cartesian_velocity is a normalized command

As I reminded you before, the field name can be misleading. These values are gain-scaled, saturated tracking errors. They do not directly measure how fast the Oculus controller or robot is moving. (i.e., it is NOT a physical velocity in standard SI).

2. Scaling and Recording the Target: cartesian_position

The outer loop gave us a normalized cartesian_velocity. Before this signal ever reaches a joint, DROID scales it back into physical units and uses it to construct an absolute pose which is recorded alongside everything else as action_dict/cartesian_position.

Remark

This construction, and the differential IK that follows it, is split between the IK wrapper and FrankaRobot.create_action_dict.

0️⃣ The cartesian_velocity

Recall that the cartesian_velocity is

1️⃣ Scale the Cartesian command

The wrapper repeats the separate $L_2$ limits on translation and rotation, then applies the default scales

For the already bounded commands from Section 1, this gives

Tip

Now we finally feel a connection to the physical world again. The dimensionless “maximum” value of 1 in translation now becomes “0.075m / control step”.

Remark

These scales define the increments used to construct a Cartesian target on each update. They do not guarantee that the robot will move to the exact target during the next 15 Hz interval.

2️⃣ Construct the recorded Cartesian target

Let the subscript $r$ mark the robot state read when the command is resolved. DROID constructs

Here, $R_{xyz}(\boldsymbol\alpha)=R_z(\alpha_z)R_y(\alpha_y)R_x(\alpha_x)$ converts extrinsic xyz Euler angles to a rotation matrix. Translation is added in the base frame, and the incremental rotation is composed on the left.

The translation ${}^S p^{E_d}[k]$ and rotation ${}^S R^{E_d}[k]$ together define the action_dict/cartesian_position.

The reference state is read at command time

A single control cycle performs two separate robot state reads:

• $\mathbf p[k]$: the saved observation (read early, before computing the action).
• $\mathbf p_r[k]$: the fresh reference (read later by create_action_dict).

DROID constructs the target as $\mathbf p^d[k] = \mathbf p_r[k] + \mathbf d_p[k]$. Why use the later read? 🤔 My guess is to ensure the intended motion is “fully expressed.”

Because the arm moves between reads, anchoring the command to the outdated $\mathbf p[k]$ would shortchange the displacement. For example: if $p[k]=0.50$ m, the robot drifts to $p_r[k]=0.51$ m by command time, and the desired delta $d_p[k]=0.02$ m. A target of $0.52$ m (based on $p[k]$) only yields $0.01$ m of actual physical movement. Adding the delta directly to the fresh $p_r[k]$ ensures the full $0.02$ m offset executes.

3. Differential IK: joint_velocity

Alright alright alright, we see the word “velocity” again in joint_velocity. Does it also lack a standard physical meaning? Is it not in SI units?

Correct, joint_velocity is also normalized! :- )

DROID uses dm_robotics’s Cartesian6dVelocityEffector for differential IK, where the input and output can be viewed as

where

• input includes
  • $\mathbf q$: the current joint position
  • $\dot{\mathbf q}$: the current joint velocity
  • $\boldsymbol\xi^d$: the desired twist
• output includes
  • $\dot{\mathbf q}^d$: the desired joint velocity

Let $F$ denote the DROID wrapper’s calculation. It treats the measured joint positions $\mathbf q_r[k]$, measured velocities $\dot{\mathbf q}_r[k]$, and the scaled 6-vector $\mathbf d[k]$ as input. The wrapper computes

where $L_q=0.2\ \mathrm{rad}$ is the default joint-step scale.

Remark

A few things are worth mentioned⭐️:

• The scaled 6-vector $\mathbf d[k]$(cartesian_velocity) is not rigorously the twist we know from robotics textbook like Modern Robotics: Mechanics, Planning, and Control. It’s more of a “Cartesian increment”.
• Vice versa, the dm_robotics solver’s numerical output is treated as a joint increment $\mathbf b^q[k]$ in radians.

And ladies and gentlemen, the normalized result $\mathbf a^q[k]$ is saved as action_dict/joint_velocity.

📌Where does the “normalized” come from?

The joint_velocity $\mathbf a^q[k]$ is normalized such that

Looking at the equation $\displaystyle \mathbf a^q[k]=\frac{\mathbf b^q[k]}{L_q}$, this bounded range might not be obvious. It works because DROID sets the same numerical limit for all 7 joints at IK configuration:

self.relative_max_joint_delta = np.array([0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2])

This array is passed to Cartesian6dVelocityEffector. Because the effector intrinsically bounds its own output. The joint increment $\mathbf b^q[k]$ produced by $F$ is already constrained to [-0.2, 0.2]rad. Therefore

4. Resolving the Joint Target: joint_position

DROID converts the joint_velocity command into an increment using $L_q=0.2\,\mathrm{rad}$ for every joint, applies infinity norm saturation if needed, and adds it to the reference configuration:

From last section, we know that $\mathbf a^q[k]$ is already normalized such that $\lVert\mathbf a^q[k]\rVert_\infty\le1$. Therefore, the preceding equation could be simplified as:

Tip

Intuitively, it caps each joint increment $\Delta\mathbf q[k]$ is at most $0.2\,\mathrm{rad}$ in magnitude. Note that this operation alone does not enforce the physical joint limits $q_{\min}, q_{\max}$ of the robot.

IK(cartesian_position) does NOT guarantee joint_position

As you might suspect, running pose-based IK on the recorded cartesian_position, even when seeded with the observed joint configuration, is not guaranteed to reproduce the recorded joint_position. Reproduction requires following DROID’s differential-IK and joint target construction path.

5. The Common Pitfall

We now have the joint target $\mathbf q^d[k]$. They are sent to their low-level controllers, but the robot does not reach them instantly.

The arm controller produces motor torques $\boldsymbol\tau(t)$, while the physical response depends on inertia, gravity, friction, contact, latency, etc. The DROID outer loop targets 15 Hz, while the Franka hardware client is configured for 1 kHz.

We therefore cannot assume

Commanded action versus observed motion

You generally cannot recover the commanded action by subtracting two consecutive observations. In particular, the expression

$\frac{{}^S p^E[k+1]-{}^S p^E[k]}{L_p}$

measures a scaled observed displacement. It is NOT generally equal to $\mathbf a_p[k]$. They may coincide in a special case, but this is not a reliable way to reconstruct actions.