HKUST PhD Chronicle, Week 47, Normalization & RL Milestones

Normalization

The biggest lesson learned this week is that normalization matters! Looking at these 2 failed sample rollouts:

When the policy failed, the robot executed the exact same trajectory regardless of where the button was actually placed on the table, i.e. the action conditioned on the observation was not working as expected:

Remark

The root cause was a hardware side offset that shifts the observations away from the standard distribution. If you are curious about the details, I have documented everything I leaned at Dataset Normalization Statistics in OpenPI and LeRobot.

Reinforcement Learning

As a robot learning researcher coming from a design background, I am proud to have finally finished the most essential parts of chapter 7 in ADM:

• policy evaluation
• policy iteration
• value iteration

That is a milestone I wanted to celebrate because I believe these fundamental concepts are the stepping stone for my long term research journey. I still remember how uncomfortable I felt when I first read Reinforcement Learning: An Introduction. As the time passes🍂, I am no longer the dude who knows nothing about RL.

Exchanging Ideas

As a student helpers at the CKSRI, we were treated with a nice buffet this Saturday. It was a blessing to chat with other PhD students from different background, for example control theory, a field I deeply appreciate but currently know very little about.

Remark

When I think of control theory, terms like like nonlinear dynamics, MPC, Lyapunov function, trajectory optimization…

I am happy to report that I found an overlap between my work (robotics) and theirs(control theory): 📄Safe, Task-Consistent Manipulation with Operational Space Control Barrier Functions. This applies Control Barrier Function(control theory) into teleoperation(robot learning). I was also able to explain the big picture, from the desired joint position $q^d$, to kinematics limits, to control barrier functions, down to the Jacobian matrix and final torque $\tau$.

However, there is still a gap in my knowledge. I can explain inverse kinematics within a linear system, but I struggled to highlight the low-level controller optimization part.

Still, it was a great conversation! As Richard Hamming said, as a researcher, you should always be open to talking and sharing ideas.