RL Quadruped Locomotion

The task is built around command following: the robot receives randomized x velocity, y velocity, yaw-rate, and heading targets, then learns joint-position actions that move the body while keeping it upright. That makes the project less about hand-designing one gait and more about training a policy that can respond to changing motion goals.

I configured the policy observation space around proprioceptive signals: relative joint positions, base linear and angular velocity, projected gravity, joint velocity, last actions, and the generated command. I also kept policy observations separate from critic observations so asymmetric actor-critic training could use richer training signals without assuming those same signals would be deployed on hardware.

Velocity tracking alone can produce unstable or unnatural behavior, so the reward design has to balance motion with posture, smoothness, and physical plausibility. I combined command-tracking rewards with upright posture, default joint posture regularization, action-rate penalties, joint-limit penalties, and fall termination.

The optional gait terms are where the project gets especially interesting: foot clearance, swing height, foot slip, air time, soft landing, and posture shaping all change what kind of gait the policy prefers. The challenge is rewarding useful walking without accidentally encouraging dragging, slipping, high-frequency actuator commands, or simulator exploits.

To keep the policy from only memorizing ideal flat-ground motion, I trained with domain randomization. Ground friction is randomized, the robot resets with varied pose and yaw, and interval pushes perturb the robot with x/y velocity disturbances during training.

Those events force the policy to practice recovery behavior. Instead of evaluating only whether it can walk in a clean scene, I can look at how it reacts when contact conditions and body state are pushed away from the nominal case.

The videos are useful because they make failure modes visible: foot dragging, body pitch drift, recovery timing, slipping, and unstable contact patterns show up quickly when the policy is played back. I used rollout review as a way to connect the reward configuration back to the gait quality it produced.

That evaluation loop is the core engineering value of the project. It turns legged locomotion into an iterative RL design problem where command sampling, observation noise, reward weights, termination thresholds, and domain randomization can be changed and then inspected in the resulting policy behavior.

The next improvements are to document the final trained checkpoint, training duration, number of parallel environments, and best reward configuration. I would also tune the gait-specific reward terms more aggressively and add clearer result plots for command-tracking accuracy, reward curves, and fall or termination rate.