Humanoid task training
I designed a task in Isaac Lab that trains a Unitree G1 humanoid to insert a plug into an outlet. I built the simulation environment around the plug and outlet, with alignment and contact as part of the task. I designed the reward function and iterated on the training hyperparameters to get the expert policy to complete the insertion.
I implemented OmniReset with a reset-state generator I wrote for this task. The generator gives the policy a range of initial states, so training can include more attempts at the later stages of insertion without waiting for the policy to reach those stages on its own. OmniReset paper and project ↗
This task is part of a sim-to-real transfer project. The video shows the simulation-trained expert, which serves as a starting point for studying how the task can transfer to the physical humanoid.
Reinforcement learning from scratch
I implemented REINFORCE, GRPO, and PPO in PyTorch. I tested my PPO implementation on Gymnasium’s Lunar Lander, where the trained policy reached a 92% success rate.
Experiments
- Experimented across RL algorithms and action-space configurations for humanoid control.
- Implemented and trained a Mamba (state-space) encoder that gives the policy a history-aware latent state.
Investigating flow-matching world action models — using flow matching in place of MLPs — and comparing ways to fine-tune pretrained models: DSRL, residual policies, and direct fine-tuning.