Learning through interaction: Q-Learning and PPO

1. Why did you start exploring reinforcement learning?

I started exploring reinforcement learning to understand how agents learn through interaction, rather than relying entirely on supervised examples or predefined rules.

I was particularly interested in the mathematical foundations of learning: how an agent estimates the value of actions, balances exploration against exploitation, and improves its policy through experience. I wanted to connect the underlying theory with observable learning behavior by implementing algorithms, training agents, and examining how their performance changed over time.

This led me to experiment with both tabular Q-Learning and Proximal Policy Optimization (PPO). Q-Learning provided a way to understand value-based learning in discrete environments, while PPO introduced me to policy optimization, neural networks, and the practical challenges of training more complex agents.

My goal was to build a reusable experimentation workflow rather than a collection of isolated notebooks. I wanted to understand the algorithms, train agents in different environments, evaluate their behavior, and make the resulting models and demonstrations accessible to others.

2. Which environments did you use?

I experimented with several reinforcement learning environments spanning discrete-state problems, continuous-control tasks, and simulated agent behavior.

These environments allowed me to explore different learning challenges.

FrozenLake and Taxi provided relatively compact environments in which I could investigate value updates and exploration strategies. LunarLander offered a more demanding control problem where a neural policy must learn behavior from repeated interactions. CartPole provided a familiar benchmark for examining training progress and convergence.

I also explored the Huggy environment using Unity ML-Agents-related tooling, including exporting a trained policy to ONNX.

The exact algorithm-to-environment mapping and observation/action-space configurations should be confirmed against the final training scripts, particularly for any customized environments.

3. How did you implement Q-Learning and PPO?

I worked with two complementary reinforcement learning approaches.

Q-Learning

For discrete environments such as FrozenLake and Taxi, I worked with tabular Q-Learning.

The central idea is to maintain an estimate of the expected discounted return associated with taking an action in a particular state. The update rule is:

Here, \(\alpha\) is the learning rate, \(\gamma\) is the discount factor, and the expression inside the brackets is the temporal-difference error.

My experimentation involved the learning loop, reward collection, exploration, and evaluation of the learned policy. An important distinction is that tabular Q-Learning directly maintains action-value estimates rather than learning a neural-network policy.

Proximal Policy Optimization

For more complex tasks, I used PPO, working with PyTorch and Stable-Baselines3.

PPO is an on-policy policy-optimization algorithm that uses a clipped objective to limit excessively large policy updates. It commonly combines policy and value-function learning with advantage estimation.

My work involved configuring training runs, training agents in Gymnasium-compatible environments, evaluating their performance, and saving trained models for later use.

The project combines direct algorithmic experimentation with established RL tooling. I would distinguish the tabular Q-Learning implementation from the PPO implementation provided by Stable-Baselines3, rather than claim that both algorithms were independently implemented from scratch.

4. How was your training framework structured?

I organized the project around a repeatable training and evaluation workflow.

Environment initialization: Configure the Gymnasium environment, observation and action spaces, and episode termination conditions.

Agent configuration: Select the algorithm and its hyperparameters, including learning rate, discount factor, exploration settings, or PPO-specific optimization parameters.

Training: Run repeated environment interactions, collect rewards, and update the agent according to the selected algorithm.

Checkpointing: Save trained models so that experiments can be resumed or evaluated without retraining from scratch.

Evaluation: Test trained agents separately from training and examine their reward-based performance.

Visualization: Record gameplay or environment rollouts to inspect agent behavior and identify qualitative successes or failures.

Model publishing: Package trained agents and supporting documentation for sharing through GitHub and the Hugging Face Hub.

The main technology stack included Python, PyTorch, NumPy, Gymnasium, and Stable-Baselines3, with Unity ML-Agents-related tooling for the Huggy experiment.

The repository includes saved-model artifacts, notebooks, and replay-video assets. The workflow was designed to make experiments easier to reproduce, inspect, and share.

5. What experiments did you run?

My experimentation focused on training agents in environments with different learning dynamics and evaluating how effectively they learned useful policies.

The primary dimensions of interest included:

• Learning rate: How aggressively the agent updates its value estimates or policy.
• Discount factor: How strongly future rewards influence current decisions.
• Exploration strategy: How the agent balances discovering new actions with exploiting previously successful behavior.
• Training duration: How performance changes as the number of environment interactions increases.
• Policy optimization: How PPO's optimization configuration affects learning and stability.
• Evaluation conditions: Whether a trained agent performs consistently across separate episodes rather than succeeding in a single favorable rollout.

For tabular Q-Learning, exploration and value-update behavior are particularly important. For PPO, the optimization process, rollout collection, and policy-update configuration introduce additional sources of variation.

The project also involved checkpointing and model evaluation, allowing trained policies to be tested and visualized after training.

Exact hyperparameter sweeps, network architectures, reward modifications, and experiment counts should be reported from the configuration files and training logs rather than reconstructed from memory.

6. What results did you observe?

One of the reported results in my project materials is that CartPole-v1 approached convergence at approximately 100,000 training steps.

This provides a useful reference point for discussing training progress, but the precise definition of convergence should be tied to the evaluation protocol used in the experiment.

For example, a rigorous benchmark could define success using the mean episodic return over a fixed window of evaluation episodes, with a stated reward threshold and a fixed evaluation procedure.

Beyond this result, I trained agents for discrete environments such as FrozenLake and Taxi and worked with PPO-based agents for LunarLander and Huggy. I also published trained-agent artifacts through the Hugging Face Hub.

The project helped me move beyond observing whether an agent eventually succeeds to thinking about how to measure learning progress, evaluate a policy separately from training, and preserve trained models for reproducible comparisons.

Exact mean rewards, success rates, reward curves, and variance across random seeds should be added only after checking the corresponding evaluation records.

7. What problems did you encounter?

Several challenges emerged from working with different reinforcement learning algorithms and environments.

Exploration versus exploitation: An agent that exploits early successes too aggressively may fail to discover better strategies. Conversely, excessive exploration can delay learning.

Training variability: Reinforcement learning outcomes can vary across random seeds and environment trajectories. A successful episode does not necessarily imply that a policy has learned a consistently good strategy.

Hyperparameter sensitivity: Learning rates, exploration schedules, and policy-optimization settings can substantially influence learning speed and stability.

Evaluation reliability: Training rewards alone can give an incomplete picture of policy quality. Separate evaluation episodes are important for determining whether an agent has learned behavior that generalizes beyond its training experience.

Environment and tooling compatibility: Different environments have different observation and action spaces, termination conditions, and model interfaces. Saving, loading, and deploying agents requires attention to the particular environment and algorithm.

Reproducibility: Comparing experiments requires consistent configurations, documented training conditions, and reliable model checkpoints.

These challenges reinforced the importance of treating reinforcement learning as an experimental discipline rather than simply running an algorithm until a reward curve looks promising.

8. What did you learn, and what would you improve?

The project helped me understand the differences between value-based learning and policy optimization in practical settings.

Q-Learning made the relationship between rewards, value estimates, exploration, and policy improvement concrete. PPO introduced me to the additional complexity of neural policies and optimization-based training.

I also learned that training an agent is only part of the work. Reliable evaluation, reproducible experiments, checkpoint management, visualization, and clear documentation are necessary to understand and communicate the results.

If I extended the framework, I would prioritize:

1. Reproducible benchmarks: Run multiple random seeds and report mean returns, variability, and success rates.
2. Learning-curve analysis: Track episodic rewards and evaluation performance throughout training rather than relying only on final checkpoints.
3. Controlled ablations: Change one hyperparameter at a time to investigate its effect on convergence and stability.
4. Standardized evaluation: Separate training and evaluation environments and use consistent evaluation conditions.
5. Experiment tracking: Store configurations, software versions, random seeds, checkpoints, and evaluation summaries together.
6. Algorithm comparisons: Compare Q-Learning and PPO where the environment and action-space assumptions make a meaningful comparison possible, while recognizing that tabular Q-Learning is not directly interchangeable with PPO on every task.
7. Deeper algorithmic investigation: Examine how policy-update constraints, advantage estimation, exploration, and reward structure influence learning behavior.

This work also strengthened my interest in understanding the mechanisms behind learning algorithms, including why training succeeds in some configurations and becomes unstable in others.

9. Where can people see the work?

GitHub repository: https://github.com/HardikLovesTech/ReinforcementLearning

The repository contains the reinforcement learning experimentation work, including code, notebooks, model artifacts, and replay materials.

Published Taxi-v3 agent: https://huggingface.co/HardikLovesTech/Taxi-v3

Published PPO LunarLander-v3 agent: https://huggingface.co/HardikLovesTech/ppo-LunarLander-v3

Published PPO Huggy agent: https://huggingface.co/HardikLovesTech/ppo-Huggy

The Hugging Face artifacts provide concrete evidence of model packaging and sharing, while the GitHub repository offers a starting point for examining the implementation and experimentation workflow.

For the portfolio, I would highlight the repository, selected agent pages, a representative training curve, and a replay video. Together, these assets demonstrate both the algorithmic work and the complete workflow from training through evaluation and publication.