Learning word representations from context

1. Why did you choose Word2Vec?

I chose Word2Vec to understand how neural networks can learn useful representations of language from context.

What interested me was that words do not need to be assigned manually designed semantic features. Instead, a model can learn vector representations by predicting relationships between words in a corpus. Words that occur in similar contexts can develop similar representations, allowing some semantic and syntactic relationships to emerge from the training objective.

I wanted to understand the mechanics behind this process rather than simply load embeddings from a pretrained language model.

Word2Vec was a natural starting point because its relatively simple architecture exposes the fundamental ideas behind representation learning: context windows, embedding matrices, prediction objectives, gradient-based optimization, and the geometry of the learned vector space.

The project also gave me an opportunity to implement the same learning approach using NumPy and PyTorch, helping me connect the underlying mathematics with practical neural-network implementations.

2. What did you implement from scratch?

I implemented both major Word2Vec architectures: Continuous Bag of Words (CBOW) and Skip-gram.

CBOW: The model uses surrounding context words to predict a target word. Context embeddings are combined to form a representation that is used to score candidate target words.

Skip-gram: The model uses a target word to predict surrounding context words. Rather than aggregating the context into a single representation, it learns to predict words that occur near the input word.

I implemented the learning approach using NumPy first and then reproduced it in PyTorch.

The implementation work covered the core architecture and learning workflow, including constructing word-to-index mappings, preparing context-target training examples, maintaining embedding representations, and optimizing the model using a negative-sampling objective.

The NumPy implementation helped me work closer to the underlying mathematical operations, while the PyTorch version allowed me to express the model using tensors, automatic differentiation, and standard training utilities.

I would distinguish this from implementing every low-level operation in a production-grade Word2Vec system. The objective was to understand and reproduce the core learning method, not to build a fully optimized large-scale embedding-training library.

3. What mathematical concepts did you study?

The project involved understanding how embedding vectors can be learned through a probabilistic prediction objective.

CBOW objective

Given context words \(w_{t-c},\ldots,w_{t+c}\), excluding the target position, CBOW combines their embeddings to predict the center word \(w_t\).

A simplified context representation is:

where \(v_w\) is the embedding of a context word and \(C_t\) is the context window.

The model uses this representation to score candidate target words.

Skip-gram objective

Skip-gram reverses the prediction direction. Given a center word, it predicts surrounding context words.

For a context window of radius \(c\), the objective is to maximize the log-probability of observing context words around the center word:

This encourages the model to learn representations that are predictive of local linguistic context.

Negative sampling

Computing a normalized probability over a very large vocabulary can be expensive. Negative sampling replaces the full vocabulary prediction problem with a binary discrimination objective involving observed word-context pairs and sampled negative pairs.

For a positive pair \((w,c)\), a simplified objective is:

Here, \(v_w\) is the input embedding, \(v_c'\) is the output embedding for a genuine context word, \(v_{n_i}'\) represents a sampled negative word, \(K\) is the number of negative samples, and \(\sigma\) is the sigmoid function.

The model learns to assign higher compatibility scores to genuine word-context pairs and lower scores to sampled negative pairs.

Working through these objectives helped me understand how an apparently simple prediction task can produce a structured embedding space without explicit semantic labels.

4. What dataset did you use?

I trained the models on a text corpus used for learning word-context relationships.

The corpus was preprocessed to convert text into tokens and construct a vocabulary that could be used to represent words as integer indices. Context-target examples were then generated according to the selected context window.

The preprocessing and vocabulary representation are important because Word2Vec operates over discrete word identities while learning continuous vector representations.

The exact corpus name, number of tokens, vocabulary size, tokenization procedure, treatment of rare words, and train-validation split need to be recovered from the original project files before publication.

I would not substitute a toy example or a different public corpus for the actual training dataset.

5. What training configurations did you use?

The project involved training CBOW and Skip-gram models using NumPy and PyTorch.

The principal configuration choices for Word2Vec include:

• Embedding dimensionality.
• Context-window size.
• Number of negative samples.
• Learning rate and optimization method.
• Number of training epochs.
• Vocabulary construction and rare-word handling.
• Batch size and negative-sampling distribution.

These choices influence the information captured by the embeddings, the computational cost of training, and the quality of the learned representation.

The NumPy and PyTorch implementations provided two ways to express the same underlying learning approach. The former exposed the mathematical operations directly, while the latter made it easier to use automatic differentiation and tensor-based optimization.

The precise embedding dimensions, batch size, learning rate, optimizer settings, number of epochs, and hardware should be confirmed from the training code rather than estimated retrospectively.

6. What experiments did you run?

The main work involved implementing CBOW and Skip-gram, reproducing the approach in both NumPy and PyTorch, and inspecting the learned embedding space.

These implementations allowed me to examine the differences between the two prediction objectives and understand how each learns word representations from local context.

I also visualized the learned embeddings to explore their distribution and relationships in vector space.

I would characterize this as an implementation and exploratory-analysis project rather than a comprehensive benchmark study. I do not have verified evidence of systematic experiments comparing multiple embedding dimensions, context-window sizes, negative-sampling counts, or training-speed configurations.

A natural extension would be to vary these settings independently and measure their effects on training time, nearest-neighbor quality, and downstream performance.

7. How did you evaluate the embeddings?

I explored the learned representations through embedding visualization and inspection of relationships in the vector space.

A useful way to investigate these representations is cosine similarity:

Words with similar vectors have high cosine similarity, which can indicate that they appear in similar contexts or share some linguistic properties.

Other useful evaluation methods include nearest-neighbor inspection, semantic analogy tasks, and intrinsic benchmarks designed to test semantic or syntactic relationships.

For this project, I can substantiate the implementation of both architectures and the visualization of learned embeddings. I do not have verified quantitative benchmark scores or documented analogy-task results to report.

I would therefore use actual embedding visualizations and selected nearest-neighbor examples as evidence, rather than claim a numerical improvement over a baseline that was not systematically evaluated.

8. What problems did you encounter?

The project exposed several important considerations in learning word representations.

Context construction: Training examples must correctly represent the relationship between target words and their surrounding context. Errors in window construction can change the learning objective itself.

Negative sampling: The sampled negative examples influence which distinctions the model learns. Negative sampling must be implemented consistently with the intended objective.

Embedding optimization: The model learns separate input and output representations. Their updates must be consistent with the objective and gradient calculations.

Vocabulary limitations: Rare words provide fewer training examples, which can result in less reliable embeddings. A limited corpus also restricts the relationships the model can learn.

Numerical computation: Operations involving sigmoid functions and logarithms can become numerically unstable if implemented without appropriate care, particularly when the model produces extreme scores.

Implementation differences: Reproducing the model in NumPy and PyTorch requires the same mathematical objective and comparable training conditions if their outputs are to be compared meaningfully.

These challenges helped me appreciate that representation learning depends not only on the architecture, but also on the quality of the training examples, the optimization procedure, and the way the objective is implemented.

9. What did you learn, and what would you improve?

The project gave me a foundational understanding of how distributed representations emerge from predictive learning objectives.

I learned that an embedding is not inherently a dictionary definition of a word. It is a learned vector whose properties reflect the training data, the model architecture, and the objective used to optimize it.

Implementing CBOW and Skip-gram also helped me understand how different prediction directions lead to different learning procedures, even when both aim to capture relationships between words and their contexts.

If I extended the project, I would:

1. Compare CBOW and Skip-gram under controlled training conditions.
2. Investigate how context-window size and embedding dimensionality affect nearest-neighbor quality.
3. Evaluate the effect of negative-sampling configuration on representation quality.
4. Measure training speed and memory usage across the NumPy and PyTorch implementations.
5. Evaluate the embeddings on semantic similarity and analogy benchmarks.
6. Compare static Word2Vec embeddings with contextual representations from transformer-based language models.

The last direction particularly interests me because Word2Vec assigns a single learned representation to each vocabulary item, whereas contextual models can represent the same word differently depending on how it is used in a sentence.

This creates a useful starting point for understanding the progression from static distributional representations to modern contextual language representations.

10. Where can people see the work?

GitHub repository: https://github.com/HardikLovesTech/WordEmbedding

The repository is the main artifact for exploring the implementation.

For the portfolio, I would include:

• A comparison diagram illustrating CBOW and Skip-gram.
• The negative-sampling objective and an explanation of the gradient updates.
• A short comparison of the NumPy and PyTorch implementations.
• A visualization of the learned embedding space.
• Selected nearest-neighbor examples, provided they can be reproduced from the trained embeddings.
• A concise explanation of the dataset and training configuration.

The project's value is in making the learning process understandable and inspectable, from the mathematical objective through implementation to the resulting vector representations.

Overall, this project represents a foundational exploration of representation learning: implementing established methods, understanding their objectives, and examining the structure that emerges from training on language data.