Language Model with Neural Net

Code Source


This project works with word embeddings and making neural networks learn about words. 📖

We could try to match statistics about the words, or we could train a network that takes a sequence of words as input and learns to predict the word that comes next.

Visual 2 Visual 3 Visual 1
Result Visualization

Starter code and data

Look at the file raw_sentences.txt.

No , he says now . And what did he do ? The money ‘s there . That was less than a year ago . But he made only the first . There ‘s still time for them to do it . But he should nt have . They have to come down to the people . I do nt know where that is . No , I would nt . Who Will It Be ? And no , I was not the one . You could do a Where are they now ? There ‘s no place like it that I know of . Be here now , and so on . It ‘s not you or him , it ‘s both of you . So it ‘s not going to get in my way . When it ‘s time to go , it ‘s time to go . No one ‘s going to do any of it for us . Well , I want more . Will they make it ? Who to take into school or not take into school ? But it ‘s about to get one just the same . We all have it . So we will be .

It contains the sentences that we are going to analyse These sentences are fairly simple ones and cover a vocabulary of only 250 words.

First, data import

import collections
import pickle
import numpy as np
from tqdm import tqdm
import pylab
use_colab = True
if use_colab:
  from google.colab import drive
import os

TINY = 1e-30
EPS = 1e-4
nax = np.newaxis
if use_colab:
  drive_name = '/content/drive'
  drive_folder = # your drive folder
  drive_location = drive_name + '/My Drive/' + drive_folder  # Change this to where your files are located
  # set the drive_location variable to whereever the extracted contents are.
  drive_location = os.getcwd()
  # print(drive_location)

data_location = drive_location + '/' + ''
PARTIALLY_TRAINED_MODEL = drive_location + '/' + ''
Go to this URL in a browser: ****

Enter your authorization code:
Mounted at /content/drive

We have already extracted the 4-grams from this dataset and divided them into training, validation, and test sets. To inspect this data, run the following:

data = pickle.load(open(data_location, 'rb'))
[[ 27  25  89]
 [183  43 248]
 [182  31  75]
 [116 246 200]
 [222 189 248]
 [ 41  73  25]
 [241  31 222]
 [222  31 157]
 [ 73  31 220]
 [ 41 191  90]]
[143 116 121 185   5  31  31 143  31  67]

Now data is a Python dict which contains the vocabulary, as well as the inputs and targets for all three splits of the data. data[‘vocab’] is a list of the 250 words in the dictionary; data[‘vocab’][0] is the word with index 0, and so on. data[‘train_inputs’] is a 372,500 x 3 matrix where each row gives the indices of the 3 context words for one of the 372,500 training cases.

data[‘train_targets’] is a vector giving the index of the target word for each training case. The validation and test sets are handled analogously.

Part 1: GLoVE Word Representations

Now, we will implement a simplified version of the GLoVE embedding with the loss defined as

\(L(\{\mathbf{w}_i,b_i\}_{i=1}^V) = \sum_{i,j=1}^V (\mathbf{w}_i^\top\mathbf{w}_j + b_i + b_j - \log X_{ij})^2\).

Note that each word is represented by a d-dimensional vector \(\mathbf{w}_i\) and a scalar bias \(b_i\).

We have provided a few functions for training the embedding:

  • calculate_log_co_occurence computes the log co-occurrence matrix of a given corpus
  • train_GLoVE runs momentum gradient descent to optimize the embedding
  • loss_GLoVE: INPUT - \(V\times d\) matrix \(W\) (collection of \(V\) embedding vectors, each d-dimensional); \(V\times 1\) vector \(\mathbf{b}\) (collection of \(V\) bias terms); \(V \times V\) log co-occurrence matrix. OUTPUT - loss of the GLoVE objective
  • grad_GLoVE: INPUT - \(V\times d\) matrix \(W\), \(V\times 1\) vector b, and \(V\times V\) log co-occurrence matrix. OUTPUT - \(V\times d\) matrix grad_W containing the gradient of the loss function w.r.t. \(W\); \(V\times 1\) vector grad_b which is the gradient of the loss function w.r.t. \(\mathbf{b}\).

The math just looks awful huh? Some code will make sense!

vocab_size = 250

def calculate_log_co_occurence(word_data):
  "Compute the log-co-occurence matrix for our data."
  log_co_occurence = np.zeros((vocab_size, vocab_size))
  for input in word_data:
    log_co_occurence[input[0], input[1]] += 1
    log_co_occurence[input[1], input[2]] += 1
    # If we want symmetric co-occurence can also increment for these.
    # Optional: How would you generalize the model if our target co-occurence isn't symmetric?
    log_co_occurence[input[1], input[0]] += 1
    log_co_occurence[input[2], input[1]] += 1
  delta_smoothing = 0.5  # A hyperparameter.  You can play with this if you want.
  log_co_occurence += delta_smoothing  # Add delta so log doesn't break on 0's.
  log_co_occurence = np.log(log_co_occurence)
  return log_co_occurence

log_co_occurence_train = calculate_log_co_occurence(data['train_inputs'])
log_co_occurence_valid = calculate_log_co_occurence(data['valid_inputs'])

calculate the gradient of the loss function w.r.t. the parameters \(W\) and \(\mathbf{b}\). You should vectorize the computation, i.e. not loop over every word.

def loss_GLoVE(W, b, log_co_occurence):
  "Compute the GLoVE loss."
  n,_ = log_co_occurence.shape
  return np.sum((W @ W.T + b @ np.ones([1,n]) + np.ones([n,1])@b.T - log_co_occurence)**2)

def grad_GLoVE(W,  b, log_co_occurence):
  "Return the gradient of GLoVE objective w.r.t W and b."
  "INPUT: W - Vxd; b - Vx1; log_co_occurence: VxV"
  "OUTPUT: grad_W - Vxd; grad_b - Vx1"
  n,_ = log_co_occurence.shape

  grad_W = 4*(W @ W.T + b @ np.ones([1,n]) + np.ones([n,1]) @ b.T - log_co_occurence) @ W
  grad_b = 4*(W @ W.T + b @ np.ones([1,n]) + np.ones([n,1]) @ b.T - log_co_occurence) @ np.ones([n,1])

  return grad_W, grad_b

def train_GLoVE(W, b, log_co_occurence_train, log_co_occurence_valid, n_epochs, do_print=False):
  "Traing W and b according to GLoVE objective."
  n,_ = log_co_occurence_train.shape
  learning_rate = 0.2 / n  # A hyperparameter.  You can play with this if you want.
  for epoch in range(n_epochs):
    grad_W, grad_b = grad_GLoVE(W, b, log_co_occurence_train)
    W -= learning_rate * grad_W
    b -= learning_rate * grad_b
    train_loss, valid_loss = loss_GLoVE(W, b, log_co_occurence_train), loss_GLoVE(W, b, log_co_occurence_valid)
    if do_print:
      print(f"Train Loss: {train_loss}, valid loss: {valid_loss}, grad_norm: {np.sum(grad_w**2)}")
  return W, b, train_loss, valid_loss

Train the GLoVE model for a range of embedding dimensions

n_epochs = 500  # A hyperparameter.  You can play with this if you want.
# embedding_dims = np.array([1,2,7,10,11,12,15,50,100, 120, 300])  # Play with this
embedding_dims = np.array([1,2,256])
final_train_losses, final_val_losses = [], []  # Store the final losses for graphing
W_final_2d, b_final_2d = None, None
do_print = False  # If you want to see diagnostic information during training
for embedding_dim in tqdm(embedding_dims):
  init_variance = 0.1  # A hyperparameter.  You can play with this if you want.
  W = init_variance * np.random.normal(size=(250, embedding_dim))
  b = init_variance * np.random.normal(size=(250, 1))
  if do_print:
    print(f"Training for embedding dimension: {embedding_dim}")
  W_final, b_final, train_loss, valid_loss = train_GLoVE(W, b, log_co_occurence_train, log_co_occurence_valid, n_epochs, do_print=do_print)
  if embedding_dim == 2:
    # Save a parameter copy if we are training 2d embedding for visualization later
    W_final_2d = W_final
    b_final_2d = b_final
  final_train_losses += [train_loss]
  final_val_losses += [valid_loss]
  if do_print:
    print(f"Final validation loss: {valid_loss}")

100%|██████████| 3/3 [00:11<00:00,  3.85s/it]

Plot the training and validation losses against the embedding dimension.



Some questions to ask yourself:

  1. Given the vocabulary size \(V\) and embedding dimensionality \(d\), how many parameters does the GLoVE model have?
\[V\times d + d\]
  1. Write the gradient of the loss function with respect to one parameter vector \(\mathbf{w}_i\).

\(\frac{\partial L}{\partial w_i} = 2\sum_{j=1, j\neq i}^{V}(w_i^Tw_j + b_i + b_j - \log{X_{ij}})w_j\) \(= 4\sum_{j=1}^{V}(w_i^Tw_j + b_i + b_j - \log{X_{ij}})w_j\)

\(\frac{\partial L}{\partial b_i} = 2\sum_{j=1, j\neq i}^{V}w_i^Tw_j + b_i + b_j - \log{X_{ij}}\) \(= 4\sum_{j=1}^{V}w_i^Tw_j + b_i + b_j - \log{X_{ij}}\)

  1. Train the model with varying dimensionality \(d\). Which \(d\) leads to optimal validation performance? Why does / doesn’t larger \(d\) always lead to better validation error?

when d = 11, d leads to optimal validation performance with the smallest validation error. This dimentionality indicates a possible encoding of \(2^{11} = 2048\) ways. Model may be over fit if dimension is too large, while underfit if dimension is too small.

Part 2: Network Architecture

    • Word embedding weight: \(250\times 16\)

    • Embedded to hidden weight: \(3\times16\times 128\)

    • Hidden to output weight: \(128\times 250\)

    • Hidden bias: \(128\times 1\)

    • Output bias: \(250\times1\)

    • Total: 42522 parameters. The hidden to output weight has the most parameters

  1. A 4-grams model is a model predict the 4th word from 3 previous words. That is, we have 250 vocabularies in total, there is a combination of \(250^4 = 3906250000\) possible non-repeated outcomes.

Part 3: Training the model

There are three classes defined in this part: Params, Activations, Model.

class Params(object):
    """A class representing the trainable parameters of the model. This class has five fields:
           word_embedding_weights, a matrix of size N_V x D, where N_V is the number of words in the vocabulary
                   and D is the embedding dimension.
           embed_to_hid_weights, a matrix of size N_H x 3D, where N_H is the number of hidden units. The first D
                   columns represent connections from the embedding of the first context word, the next D columns
                   for the second context word, and so on.
           hid_bias, a vector of length N_H
           hid_to_output_weights, a matrix of size N_V x N_H
           output_bias, a vector of length N_V"""

    def __init__(self, word_embedding_weights, embed_to_hid_weights, hid_to_output_weights,
                 hid_bias, output_bias):
        self.word_embedding_weights = word_embedding_weights
        self.embed_to_hid_weights = embed_to_hid_weights
        self.hid_to_output_weights = hid_to_output_weights
        self.hid_bias = hid_bias
        self.output_bias = output_bias

    def copy(self):
        return self.__class__(self.word_embedding_weights.copy(), self.embed_to_hid_weights.copy(),
                              self.hid_to_output_weights.copy(), self.hid_bias.copy(), self.output_bias.copy())

    def zeros(cls, vocab_size, context_len, embedding_dim, num_hid):
        """A constructor which initializes all weights and biases to 0."""
        word_embedding_weights = np.zeros((vocab_size, embedding_dim))
        embed_to_hid_weights = np.zeros((num_hid, context_len * embedding_dim))
        hid_to_output_weights = np.zeros((vocab_size, num_hid))
        hid_bias = np.zeros(num_hid)
        output_bias = np.zeros(vocab_size)
        return cls(word_embedding_weights, embed_to_hid_weights, hid_to_output_weights,
                   hid_bias, output_bias)

    def random_init(cls, init_wt, vocab_size, context_len, embedding_dim, num_hid):
        """A constructor which initializes weights to small random values and biases to 0."""
        word_embedding_weights = np.random.normal(0., init_wt, size=(vocab_size, embedding_dim))
        embed_to_hid_weights = np.random.normal(0., init_wt, size=(num_hid, context_len * embedding_dim))
        hid_to_output_weights = np.random.normal(0., init_wt, size=(vocab_size, num_hid))
        hid_bias = np.zeros(num_hid)
        output_bias = np.zeros(vocab_size)
        return cls(word_embedding_weights, embed_to_hid_weights, hid_to_output_weights,
                   hid_bias, output_bias)

    def __mul__(self, a):
        return self.__class__(a * self.word_embedding_weights,
                              a * self.embed_to_hid_weights,
                              a * self.hid_to_output_weights,
                              a * self.hid_bias,
                              a * self.output_bias)

    def __rmul__(self, a):
        return self * a

    def __add__(self, other):
        return self.__class__(self.word_embedding_weights + other.word_embedding_weights,
                              self.embed_to_hid_weights + other.embed_to_hid_weights,
                              self.hid_to_output_weights + other.hid_to_output_weights,
                              self.hid_bias + other.hid_bias,
                              self.output_bias + other.output_bias)

    def __sub__(self, other):
        return self + -1. * other

class Activations(object):
    """A class representing the activations of the units in the network. This class has three fields:

        embedding_layer, a matrix of B x 3D matrix (where B is the batch size and D is the embedding dimension),
                representing the activations for the embedding layer on all the cases in a batch. The first D
                columns represent the embeddings for the first context word, and so on.
        hidden_layer, a B x N_H matrix representing the hidden layer activations for a batch
        output_layer, a B x N_V matrix representing the output layer activations for a batch"""

    def __init__(self, embedding_layer, hidden_layer, output_layer):
        self.embedding_layer = embedding_layer
        self.hidden_layer = hidden_layer
        self.output_layer = output_layer

def get_batches(inputs, targets, batch_size, shuffle=True):
    """Divide a dataset (usually the training set) into mini-batches of a given size.

    if inputs.shape[0] % batch_size != 0:
        raise RuntimeError('The number of data points must be a multiple of the batch size.')
    num_batches = inputs.shape[0] // batch_size

    if shuffle:
        idxs = np.random.permutation(inputs.shape[0])
        inputs = inputs[idxs, :]
        targets = targets[idxs]

    for m in range(num_batches):
        yield inputs[m * batch_size:(m + 1) * batch_size, :], \
              targets[m * batch_size:(m + 1) * batch_size]

Now, we will implement a method which computes the gradient using backpropagation.

To start out, the Model class contains several important methods used in training:

  • compute_activations computes the activations of all units on a given input batch
  • compute_loss computes the total cross-entropy loss on a mini-batch
  • evaluate computes the average cross-entropy loss for a given set of inputs and targets

You will need to complete the implementation of two additional methods which are needed for training:

  • compute_loss_derivative computes the derivative of the loss function with respect to the output layer inputs.
  • back_propagate is the function which computes the gradient of the loss with respect to model parameters using backpropagation. It uses the derivatives computed by compute_loss_derivative. Some parts are already filled in for you, but you need to compute the matrices of derivatives for embed_to_hid_weights, hid_bias, hid_to_output_weights, and output_bias. These matrices have the same sizes as the parameter matrices (see previous section).
  class Model(object):
    """A class representing the language model itself. This class contains various methods used in training
    the model and visualizing the learned representations. It has two fields:

        params, a Params instance which contains the model parameters
        vocab, a list containing all the words in the dictionary; vocab[0] is the word with index
               0, and so on."""

    def __init__(self, params, vocab):
        self.params = params
        self.vocab = vocab

        self.vocab_size = len(vocab)
        self.embedding_dim = self.params.word_embedding_weights.shape[1]
        self.embedding_layer_dim = self.params.embed_to_hid_weights.shape[1]
        self.context_len = self.embedding_layer_dim // self.embedding_dim
        self.num_hid = self.params.embed_to_hid_weights.shape[0]

    def copy(self):
        return self.__class__(self.params.copy(), self.vocab[:])

    def random_init(cls, init_wt, vocab, context_len, embedding_dim, num_hid):
        """Constructor which randomly initializes the weights to Gaussians with standard deviation init_wt
        and initializes the biases to all zeros."""
        params = Params.random_init(init_wt, len(vocab), context_len, embedding_dim, num_hid)
        return Model(params, vocab)

    def indicator_matrix(self, targets):
        """Construct a matrix where the kth entry of row i is 1 if the target for example
        i is k, and all other entries are 0."""
        batch_size = targets.size
        expanded_targets = np.zeros((batch_size, len(self.vocab)))
        expanded_targets[np.arange(batch_size), targets] = 1.
        return expanded_targets

    def compute_loss_derivative(self, output_activations, expanded_target_batch):
        """Compute the derivative of the cross-entropy loss function with respect to the inputs
        to the output units. In particular, the output layer computes the softmax

            y_i = e^{z_i} / \sum_j e^{z_j}.

        This function should return a batch_size x vocab_size matrix, where the (i, j) entry
        is dC / dz_j computed for the ith training case, where C is the loss function

            C = -sum(t_i log y_i).

        The arguments are as follows:

            output_activations - the activations of the output layer, i.e. the y_i's.
            expanded_target_batch - each row is the indicator vector for a target word,
                i.e. the (i, j) entry is 1 if the i'th word is j, and 0 otherwise."""

        return output_activations-expanded_target_batch

    def compute_loss(self, output_activations, expanded_target_batch):
        """Compute the total loss over a mini-batch. expanded_target_batch is the matrix obtained
        by calling indicator_matrix on the targets for the batch."""
        return -np.sum(expanded_target_batch * np.log(output_activations + TINY))

    def compute_activations(self, inputs):
        """Compute the activations on a batch given the inputs. Returns an Activations instance.
        You should try to read and understand this function, since this will give you clues for
        how to implement back_propagate."""

        batch_size = inputs.shape[0]
        if inputs.shape[1] != self.context_len:
            raise RuntimeError('Dimension of the input vectors should be {}, but is instead {}'.format(
                self.context_len, inputs.shape[1]))

        # Embedding layer
        # Look up the input word indies in the word_embedding_weights matrix
        embedding_layer_state = np.zeros((batch_size, self.embedding_layer_dim))
        for i in range(self.context_len):
            embedding_layer_state[:, i * self.embedding_dim:(i + 1) * self.embedding_dim] = \
                self.params.word_embedding_weights[inputs[:, i], :]

        # Hidden layer
        inputs_to_hid =, self.params.embed_to_hid_weights.T) + \
        # Apply logistic activation function
        hidden_layer_state = 1. / (1. + np.exp(-inputs_to_hid))

        # Output layer
        inputs_to_softmax =, self.params.hid_to_output_weights.T) + \

        # Subtract maximum.
        # Remember that adding or subtracting the same constant from each input to a
        # softmax unit does not affect the outputs. So subtract the maximum to
        # make all inputs <= 0. This prevents overflows when computing their exponents.
        inputs_to_softmax -= inputs_to_softmax.max(1).reshape((-1, 1))

        output_layer_state = np.exp(inputs_to_softmax)
        output_layer_state /= output_layer_state.sum(1).reshape((-1, 1))

        return Activations(embedding_layer_state, hidden_layer_state, output_layer_state)

    def back_propagate(self, input_batch, activations, loss_derivative):
        """Compute the gradient of the loss function with respect to the trainable parameters
        of the model. The arguments are as follows:

             input_batch - the indices of the context words
             activations - an Activations class representing the output of Model.compute_activations
             loss_derivative - the matrix of derivatives computed by compute_loss_derivative

        # The matrix with values dC / dz_j, where dz_j is the input to the jth hidden unit,
        # i.e. y_j = 1 / (1 + e^{-z_j})
        hid_deriv =, self.params.hid_to_output_weights) \
                    * activations.hidden_layer * (1. - activations.hidden_layer)

        hid_to_output_weights_grad =, activations.hidden_layer)
        output_bias_grad =, np.ones((activations.hidden_layer.shape[0],1))).ravel()
        embed_to_hid_weights_grad =, activations.embedding_layer)
        hid_bias_grad =, np.ones((activations.hidden_layer.shape[0], 1))).ravel()

        # The matrix of derivatives for the embedding layer
        embed_deriv =, self.params.embed_to_hid_weights)

        # Embedding layer
        word_embedding_weights_grad = np.zeros((self.vocab_size, self.embedding_dim))
        for w in range(self.context_len):
            word_embedding_weights_grad +=[:, w]).T,
                                                  embed_deriv[:, w * self.embedding_dim:(w + 1) * self.embedding_dim])

        return Params(word_embedding_weights_grad, embed_to_hid_weights_grad, hid_to_output_weights_grad,
                      hid_bias_grad, output_bias_grad)

    def evaluate(self, inputs, targets, batch_size=100):
        """Compute the average cross-entropy over a dataset.

            inputs: matrix of shape D x N
            targets: one-dimensional matrix of length N"""

        ndata = inputs.shape[0]

        total = 0.
        for input_batch, target_batch in get_batches(inputs, targets, batch_size):
            activations = self.compute_activations(input_batch)
            expanded_target_batch = self.indicator_matrix(target_batch)
            cross_entropy = -np.sum(expanded_target_batch * np.log(activations.output_layer + TINY))
            total += cross_entropy

        return total / float(ndata)

    def display_nearest_words(self, word, k=10):
        """List the k words nearest to a given word, along with their distances."""

        if word not in self.vocab:
            print('Word "{}" not in vocabulary.'.format(word))

        # Compute distance to every other word.
        idx = self.vocab.index(word)
        word_rep = self.params.word_embedding_weights[idx, :]
        diff = self.params.word_embedding_weights - word_rep.reshape((1, -1))
        distance = np.sqrt(np.sum(diff ** 2, axis=1))

        # Sort by distance.
        order = np.argsort(distance)
        order = order[1:1 + k]  # The nearest word is the query word itself, skip that.
        for i in order:
            print('{}: {}'.format(self.vocab[i], distance[i]))

    def predict_next_word(self, word1, word2, word3, k=10):
        """List the top k predictions for the next word along with their probabilities.
            word1: The first word as a string.
            word2: The second word as a string.
            word3: The third word as a string.
            k: The k most probable predictions are shown.
        Example usage:
            model.predict_next_word('john', 'might', 'be', 3)
            model.predict_next_word('life', 'in', 'new', 3)"""

        if word1 not in self.vocab:
            raise RuntimeError('Word "{}" not in vocabulary.'.format(word1))
        if word2 not in self.vocab:
            raise RuntimeError('Word "{}" not in vocabulary.'.format(word2))
        if word3 not in self.vocab:
            raise RuntimeError('Word "{}" not in vocabulary.'.format(word3))

        idx1, idx2, idx3 = self.vocab.index(word1), self.vocab.index(word2), self.vocab.index(word3)
        input = np.array([idx1, idx2, idx3]).reshape((1, -1))
        activations = self.compute_activations(input)
        prob = activations.output_layer.ravel()
        idxs = np.argsort(prob)[::-1]  # sort descending
        for i in idxs[:k]:
            print('{} {} {} {} Prob: {:1.5f}'.format(word1, word2, word3, self.vocab[i], prob[i]))

    def word_distance(self, word1, word2):
        """Compute the distance between the vector representations of two words."""

        if word1 not in self.vocab:
            raise RuntimeError('Word "{}" not in vocabulary.'.format(word1))
        if word2 not in self.vocab:
            raise RuntimeError('Word "{}" not in vocabulary.'.format(word2))

        idx1, idx2 = self.vocab.index(word1), self.vocab.index(word2)
        word_rep1 = self.params.word_embedding_weights[idx1, :]
        word_rep2 = self.params.word_embedding_weights[idx2, :]
        diff = word_rep1 - word_rep2
        return np.sqrt(np.sum(diff ** 2))

Perform routine checking.check_gradients, which checks your gradients using finite differences.

def relative_error(a, b):
    return np.abs(a - b) / (np.abs(a) + np.abs(b))

def check_output_derivatives(model, input_batch, target_batch):
    def softmax(z):
        z = z.copy()
        z -= z.max(1).reshape((-1, 1))
        y = np.exp(z)
        y /= y.sum(1).reshape((-1, 1))
        return y

    batch_size = input_batch.shape[0]
    z = np.random.normal(size=(batch_size, model.vocab_size))
    y = softmax(z)

    expanded_target_batch = model.indicator_matrix(target_batch)
    loss_derivative = model.compute_loss_derivative(y, expanded_target_batch)

    if loss_derivative is None:
        print('Loss derivative not implemented yet.')
        return False

    if loss_derivative.shape != (batch_size, model.vocab_size):
        print('Loss derivative should be size {} but is actually {}.'.format(
            (batch_size, model.vocab_size), loss_derivative.shape))
        return False

    def obj(z):
        y = softmax(z)
        return model.compute_loss(y, expanded_target_batch)

    for count in range(1000):
        i, j = np.random.randint(0, loss_derivative.shape[0]), np.random.randint(0, loss_derivative.shape[1])

        z_plus = z.copy()
        z_plus[i, j] += EPS
        obj_plus = obj(z_plus)

        z_minus = z.copy()
        z_minus[i, j] -= EPS
        obj_minus = obj(z_minus)

        empirical = (obj_plus - obj_minus) / (2. * EPS)
        rel = relative_error(empirical, loss_derivative[i, j])
        if rel > 1e-4:
            print('The loss derivative has a relative error of {}, which is too large.'.format(rel))
            return False

    print('The loss derivative looks OK.')
    return True

def check_param_gradient(model, param_name, input_batch, target_batch):
    activations = model.compute_activations(input_batch)
    expanded_target_batch = model.indicator_matrix(target_batch)
    loss_derivative = model.compute_loss_derivative(activations.output_layer, expanded_target_batch)
    param_gradient = model.back_propagate(input_batch, activations, loss_derivative)

    def obj(model):
        activations = model.compute_activations(input_batch)
        return model.compute_loss(activations.output_layer, expanded_target_batch)

    dims = getattr(model.params, param_name).shape
    is_matrix = (len(dims) == 2)

    if getattr(param_gradient, param_name).shape != dims:
        print('The gradient for {} should be size {} but is actually {}.'.format(
            param_name, dims, getattr(param_gradient, param_name).shape))

    for count in range(1000):
        if is_matrix:
            slc = np.random.randint(0, dims[0]), np.random.randint(0, dims[1])
            slc = np.random.randint(dims[0])

        model_plus = model.copy()
        getattr(model_plus.params, param_name)[slc] += EPS
        obj_plus = obj(model_plus)

        model_minus = model.copy()
        getattr(model_minus.params, param_name)[slc] -= EPS
        obj_minus = obj(model_minus)

        empirical = (obj_plus - obj_minus) / (2. * EPS)
        exact = getattr(param_gradient, param_name)[slc]
        rel = relative_error(empirical, exact)
        if rel > 1e-4:
            print('The loss derivative has a relative error of {}, which is too large.'.format(rel))
            return False

    print('The gradient for {} looks OK.'.format(param_name))

def load_partially_trained_model():
    obj = pickle.load(open(PARTIALLY_TRAINED_MODEL, 'rb'))
    params = Params(obj['word_embedding_weights'], obj['embed_to_hid_weights'],
                                   obj['hid_to_output_weights'], obj['hid_bias'],
    vocab = obj['vocab']
    return Model(params, vocab)

def check_gradients():
    """Check the computed gradients using finite differences."""

    np.seterr(all='ignore')  # suppress a warning which is harmless

    model = load_partially_trained_model()
    data_obj = pickle.load(open(data_location, 'rb'))
    train_inputs, train_targets = data_obj['train_inputs'], data_obj['train_targets']
    input_batch = train_inputs[:100, :]
    target_batch = train_targets[:100]

    if not check_output_derivatives(model, input_batch, target_batch):

    for param_name in ['word_embedding_weights', 'embed_to_hid_weights', 'hid_to_output_weights',
                       'hid_bias', 'output_bias']:
        check_param_gradient(model, param_name, input_batch, target_batch)

def print_gradients():
    """Print out certain derivatives for grading."""

    model = load_partially_trained_model()
    data_obj = pickle.load(open(data_location, 'rb'))
    train_inputs, train_targets = data_obj['train_inputs'], data_obj['train_targets']
    input_batch = train_inputs[:100, :]
    target_batch = train_targets[:100]

    activations = model.compute_activations(input_batch)
    expanded_target_batch = model.indicator_matrix(target_batch)
    loss_derivative = model.compute_loss_derivative(activations.output_layer, expanded_target_batch)
    param_gradient = model.back_propagate(input_batch, activations, loss_derivative)

    print('loss_derivative[2, 5]', loss_derivative[2, 5])
    print('loss_derivative[2, 121]', loss_derivative[2, 121])
    print('loss_derivative[5, 33]', loss_derivative[5, 33])
    print('loss_derivative[5, 31]', loss_derivative[5, 31])
    print('param_gradient.word_embedding_weights[27, 2]', param_gradient.word_embedding_weights[27, 2])
    print('param_gradient.word_embedding_weights[43, 3]', param_gradient.word_embedding_weights[43, 3])
    print('param_gradient.word_embedding_weights[22, 4]', param_gradient.word_embedding_weights[22, 4])
    print('param_gradient.word_embedding_weights[2, 5]', param_gradient.word_embedding_weights[2, 5])
    print('param_gradient.embed_to_hid_weights[10, 2]', param_gradient.embed_to_hid_weights[10, 2])
    print('param_gradient.embed_to_hid_weights[15, 3]', param_gradient.embed_to_hid_weights[15, 3])
    print('param_gradient.embed_to_hid_weights[30, 9]', param_gradient.embed_to_hid_weights[30, 9])
    print('param_gradient.embed_to_hid_weights[35, 21]', param_gradient.embed_to_hid_weights[35, 21])
    print('param_gradient.hid_bias[10]', param_gradient.hid_bias[10])
    print('param_gradient.hid_bias[20]', param_gradient.hid_bias[20])
    print('param_gradient.output_bias[0]', param_gradient.output_bias[0])
    print('param_gradient.output_bias[1]', param_gradient.output_bias[1])
    print('param_gradient.output_bias[2]', param_gradient.output_bias[2])
    print('param_gradient.output_bias[3]', param_gradient.output_bias[3])

Print out the gradients

# check_gradients()
loss_derivative[2, 5] 0.001112231773782498
loss_derivative[2, 121] -0.9991004720395987
loss_derivative[5, 33] 0.0001903237803173703
loss_derivative[5, 31] -0.7999757709589483

param_gradient.word_embedding_weights[27, 2] -0.27199539981936866
param_gradient.word_embedding_weights[43, 3] 0.8641722267354154
param_gradient.word_embedding_weights[22, 4] -0.2546730202374648
param_gradient.word_embedding_weights[2, 5] 0.0

param_gradient.embed_to_hid_weights[10, 2] -0.6526990313918257
param_gradient.embed_to_hid_weights[15, 3] -0.13106433000472612
param_gradient.embed_to_hid_weights[30, 9] 0.11846774618169399
param_gradient.embed_to_hid_weights[35, 21] -0.10004526104604386

param_gradient.hid_bias[10] 0.25376638738156426
param_gradient.hid_bias[20] -0.03326739163635369

param_gradient.output_bias[0] -2.062759603217304
param_gradient.output_bias[1] 0.03902008573921689
param_gradient.output_bias[2] -0.7561537928318482
param_gradient.output_bias[3] 0.21235172051123635

Now we fininshed the implementation and down to the training The function train implements the main training procedure. It takes two arguments:

  • embedding_dim: The number of dimensions in the distributed representation.
  • num_hid: The number of hidden units

As the model trains, the script prints out some numbers that tell you how well the training is going.

It shows:

  • The cross entropy on the last 100 mini-batches of the training set. This is shown after every 100 mini-batches.
  • The cross entropy on the entire validation set every 1000 mini-batches of training.

At the end of training, this function shows the cross entropies on the training, validation and test sets. It will return a Model instance.

_train_inputs = None
_train_targets = None
_vocab = None

DEFAULT_TRAINING_CONFIG = {'batch_size': 100,  # the size of a mini-batch
                           'learning_rate': 0.1,  # the learning rate
                           'momentum': 0.9,  # the decay parameter for the momentum vector
                           'epochs': 50,  # the maximum number of epochs to run
                           'init_wt': 0.01,  # the standard deviation of the initial random weights
                           'context_len': 3,  # the number of context words used
                           'show_training_CE_after': 100,  # measure training error after this many mini-batches
                           'show_validation_CE_after': 1000,  # measure validation error after this many mini-batches

def find_occurrences(word1, word2, word3):
    """Lists all the words that followed a given tri-gram in the training set and the number of
    times each one followed it."""

    # cache the data so we don't keep reloading
    global _train_inputs, _train_targets, _vocab
    if _train_inputs is None:
        data_obj = pickle.load(open(data_location, 'rb'))
        _vocab = data_obj['vocab']
        _train_inputs, _train_targets = data_obj['train_inputs'], data_obj['train_targets']

    if word1 not in _vocab:
        raise RuntimeError('Word "{}" not in vocabulary.'.format(word1))
    if word2 not in _vocab:
        raise RuntimeError('Word "{}" not in vocabulary.'.format(word2))
    if word3 not in _vocab:
        raise RuntimeError('Word "{}" not in vocabulary.'.format(word3))

    idx1, idx2, idx3 = _vocab.index(word1), _vocab.index(word2), _vocab.index(word3)
    idxs = np.array([idx1, idx2, idx3])

    matches = np.all(_train_inputs == idxs.reshape((1, -1)), 1)

    if np.any(matches):
        counts = collections.defaultdict(int)
        for m in np.where(matches)[0]:
            counts[_vocab[_train_targets[m]]] += 1

        word_counts = sorted(list(counts.items()), key=lambda t: t[1], reverse=True)
        print('The tri-gram "{} {} {}" was followed by the following words in the training set:'.format(
            word1, word2, word3))
        for word, count in word_counts:
            if count > 1:
                print('    {} ({} times)'.format(word, count))
                print('    {} (1 time)'.format(word))
        print('The tri-gram "{} {} {}" did not occur in the training set.'.format(word1, word2, word3))

def train(embedding_dim, num_hid, config=DEFAULT_TRAINING_CONFIG):
    """This is the main training routine for the language model. It takes two parameters:

        embedding_dim, the dimension of the embeddilanguage_model.pyng space
        num_hid, the number of hidden units."""

    # Load the data
    data_obj = pickle.load(open(data_location, 'rb'))
    vocab = data_obj['vocab']
    train_inputs, train_targets = data_obj['train_inputs'], data_obj['train_targets']
    valid_inputs, valid_targets = data_obj['valid_inputs'], data_obj['valid_targets']
    test_inputs, test_targets = data_obj['test_inputs'], data_obj['test_targets']

    # Randomly initialize the trainable parameters
    model = Model.random_init(config['init_wt'], vocab, config['context_len'], embedding_dim, num_hid)

    # Variables used for early stopping
    best_valid_CE = np.infty
    end_training = False

    # Initialize the momentum vector to all zeros
    delta = Params.zeros(len(vocab), config['context_len'], embedding_dim, num_hid)

    this_chunk_CE = 0.
    batch_count = 0
    for epoch in range(1, config['epochs'] + 1):
        if end_training:

        print('Epoch', epoch)

        for m, (input_batch, target_batch) in enumerate(get_batches(train_inputs, train_targets,
            batch_count += 1

            # Forward propagate
            activations = model.compute_activations(input_batch)

            # Compute loss derivative
            expanded_target_batch = model.indicator_matrix(target_batch)
            loss_derivative = model.compute_loss_derivative(activations.output_layer, expanded_target_batch)
            loss_derivative /= config['batch_size']

            # Measure loss function
            cross_entropy = model.compute_loss(activations.output_layer, expanded_target_batch) / config['batch_size']
            this_chunk_CE += cross_entropy
            if batch_count % config['show_training_CE_after'] == 0:
                print('Batch {} Train CE {:1.3f}'.format(
                    batch_count, this_chunk_CE / config['show_training_CE_after']))
                this_chunk_CE = 0.

            # Backpropagate
            loss_gradient = model.back_propagate(input_batch, activations, loss_derivative)

            # Update the momentum vector and model parameters
            delta = config['momentum'] * delta + loss_gradient
            model.params -= config['learning_rate'] * delta

            # Validate
            if batch_count % config['show_validation_CE_after'] == 0:
                print('Running validation...')
                cross_entropy = model.evaluate(valid_inputs, valid_targets)
                print('Validation cross-entropy: {:1.3f}'.format(cross_entropy))

                if cross_entropy > best_valid_CE:
                    print('Validation error increasing!  Training stopped.')
                    end_training = True

                best_valid_CE = cross_entropy

    train_CE = model.evaluate(train_inputs, train_targets)
    print('Final training cross-entropy: {:1.3f}'.format(train_CE))
    valid_CE = model.evaluate(valid_inputs, valid_targets)
    print('Final validation cross-entropy: {:1.3f}'.format(valid_CE))
    test_CE = model.evaluate(test_inputs, test_targets)
    print('Final test cross-entropy: {:1.3f}'.format(test_CE))

    return model

Run the training.

embedding_dim = 16
num_hid = 128
trained_model = train(embedding_dim, num_hid)
Epoch 1
Batch 100 Train CE 4.543
Batch 200 Train CE 4.430
Batch 300 Train CE 4.456
Batch 400 Train CE 4.460
Batch 500 Train CE 4.446
Batch 600 Train CE 4.423
Batch 700 Train CE 4.445
Batch 800 Train CE 4.440
Batch 900 Train CE 4.415
Batch 1000 Train CE 4.392
Running validation...
Validation cross-entropy: 4.435


Epoch 6
Batch 18700 Train CE 2.751
Batch 18800 Train CE 2.731
Batch 18900 Train CE 2.717
Batch 19000 Train CE 2.727
Running validation...
Validation cross-entropy: 2.772
Validation error increasing!  Training stopped.

Final training cross-entropy: 2.729
Final validation cross-entropy: 2.772
Final test cross-entropy: 2.775

Part 4: Analysis

Trainings & Embeddings

In this part, we first train a model with a 16-dimensional embedding and 128 hidden units, as discussed in the previous section;

we will use this trained model for the remainder of this section.

These methods of the Model class can be used for analyzing the model after the training is done:

  • display_nearest_words lists the words whose embedding vectors are nearest to the given word
  • word_distance computes the distance between the embeddings of two words
  • predict_next_word shows the possible next words the model considers most likely, along with their probabilities

We also include:

  • tsne_plot_representation creates a 2-dimensional embedding of the distributed representation space using an algorithm called t-SNE. Nearby points in this 2-D space are meant to correspond to nearby points in the 16-D space.
import numpy as Math

def Hbeta(D=Math.array([]), beta=1.0):
    """Compute the perplexity and the P-row for a specific value of the precision of a Gaussian distribution."""

    # Compute P-row and corresponding perplexity
    P = Math.exp(-D.copy() * beta);
    sumP = sum(P);
    H = Math.log(sumP) + beta * Math.sum(D * P) / sumP;
    P = P / sumP;
    return H, P;

def x2p(X=Math.array([]), tol=1e-5, perplexity=30.0):
    """Performs a binary search to get P-values in such a way that each conditional Gaussian has the same perplexity."""

    # Initialize some variables
    print("Computing pairwise distances...")
    (n, d) = X.shape;
    sum_X = Math.sum(Math.square(X), 1);
    D = Math.add(Math.add(-2 *, X.T), sum_X).T, sum_X);
    P = Math.zeros((n, n));
    beta = Math.ones((n, 1));
    logU = Math.log(perplexity);

    # Loop over all datapoints
    for i in range(n):

        # Print progress
        if i % 500 == 0:
            print("Computing P-values for point ", i, " of ", n, "...")

        # Compute the Gaussian kernel and entropy for the current precision
        betamin = -Math.inf;
        betamax = Math.inf;
        Di = D[i, Math.concatenate((Math.r_[0:i], Math.r_[i + 1:n]))];
        (H, thisP) = Hbeta(Di, beta[i]);

        # Evaluate whether the perplexity is within tolerance
        Hdiff = H - logU;
        tries = 0;
        while Math.abs(Hdiff) > tol and tries < 50:

            # If not, increase or decrease precision
            if Hdiff > 0:
                betamin = beta[i];
                if betamax == Math.inf or betamax == -Math.inf:
                    beta[i] = beta[i] * 2;
                    beta[i] = (beta[i] + betamax) / 2;
                betamax = beta[i];
                if betamin == Math.inf or betamin == -Math.inf:
                    beta[i] = beta[i] / 2;
                    beta[i] = (beta[i] + betamin) / 2;

            # Recompute the values
            (H, thisP) = Hbeta(Di, beta[i]);
            Hdiff = H - logU;
            tries = tries + 1;

        # Set the final row of P
        P[i, Math.concatenate((Math.r_[0:i], Math.r_[i + 1:n]))] = thisP;

    # Return final P-matrix
    print("Mean value of sigma: ", Math.mean(Math.sqrt(1 / beta)))
    return P;

def pca(X=Math.array([]), no_dims=50):
    """Runs PCA on the NxD array X in order to reduce its dimensionality to no_dims dimensions."""

    print("Preprocessing the data using PCA...")
    (n, d) = X.shape;
    X = X - Math.tile(Math.mean(X, 0), (n, 1));
    (l, M) = Math.linalg.eig(, X));
    Y =, M[:, 0:no_dims]);
    return Y;

def tsne(X=Math.array([]), no_dims=2, initial_dims=50, perplexity=30.0):
    """Runs t-SNE on the dataset in the NxD array X to reduce its dimensionality to no_dims dimensions.
    The syntaxis of the function is Y = tsne.tsne(X, no_dims, perplexity), where X is an NxD NumPy array."""

    # Check inputs
    if X.dtype != "float64":
        print("Error: array X should have type float64.");
        return -1;
    # if no_dims.__class__ != "<type 'int'>":			# doesn't work yet!
    #	print("Error: number of dimensions should be an integer.");
    #	return -1;

    # Initialize variables
    X = pca(X, initial_dims);
    (n, d) = X.shape;
    max_iter = 1000;
    initial_momentum = 0.5;
    final_momentum = 0.8;
    eta = 500;
    min_gain = 0.01;
    Y = Math.random.randn(n, no_dims);
    dY = Math.zeros((n, no_dims));
    iY = Math.zeros((n, no_dims));
    gains = Math.ones((n, no_dims));

    # Compute P-values
    P = x2p(X, 1e-5, perplexity);
    P = P + Math.transpose(P);
    P = P / Math.sum(P);
    P = P * 4;  # early exaggeration
    P = Math.maximum(P, 1e-12);

    # Run iterations
    for iter in range(max_iter):

        # Compute pairwise affinities
        sum_Y = Math.sum(Math.square(Y), 1);
        num = 1 / (1 + Math.add(Math.add(-2 *, Y.T), sum_Y).T, sum_Y));
        num[range(n), range(n)] = 0;
        Q = num / Math.sum(num);
        Q = Math.maximum(Q, 1e-12);

        # Compute gradient
        PQ = P - Q;
        for i in range(n):
            dY[i, :] = Math.sum(Math.tile(PQ[:, i] * num[:, i], (no_dims, 1)).T * (Y[i, :] - Y), 0);

        # Perform the update
        if iter < 20:
            momentum = initial_momentum
            momentum = final_momentum
        gains = (gains + 0.2) * ((dY > 0) != (iY > 0)) + (gains * 0.8) * ((dY > 0) == (iY > 0));
        gains[gains < min_gain] = min_gain;
        iY = momentum * iY - eta * (gains * dY);
        Y = Y + iY;
        Y = Y - Math.tile(Math.mean(Y, 0), (n, 1));

        # Compute current value of cost function
        if (iter + 1) % 10 == 0:
            C = Math.sum(P * Math.log(P / Q));
            print("Iteration ", (iter + 1), ": error is ", C)

        # Stop lying about P-values
        if iter == 100:
            P = P / 4;

    # Return solution
    return Y;

def tsne_plot_representation(model):
    """Plot a 2-D visualization of the learned representations using t-SNE."""
    mapped_X = tsne(model.params.word_embedding_weights)
    pylab.figure(figsize=(12, 9))
    for i, w in enumerate(model.vocab):
        pylab.text(mapped_X[i, 0], mapped_X[i, 1], w)
    pylab.xlim(mapped_X[:, 0].min(), mapped_X[:, 0].max())
    pylab.ylim(mapped_X[:, 1].min(), mapped_X[:, 1].max())

def tsne_plot_GLoVE_representation(W_final, b_final):
    """Plot a 2-D visualization of the learned representations using t-SNE."""
    mapped_X = tsne(W_final)
    pylab.figure(figsize=(12, 9))
    data_obj = pickle.load(open(data_location, 'rb'))
    for i, w in enumerate(data_obj['vocab']):
        pylab.text(mapped_X[i, 0], mapped_X[i, 1], w)
    pylab.xlim(mapped_X[:, 0].min(), mapped_X[:, 0].max())
    pylab.ylim(mapped_X[:, 1].min(), mapped_X[:, 1].max())

def plot_2d_GLoVE_representation(W_final, b_final):
    """Plot a 2-D visualization of the learned representations."""
    mapped_X = W_final
    pylab.figure(figsize=(12, 9))
    data_obj = pickle.load(open(data_location, 'rb'))
    for i, w in enumerate(data_obj['vocab']):
        pylab.text(mapped_X[i, 0], mapped_X[i, 1], w)
    pylab.xlim(mapped_X[:, 0].min(), mapped_X[:, 0].max())
    pylab.ylim(mapped_X[:, 1].min(), mapped_X[:, 1].max())

Let’s Play Around

Now we have our model here, let’s see how it behaves in practice!

trained_model.predict_next_word("she", "is", "a")
find_occurrences("she", "is", "a")
she is a good Prob: 0.24169
she is a part Prob: 0.07294
she is a very Prob: 0.06827
she is a family Prob: 0.06452
she is a man Prob: 0.05457
she is a big Prob: 0.05185
she is a home Prob: 0.03534
she is a team Prob: 0.03325
she is a long Prob: 0.02967
she is a new Prob: 0.02846
The tri-gram "she is a" was followed by the following words in the training set:
    year (1 time)
    big (1 time)
    good (1 time)
  • Test on “she is a”, with “part”, “family” being very plausible 4 th word in that context, while not appearing in the dataset

  • Compared with the second tsne GLoVE plot which has embedded dimension of 256, the first tsne plot has 250 embedded dimension, converges with similar 110 iterations, and has smaller error of about 0.633. The first model trains much slower than GLoVE model, while this tsne graph has more distinguishable clusters, which classifies better than GLoVE model. Some example cluster of words in the first model are: (“has”, “have”, “had”), (“might”, “will”, “would”, “should”, “can”, “may”, “could”), while the second model using GLoVE groups “see” and “say” with “would”, “have” with “called”. The first model with neural net has more precise and clear classification result than the second model.

    For the third plot, it plots the 2d GLoVE model without tsne, while the fourth plot represents the same model with tsne, converges around 110 iteration with about 0.304 error. Compared with the third plot, the ourth plot with tsne has better clustering, while the third plot is more dispersed and distracted.

(250, 16)
Preprocessing the data using PCA...
Computing pairwise distances...
Computing P-values for point  0  of  250 ...
Mean value of sigma:  1.0810407640085655
Iteration  10 : error is  13.904534336872707
Iteration  20 : error is  13.649706568082056
Iteration  100 : error is  14.403181497189895
Iteration  110 : error is  1.7822399276522192
Iteration  120 : error is  1.3267981771003918
Iteration  130 : error is  1.0994066587715356
Iteration  140 : error is  0.9820645964050435
Iteration  150 : error is  0.8893447064598181
Iteration  160 : error is  0.8501889895414241
Iteration  170 : error is  0.7967478450970407
Iteration  910 : error is  0.6337974076653053
Iteration  920 : error is  0.6337973914251336
Iteration  930 : error is  0.6337973768656655
Iteration  940 : error is  0.6337973643741731
Iteration  950 : error is  0.6337973537264191
Iteration  960 : error is  0.6337973444924512
Iteration  970 : error is  0.6337973367838915
Iteration  980 : error is  0.6337973301302106
Iteration  990 : error is  0.6337973244539044
Iteration  1000 : error is  0.6337973197804718


tsne_plot_GLoVE_representation(W_final, b_final)
Preprocessing the data using PCA...
Computing pairwise distances...
Computing P-values for point  0  of  250 ...

/usr/local/lib/python3.6/dist-packages/ ComplexWarning: Casting complex values to real discards the imaginary part

Mean value of sigma:  0.7414499269370552
Iteration  10 : error is  17.112574187692882
Iteration  100 : error is  17.276058343878077
Iteration  110 : error is  2.445460592164862
Iteration  120 : error is  1.7831495425683555
Iteration  130 : error is  1.514816424123761
Iteration  140 : error is  1.408098339641312
Iteration  990 : error is  1.1293082792737912
Iteration  1000 : error is  1.1293001798713274


plot_2d_GLoVE_representation(W_final_2d, b_final_2d)


tsne_plot_GLoVE_representation(W_final_2d, b_final_2d)
Preprocessing the data using PCA...
Computing pairwise distances...
Computing P-values for point  0  of  250 ...
Mean value of sigma:  0.23915790027001896
Iteration  10 : error is  12.573822139331822
Iteration  20 : error is  10.52761808422471
Iteration  100 : error is  10.378539324143624
Iteration  110 : error is  0.8264687306050834
Iteration  120 : error is  0.5160751571468272
Iteration  130 : error is  0.4342438805667565
Iteration  140 : error is  0.39366570658201167
Iteration  980 : error is  0.3044384562592181
Iteration  990 : error is  0.3044384217150869
Iteration  1000 : error is  0.3044383911881474


  • As we can see, “new” and “york” are not close together, which is also as expected, because if two words close together, this means that the similarity between two words are very large. While “new” and “york” have very different meanings and nature in language, and should not be categorized into the same group.
print(trained_model.word_distance("new", "york"))
old: 2.3600505246264323
white: 2.4383030498943365
back: 2.5759303604425643
american: 2.6466202075516416
such: 2.6599246057735786
own: 2.6878098904671783
political: 2.701124031526305
national: 2.7494668874744193
several: 2.7643670072967392
federal: 2.8155523660546993
public: 0.9048826050086719
music: 0.9105096376369168
university: 0.9309985183745197
city: 0.9640999583181336
department: 0.9810341593819769
ms.: 0.985606172603071
john: 1.0174907854088202
school: 1.035702631398839
general: 1.0443388378683818
team: 1.0641153998658832
  • Comparing (“goverment”, “political”) and (“government”, “university”), “government” is more close to “university” rather than “political”. This is plausible, because both of the government and university are noun in a sentance, while political is subjective.
print(trained_model.word_distance("government", "political"))
print(trained_model.word_distance("government", "university"))

Looks nice huh? Stay sharp! 😎