Context

In my previous post I implemented a nerual network to understand handwritten digits using matrix math functions I implemented myself.

Luckily, I don’t have to be doing this going forward in the future. The NumPy library will do this all for me. Numpy is the one of the foundational libraries in the Python scientific computing ecosystem. It provides a high-performance, multidimentional array object which makes it fast and easy to work with matrices.

In this post, I’ll show you how I use NumPy to replace the hand written math functions I wrote.

The Original Code Using Only Python

Below is the original code with my owm matrix multiplication functions.

def flatten_image(image):
    return list(itertools.chain.from_iterable(image))

def weighted_sum(a, b):
    assert(len(a) == len(b))
    output = 0
    for i in range(len(a)):
        output += (a[i] * b[i])
    return output

def vector_matrix_multiplication(a, b):
    output = [0 for i in range(10)]
    for i in range(len(output)):
        assert(len(a) == len(b[i]))
        output[i] = weighted_sum(a, b[i])
    return output

def zeros_matrix(rows, cols):
    output = []
    for r in range(rows):
        output.append([0 for col in range(cols)])
    return output

def outer_product(a, b):
    output = zeros_matrix(len(a), len(b))
    for i in range(len(a)):
        for j in range(len(b)):
            output[i][j] = a[i] * b[j]
    return output

class NeuralNet:
    def __init__(self):
        self.weights = [
            [0.0000 for i in range(784)],
            [0.0001 for i in range(784)],
            [0.0002 for i in range(784)],
            [0.0003 for i in range(784)],
            [0.0004 for i in range(784)],
            [0.0005 for i in range(784)],
            [0.0006 for i in range(784)],
            [0.0007 for i in range(784)],
            [0.0008 for i in range(784)],
            [0.0009 for i in range(784)]
        ]
        self.alpha = 0.0000001

    def predict(self, input):
        return vector_matrix_multiplication(input, self.weights)

    def train(self, input, labels, epochs):
        for i in range(epochs):
            for j in range(len(input)):
                pred = self.predict(input[j])

                label = labels[j]
                goal = [0 for k in range(10)]
                goal[label] = 1

                error = [0 for k in range(10)]
                delta = [0 for k in range(10)]

                for a in range(len(goal)):
                    delta[a] = pred[a] - goal[a]
                    error[a] = delta[a] ** 2

                weight_deltas = outer_product(delta, input[j])

                for x in range(len(self.weights)):
                    for y in range(len(self.weights[0])):
                        self.weights[x][y] -= (self.alpha * weight_deltas[x][y])

Implement The Helper Functions With NumPy

To help me better understand how NumPy slots into this code, I’m going to keep my helper functions but implement them using NumPy. For example, I still have a weighted sum function but instead of hand calculating the weighted sum, I use the NumPy dot function.

def flatten_image(image):
    return image.reshape(1, 28*28)

def weighted_sum(a, b):
    return a.dot(b)

def vector_matrix_multiplication(a, b):
    return np.matmul(input, weights.T)

def zeros_matrix(rows, cols):
    return np.zeros((rows, cols))

def outer_product(a, b):
    return np.outer(a, b)

class NeuralNet:
    def __init__(self):
        self.weights = np.random.random((10, 28 * 28)) * 0.0001
        self.alpha = 0.0000001

    def predict(self, input):
        return vector_matrix_multiplication(input, self.weights)

    def train(self, input, labels, epochs):
        for i in range(epochs):
            for j in range(len(input)):
                pred = self.predict(input[j])

                label = labels[j]
                goal = np.zeros(10)
                goal[label] = 1

                delta = pred - goal
                error = delta ** 2

                weight_deltas = outer_product(delta, input[j])

                self.weights -= (self.alpha * weight_deltas)

Use The NumPy Functions Inline

Already you can see the code is a lot cleaner.

If we remove those helper functions and do everything inline, the code shrinks even more. The original implmentation is 70 lines long and this one is only 26 lines.

The NumPy library is doing a lot of work for me.

def flatten_image(image):
    return image.reshape(1, 28*28)

class NeuralNet:
    def __init__(self):
        self.weights = np.random.random((10, 28 * 28)) * 0.0001
        self.alpha = 0.0000001

    def predict(self, input):
        return np.matmul(input, self.weights.T)

    def train(self, input, labels, epochs):
        for i in range(epochs):
            for j in range(len(input)):
                pred = self.predict(input[j])

                label = labels[j]
                goal = np.zeros(10)
                goal[label] = 1

                delta = pred - goal
                error = delta ** 2

                weight_deltas = np.outer(delta, input[j])

                self.weights -= (self.alpha * weight_deltas)

import itertools
import numpy as np

from keras.datasets import mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()

images = x_train
labels = y_train

prepared_images = [flatten_image(image) for image in images]
prepared_labels = np.array(labels)

nn = NeuralNet()
nn.train(prepared_images, prepared_labels, 5)

test_set = x_test
test_labels = y_test
num_correct = 0
for i in range(len(test_set)):
    prediction = nn.predict(flatten_image(test_set[i]))
    correct = test_labels[i]
    if np.argmax(prediction) == int(correct):
        num_correct += 1

print(str(num_correct/len(test_set) * 100) + "%")

76.05%

So What’s Next?

The code is now so efficient I can train on the full dataset and the accuracy gets a little bump.

In the next post we’ll see if adding a layer helps improve this accuracy even more.