We have previously discussed OR logic gates and the importance of bias units in AND gates. Here, we will introduce the XOR gate and show why logistic regression can’t model the non-linearity required for this particular problem.
As always, the full code for these examples can be found in my GitHub repository here.
XOR gates output True if either of the inputs are True, but not both. It acts like a more specific version of the OR gate:
| Input 1 | Input 2 | Output |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
If we visualize the data space we’ll have a clearer sense of what causes the issue. As you can see, there is no linear separator that can effectively split the categories:
When we try to model this using logistic regression like the previous gate examples, we run into a problem:
import numpy as np
import theano
import theano.tensor as T
# Set inputs and correct output values
inputs = [[0,0], [1,1], [0,1], [1,0]]
outputs = [0, 0, 1, 1]
# Set training parameters
alpha = 0.1 # Learning rate
training_iterations = 30000
# Define tensors
x = T.matrix("x")
y = T.vector("y")
b = theano.shared(value=1.0, name='b')
# Set random seed
rng = np.random.RandomState(2345)
# Initialize random weights
w_values = np.asarray(rng.uniform(low=-1, high=1, size=(2, 1)),
dtype=theano.config.floatX) # Force type to 32bit float for GPU
w = theano.shared(value=w_values, name='w', borrow=True)
# Theano symbolic expressions
hypothesis = T.nnet.sigmoid(T.dot(x, w) + b) # Sigmoid/logistic activation
hypothesis = T.flatten(hypothesis) # This needs to be flattened so
# hypothesis (matrix) and
# y (vector) have the same shape
cost = T.nnet.binary_crossentropy(hypothesis, y).mean() # Binary CE
updates_rules = [
(w, w - alpha * T.grad(cost, wrt=w)),
(b, b - alpha * T.grad(cost, wrt=b))
]
# Theano compiled functions
train = theano.function(inputs=[x, y], outputs=[hypothesis, cost],
updates=updates_rules)
predict = theano.function(inputs=[x], outputs=[hypothesis])
# Training
cost_history = []
for i in range(training_iterations):
h, cost = train(inputs, outputs)
cost_history.append(cost)
The network is unable to learn the correct weights due to the solution being non-linear. We can see this by looking at the training curve:
Introducing neural networks
One way to solve this problem is by adding non-linearity to the model with a hidden layer, thus turning this into a neural network model.
As always, we begin with imports and defining the data and corresponding labels:
import numpy as np import theano import theano.tensor as T # Set inputs and correct output values inputs = [[0,0], [1,1], [0,1], [1,0]] outputs = [0, 0, 1, 1]
Next, we define some training parameters. hidden_layer_nodes is used to control the number of units/neurons in the hidden layer. This will be useful when we set up the network architecture later:
alpha = 0.1 # Learning rate training_iterations = 50000 hidden_layer_nodes = 3
Then, define our tensors and weight arrays. Note the inclusion of b2 which is the bias for our hidden layer neurons and a second array of weights (w2_array) for the hidden layer to output layer connections.
# Define tensors
x = T.matrix("x")
y = T.vector("y")
b1 = theano.shared(value=1.0, name='b1')
b2 = theano.shared(value=1.0, name='b2')
# Set random seed
rng = np.random.RandomState(2345)
# Initialize weights
w1_array = np.asarray(rng.uniform(low=-1, high=1, size=(2, hidden_layer_nodes)),
dtype=theano.config.floatX) # Force type to 32bit float for GPU
w1 = theano.shared(value=w1_array, name='w1', borrow=True)
w2_array = np.asarray(rng.uniform(low=-1, high=1, size=(hidden_layer_nodes, 1)),
dtype=theano.config.floatX) # Force type to 32bit float for GPU
w2 = theano.shared(value=w2_array, name='w2', borrow=True)
We need some additional expressions to evaluate the output values of both the input (a1) and hidden (a2) layers:
a1 = T.nnet.sigmoid(T.dot(x, w1) + b1) # Input -> Hidden
a2 = T.nnet.sigmoid(T.dot(a1, w2) + b2) # Hidden -> Output
hypothesis = T.flatten(a2) # This needs to be flattened so
# hypothesis (matrix) and
# y (vector) have the same shape
cost = T.nnet.binary_crossentropy(hypothesis, y).mean() # Binary CE
Remember to add update rules for the additional weights (w2) and biases (b2) that we added:
updates_rules = [
(w1, w1 - alpha * T.grad(cost, wrt=w1)),
(w2, w2 - alpha * T.grad(cost, wrt=w2)),
(b1, b1 - alpha * T.grad(cost, wrt=b1)),
(b2, b2 - alpha * T.grad(cost, wrt=b2))
]
We finish off with some Theano functions and the actual training process:
train = theano.function(inputs=[x, y], outputs=[hypothesis, cost],
updates=updates_rules)
predict = theano.function(inputs=[x], outputs=[hypothesis])
# Training
cost_history = []
for i in range(training_iterations):
h, cost = train(inputs, outputs)
cost_history.append(cost)
The full code can be found here.
After training this neural network we can see that the cost correctly decreases over training iterations and outputs our correct predictions for the XOR gate:
