Non Linear Regression Example with Keras and Tensorflow Backend

Out of curiosity I wanted to try Keras to do non linear fitting. The simplicity of Keras made it possible to quickly try out some neural network model without deep knowledge of Tensorflow.

The data for fitting was generated using a non linear continuous function. It has five inputs and one output. Both the training set and validation set have around 1000 data points.

Y = SIN(A) x EXP(B) + COS(C x C) + POWER(D,5) – TANH(E)

I realized that adding too many hidden layers worsened the fit. Looks like for continuous functions, one hidden layer with sufficient number of nodes and good choice of activation function is sufficient. I chose hyperbolic tangent (tanh) for activation function and adam for optimizer. The results were pretty good but required some good number of iterations.

I plan to compare this with other regression algorithms available in Azure Machine Learning.

Complete code available on Github –

from keras.models import Sequential
from keras.layers import Dense
from sklearn.metrics import r2_score
import matplotlib.pyplot as plt
import numpy
%matplotlib inline

#Red data from csv file for training and validation data
TrainingSet = numpy.genfromtxt("training.csv", delimiter=",", skip_header=True)
ValidationSet = numpy.genfromtxt("validation.csv", delimiter=",", skip_header=True)

# split into input (X) and output (Y) variables
X1 = TrainingSet[:,0:5]
Y1 = TrainingSet[:,5]

X2 = ValidationSet[:,0:5]
Y2 = ValidationSet[:,5]

# create model
model = Sequential()
model.add(Dense(20, input_dim=5, init='uniform', activation='tanh'))
model.add(Dense(1, init='uniform', activation='linear'))

# Compile model
model.compile(loss='mse', optimizer='adam', metrics=['accuracy'])

# Fit the model, Y1, nb_epoch=5000, batch_size=10,  verbose=2)

# Calculate predictions
PredTestSet = model.predict(X1)
PredValSet = model.predict(X2)

# Save predictions
numpy.savetxt("trainresults.csv", PredTestSet, delimiter=",")
numpy.savetxt("valresults.csv", PredValSet, delimiter=",")
Main Code

The results were pretty good