Linear Regression with Multiple Variables

Let’s recall the cost function J(θ)J(\theta), and ∂∂θJ(θ)\frac{\partial}{\partial\theta} J(\theta):

J(θ0,θ1)=12m∑i=1m(hθ(x(i))−y(i))2=12m∑i=1m(θ0+θ1x(i)−y(i))2J(\theta_{0}, \theta_{1}) = \frac{1}{2m} \sum_{i=1}^m (h_{\theta}(x^{(i)}) - y^{(i)})^2 = \frac{1}{2m} \sum_{i=1}^m (\theta_{0} + \theta_{1}x^{(i)} - y^{(i)})^2

If we define:

θ=(θ0θ1..θn)\theta = \left( \begin{array}{ccc} \theta_{0} \\ \theta_{1} \\ . \\ . \\ \theta_{n} \end{array} \right)

a single experiment xx:

x=(x0x1..xn)x = \left( \begin{array}{ccc} x_{0} \\ x_{1} \\ . \\ . \\ x_{n} \end{array} \right)

the input XX:

X=[x(0),x(1),...x(m)]X = [x^{(0)}, x^{(1)}, ... x^{(m)}]

the output YY:

Y=[y(0),y(1),...y(m)]Y = [y^{(0)}, y^{(1)}, ... y^{(m)}]

Therefore, we can get:

J(θ)=12m(θTX−Y)(θTX−Y)TJ(\theta) = \frac{1}{2m}(\theta^{T} X - Y)(\theta^{T} X - Y)^{T} ∂∂θJ(θ)=1m((θTX−Y)⋅XT)T\frac{\partial}{\partial\theta} J(\theta) = \frac{1}{m} ((\theta^{T}X - Y)\cdot X^{T})^{T}

the iteration will be:

θ=θ−αm((θTX−Y)⋅XT)T\theta = \theta -\frac{\alpha}{m} ((\theta^{T}X - Y)\cdot X^{T})^{T}

Let’s work on the ex1data2.txt with sklearn:

%pylab inline
Populating the interactive namespace from numpy and matplotlib

import numpy as np
import matplotlib.pyplot as plt

data = np.loadtxt(open("ex1data2.txt"), delimiter=",")
X = data[:, :2]
Y = data[:, 2]
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler  


normalizer = StandardScaler().fit(X)
clf = LinearRegression()
clf.fit(normalizer.transform(X), Y)

print 'intercept: ', clf.intercept_
print 'Coefficients: ', clf.coef_
intercept:  340412.659574
Coefficients:  [ 109447.79646964   -6578.35485416]

house = [1650, 3]
clf.predict(normalizer.transform(house))