批量梯度下降中的成本函数发散

Question

我正在尝试用python实现梯度下降方法。我希望在abs(J-J_new)达到某个公差水平（即收敛）时停止计算，其中J是成本函数。经过一定次数的迭代后，计算也将停止。我尝试过几种不同的实现，在我所有的尝试中，成本函数实际上是不同的（即|J-J_new| -> inf）。这对我来说没有什么意义，我也无法确定为什么它会从我的代码中这样做。我正在用4个无关紧要的数据点测试实现。我现在已经把它注释掉了，但是x和y最终将从包含400多个数据点的文本文件中读取。下面是我能想到的最简单的实现：

# import necessary packages
import numpy as np
import matplotlib.pyplot as plt

'''
For right now, I will hard code all parameters. After all code is written and I know that I implemented the 
algorithm correctly, I will consense the code into a single function.
'''

# Trivial data set to test
x = np.array([1, 3, 6, 8])
y = np.array([3, 5, 6, 5])

# Define parameter values
alpha = 0.1
tol = 1e-06
m = y.size
imax = 100000

# Define initial values
theta_0 = np.array([0.0])   # theta_0 guess
theta_1 = np.array([0.0])   # theta_1 guess
J = sum([(theta_0 - theta_1 * x[i] - y[i])**2 for i in range(m)])

# Begin gradient descent algorithm
converged = False
inum = 0
while not converged:
    grad_0 = (1/m) * sum([(theta_0 + theta_1 * x[i] - y[i]) for i in range(m)])
    grad_1 = (1/m) * sum([(theta_0 + theta_1 * x[i] - y[i]) * x[i] for i in range(m)])
    temp_0 = theta_0 - alpha * grad_0
    temp_1 = theta_1 - alpha * grad_1
    theta_0 = temp_0
    theta_1 = temp_1
    J_new = sum([(theta_0 + theta_1 * x[i] - y[i])**2 for i in range(m)])
    if abs(J - J_new) <= tol:
        print('Converged at iteration', inum)
        converged = True
    J = J_new
    inum = inum + 1
    if inum == imax:
        print('Maximum number of iterations reached!')
        converged = True

Answer 1

我用它做了更多的实验。之所以出现这种分歧，是因为学习率太高。当我改变检查收敛性的方式时，这也很有帮助。我没有使用abs(J - J_new)来检查收敛性，而是使用abs(theta0_new - theta_0)和abs(theta1_new - theta_1)。如果这两者都在某个公差范围内，则它已收敛。我还重新缩放（标准化）了数据，这似乎也有帮助。代码如下：

# import necessary packages
import numpy as np
import matplotlib.pyplot as plt

# Gradint descent function
def gradient_descent(x,y,alpha,tol,imax):
    # size of data set
    m = y.size
    
    # Define initial values
    theta_0 = np.array([0.0])   # theta_0 initial guess
    theta_1 = np.array([0.0])   # theta_1 initial guess
    
    # Begin gradient descent algorithm
    
    convergence = False
    inum = 0
    
    # While loop continues until convergence = True
    while not convergence:
        
        # Calculate gradients for theta_0 and theta_1
        grad_0 = (1/m) * sum([(theta_0 + theta_1 * x[i] - y[i]) for i in range(m)])
        grad_1 = (1/m) * sum([(theta_0 + theta_1 * x[i] - y[i]) * x[i] for i in range(m)])
        
        # Update theta_0 and theta_1
        temp0 = theta_0 - alpha * grad_0
        temp1 = theta_1 - alpha * grad_1
        theta0_new = temp0
        theta1_new = temp1
        
        # Check convergence, and stop loop if correct conditions are met
        if abs(theta0_new - theta_0) <= tol and abs(theta1_new - theta_1) <= tol:
            print('We have convergence at iteration', inum, '!')
            convergence = True
            
        # Update theta_0 and theta_1 for next iteration
        theta_0 = theta0_new
        theta_1 = theta1_new
        
        # Increment itertion counter
        inum = inum + 1
        
        # Check iteration number, and stop loop if inum == imax
        if inum == imax:
            print('Maximum number of iterations reached. We have convergence!')
            convrgence = True
            
    # Show result   
    print('Slope=', theta_1)
    print('Intercept=', theta_0)
    print('Iteration of convergece=', inum)


# Load data from text file
data = np.loadtxt('InputData.txt')

# Define data set
x = data[:,0]
y = data[:,1]

# Rescale the data
x = x/(max(x)-min(x))
y = y/(max(y)-min(y))

# Define input parameters
alpha = 1e-02
tol = 1e-05
imax = 10000

# Function call
gradient_descent(x, y, alpha, tol, imax)

我只检查了文本文件中的数据集