用numpy进行单变量回归的神经网络只能得到线性结果

Question

我的目标是创建一个具有单个隐藏层（具有ReLU激活）的神经网络，它能够近似于一个简单的单变量平方根函数。 我用numpy实现了网络，还做了梯度检查，一切似乎都很好，除了结果：由于某种原因，我只能得到线性近似值，像这样：noisy sqrt approx

尝试更改超参数，但没有成功。有什么想法吗？在

import numpy as np

step_size = 1e-6
input_size, output_size = 1, 1
h_size = 10
train_size = 500
x_train = np.abs(np.random.randn(train_size, 1) * 1000)
y_train = np.sqrt(x_train) + np.random.randn(train_size, 1) * 0.5

#initialize weights and biases
Wxh = np.random.randn(input_size, h_size) * 0.01
bh = np.zeros((1, h_size))
Why = np.random.randn(h_size, output_size) * 0.01
by = np.zeros((1, output_size))

for i in range(300000):
    #forward pass
    h = np.maximum(0, np.dot(x_train, Wxh) + bh1)
    y_est = np.dot(h, Why) + by

    loss = np.sum((y_est - y_train)**2) / train_size
    dy = 2 * (y_est - y_train) / train_size

    print("loss: ",loss)

    #backprop at output
    dWhy = np.dot(h.T, dy)
    dby = np.sum(dy, axis=0, keepdims=True)
    dh = np.dot(dy, Why.T)

    #backprop ReLU non-linearity    
    dh[h <= 0] = 0

    #backprop Wxh, and bh
    dWxh = np.dot(x_train.T, dh)
    dbh = np.sum(dh1, axis=0, keepdims=True)

    Wxh += -step_size * dWxh
    bh += -step_size * dbh
    Why += -step_size * dWhy
    by += -step_size * dby

编辑： 问题似乎是缺乏规范化和数据是非零中心的。在将这些转换应用于训练数据之后，我成功地获得了以下结果：noisy sqrt2

Answer 1

我可以让你的代码产生一种分段线性近似：

如果我调零中心并使输入和输出范围正常化：

# normalise range and domain
x_train -= x_train.mean()
x_train /= x_train.std()
y_train -= y_train.mean()
y_train /= y_train.std()

情节是这样产生的：

^{pr2}$