Pyrotch BCELoss为相同的输入提供不同的输出

Question

我试图用pytorch的BCELoss函数计算二元分类问题的交叉熵损失。在修补时，我发现了这种奇怪的行为

from torch import nn
sigmoid = nn.Sigmoid()
loss = nn.BCELoss(reduction="sum")
target = torch.tensor([0., 1.])

input1 = torch.tensor([1., 1.], requires_grad=True)
input2 = sigmoid(torch.tensor([10., 10.], requires_grad=True))
print(input2) #tensor([1.0000, 1.0000], grad_fn=<SigmoidBackward>)

print(loss(input1, target)) #tensor(100., grad_fn=<BinaryCrossEntropyBackward>)
print(loss(input2, target)) #tensor(9.9996, grad_fn=<BinaryCrossEntropyBackward>)

既然input1和input2都有相同的值，它不应该返回相同的损耗值，而不是100和9.9996。正确的损耗值应该是100，因为我将log（0）~-无穷大相乘，在pytorch中上限为-100https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html

这里发生了什么？我哪里出了问题

Answer 1

sigmoid(10)不完全等于1：

>>> 1 / (1 + torch.exp(-torch.tensor(10.))).item()
0.9999545833234493

就你而言：

>>> sigmoid(torch.tensor([10., 10.], requires_grad=True)).tolist()
[0.9999545812606812, 0.9999545812606812]

因此input1与input2：[1.0, 1.0]与[0.9999545812606812, 0.9999545812606812]不同

让我们手动计算BCE：

def bce(x, y):
    return - (y * torch.log(x) + (1 - y) * torch.log(1 - x)).item()


# input1

x1 = torch.tensor(1.)
x2 = torch.tensor(1.)

y1 = torch.tensor(0.)
y2 = torch.tensor(1.)


print("input1:", sum([bce(x1, y1), bce(x2, y2)]))


# input2

x1 = torch.tensor(0.9999545812606812)
x2 = torch.tensor(0.9999545812606812)

y1 = torch.tensor(0.)
y2 = torch.tensor(1.)


print("input2:", sum([bce(x1, y1), bce(x2, y2)]))


input1: nan
input2: 9.999631525119185

对于input1我们得到nan，但根据文档：

Our solution is that BCELoss clamps its log function outputs to be greater than or equal to -100. This way, we can always have a finite loss value and a linear backward method.

这就是为什么我们在最后的pytorch的BCE输出中有100