下面的代码(取自here)似乎只实现了一个简单的Dropout
,既不是DropPath
也不是DropConnect
。这是真的吗
def drop_path(x, drop_prob: float = 0., training: bool = False):
"""Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).
This is the same as the DropConnect impl I created for EfficientNet, etc networks, however,
the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper...
See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for
changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use
'survival rate' as the argument.
"""
if drop_prob == 0. or not training:
return x
keep_prob = 1 - drop_prob
shape = (x.shape[0],) + (1,) * (x.ndim - 1) # work with diff dim tensors, not just 2D ConvNets
random_tensor = keep_prob + torch.rand(shape, dtype=x.dtype, device=x.device)
random_tensor.floor_() # binarize
output = x.div(keep_prob) * random_tensor
return output
不,它与^{} 不同:
让我们比较输出(为了可读性,我删除了通道之间的换行符):
正如你所看到的,它们是不同的
DropPath
是从批次中滴下整个样品,当在其{a2}的等式2中使用时,有效地导致随机深度。另一方面,Dropout
正在按预期(从docs)删除随机值:还请注意,这两种方法都基于概率来缩放输出值,即,对于相同的
p
,未调零的元素是相同的相关问题 更多 >
编程相关推荐