在Tensorflow中，分类交叉熵函数是如何工作的？

Question

我想手动实现Tensorflow中的分类交叉熵函数。我做到了：

def my_CE(y_true, y_pred):
    log_y_pred = tf.math.log(y_pred)
    element_wise = -tf.math.multiply_no_nan(x=log_y_pred, y=y_true)
    return tf.reduce_mean(tf.reduce_sum(element_wise,axis=1))

通过测试，我意识到我做了正确的事情：

true = np.array([[0.0, 1.0], [1.0, 0.0]])
pred = np.array([[0.01, 0.99], [0.01, 0.99]])

print('Tensorflow CE : ',tf.keras.losses.CategoricalCrossentropy()(true, pred))
print('My CE : ',my_CE(true, pred))



*Tensorflow CE :  tf.Tensor(2.307610273361206, shape=(), dtype=float64)
My CE :  tf.Tensor(2.307610260920796, shape=(), dtype=float64)*

但在另一次测试中，我发现答案不同：

y_true = tf.constant([[0 , 0 , 0 , 0 , 1.0]])
y_pred = tf.constant([[0 , 0 , 0 ,0 ,  .3]])

print('Tensorflow CE : ',tf.keras.losses.CategoricalCrossentropy()(y_true, y_pred).numpy())
print('My CE : ',my_CE(y_true, y_pred).numpy())


Tensorflow CE :  1.192093e-07
My CE :  1.2039728

更糟糕的是，我意识到张量流函数在我看来不起作用！ 也就是说，当我增加类是正确的概率时，不小于tensorflow损失函数的值

for i in np.arange(.1,1,0.1):
    y_true = tf.constant([[0 , 0 , 0 , 0 , 1.0]])
    y_pred = tf.constant([[0 , 0 , 0 ,0 ,  i]])
    print('Tensorflow CE : ',tf.keras.losses.CategoricalCrossentropy()(y_true, y_pred).numpy())

Tensorflow CE :  1.192093e-07
Tensorflow CE :  1.192093e-07
Tensorflow CE :  1.192093e-07
Tensorflow CE :  1.192093e-07
Tensorflow CE :  1.192093e-07
Tensorflow CE :  1.192093e-07
Tensorflow CE :  1.192093e-07
Tensorflow CE :  1.192093e-07
Tensorflow CE :  1.192093e-07

Answer 1

根据评论

You're passing a y_pred that does not sum to 1. You need to either softmax your y_pred or set from_logits to True. For more information you can read here. (paraphrased from gobrewers14)

import tensorflow as tf
import numpy as np

def my_CE(y_true, y_pred):
    log_y_pred = tf.math.log(y_pred)
    element_wise = -tf.math.multiply_no_nan(x=log_y_pred, y=y_true)
    return tf.reduce_mean(tf.reduce_sum(element_wise,axis=1))

# y_pred sum to 1
y_true = tf.constant([[0 , 0 , 0 , 0 , 1.0]])
y_pred = tf.constant([[0 , 0.7 , 0 ,0 ,  .3]])

print('Tensorflow CE : ',tf.keras.losses.CategoricalCrossentropy()(y_true, y_pred).numpy())
print('My CE : ',my_CE(y_true, y_pred).numpy())

输出：

Tensorflow CE :  1.2039728
My CE :  1.2039728

#softmax y_pred 
y_true = tf.constant([[0 , 0 , 0 , 0 , 1.0]])
y_pred = tf.nn.softmax(tf.constant([[0 , 0.0 , 0 ,0 ,  .3]]))

print('Tensorflow CE : ',tf.keras.losses.CategoricalCrossentropy()(y_true, y_pred).numpy())
print('My CE : ',my_CE(y_true, y_pred).numpy())

输出：

Tensorflow CE :  1.3770702
My CE :  1.3770702