Tensorflow的CyclicalLearningRate量表参数是什么意思？

Question

在tensorflow的^{}文档中，有一个名为scale_fn的参数

根据文件，scale_fn是

A function. Scheduling function applied in cycle

然而，这种解释并不十分清楚。 这个函数接受什么参数，应该输出什么？ 更具体地说，如何使用它？ 它是使用CyclicalLearningRate所必需的，但我找不到关于如何使用它的全面解释

Answer 1

你说得对documentation的用法不是很有启发性。一个更好的解释和一个例子可以在Super Convergence with Cyclical Learning Rates in TensorFlow中找到；引述：

from tensorflow_addons.optimizers import CyclicalLearningRate

cyclical_learning_rate = CyclicalLearningRate( 
  initial_learning_rate=3e-7,  maximal_learning_rate=3e-5, 
  step_size=2360,  scale_fn=lambda x: 1 / (2.0 ** (x - 1)), 
  scale_mode='cycle')

Scale function

The Scale function is the function controlling the change from the initial learning rate to the maximal learning rate and back to the initial learning rate. In [Smith's] paper this is one of triangular, triangular 2 or exponential range. In my own experiments with models for generative images (for example super resolution) I have found Triangular 2 to be most effective.

Triangular: A basic triangular cycle with no amplitude scaling:

lambda x: 1.0

Triangular 2: A basic triangular cycle that scales initial amplitude by half with each cycle:

lambda x: 1 / (2.0 ** (x — 1))

Exponential range: A cycle that scales initial amplitude by gamma to the power of the cycle iterations with each cycle:

lambda x: gamma ** x

和FWIW，作者还指出：

I believe that maybe a lack of clarity on these parameters is one of the reasons this TensorFlow learning rate schedule is not in wider use.