Learning rate
OrcaNet provides multiple ways to easily determine a learning rate schedule
for the training. For this, a
learning rate varibale is stored in organizer.cfg.learning_rate, which is set
to None per default (no change to the current learning rate of the model).
Depending on what type this attribute is, the learning rate schedule
during the training will be one of the following:
Float
The learning rate will be constantly this value, for all epochs.
Tuple
A tuple (or list) of two floats: The first float gives the learning rate in epoch 1 file 1, and the second float gives the decrease of the learning rate per file.
For example, if organizer.cfg.learning_rate = [0.1, 0.4] is used,
the learning rates will be 0.1, 0.06, 0.036, …
Function
A custom learning rate schedule. The function has to have exactly two input parameters: The epoch and the file number (in this order). It must return the learning rate for this (epoch, fileno) pair.
String
A custom learning rate schedule in the form of a txt document. It is the path to a csv file inside the output folder the organizer was initialized with. This file must contain 3 columns with the epoch, fileno, and the value the lr will be set to when reaching this epoch/fileno.
An example can be found in orcanet/examples/learning_rate.csv:
1# Example for setting the learning rate with a csv file.
2#
3# Place this in the main folder of an orcanet training, then use
4# learning_rate="learning_rate.csv"
5# in the configuration to use it.
6#
7# The file must have three columns, containing the epoch, the file number,
8# as well as the learning rate that will be used for the given epoch/file number pair.
9# Indices start from 1, i.e. in the following example,
10# the lr for the first epoch (= file 1 in epoch 1)
11# is set to 0.002 in the first row (the one starting with 1 1).
12# It will still be 0.002 for epoch 1 file 2, and will be changed to 0.001
13# in epoch 1 file 3.
14#
15# Epoch Fileno LR
161 1 0.002
171 3 0.001
182 1 0.0005
192 3 0.0001