fastai.v3,

Part2 lesson 9, 05 anneal | fastai 2019 course -v3

Follow Apr 22, 2021 · 1 min read
Part2 lesson 9, 05 anneal | fastai 2019 course -v3
Share this

image reference: https://arxiv.org/pdf/1608.03983v5.pdf

Code Manual

question-9-1: implement two new callbacks 1) Recorder: records scheduled learning rate & plot 2) ParamScheduler: schedule any hyperparameter (as long as it’s registered in the state_dict of optimizer)

  • hint. consider when to save parameters
  • hint. self - learn object

/question-9-2: implement annealer decorator and cosine scheduler - Linear, Cosine, Exponential

question: plot above annealers start from 2 end with 0.01 (you’d need monkey patch code)

question: implement combine scheduler and test it with

  • changes the learning rate (0.3, 0.7) at 0% ~ 35% of training and (0.7, 0.01) of learning rate to the left process.
    • feed this function to Recorder / ParamScheduler callback and plot loss / lr