THE PR
Building on all the amazing work done by @lessw2020 on the Fastai Forums combining RAdam and AdamQH with Lookahead (resulting in Ranger and RangerQH respectively), I ported AdamQH over to fastai v2 to enable the use of RangerQH in the updated library…. making it my first ever open source PR!
I’ve pushed code to production at work in Facebook, but it feels different (great!) to push to a public resource that has benefitted myself and many other so much.
The Twist
More homework!
Jeremy Howard had the great idea of making the Quasi Hyperbolic element a parameter that other steppers could call, making it more flexible and applicable to other optimizers. I still have to look into whether my slow brain can figure it out, but on the surface it sounds like a promising idea.
More work <3