SGD with momentum is a variation of the SGD where we use the gradient at the previous iteration in the current update. T

[answerhappygod]

Sgd With Momentum Is A Variation Of The Sgd Where We Use The Gradient At The Previous Iteration In The Current Update T 1 (46.76 KiB) Viewed 37 times

SGD with momentum is a variation of the SGD where we use the gradient at the previous iteration in the current update. The update with momentum is as follows: ar ᎧL w(+1) = (6) (w"))-(w(-1) ow aw L(w) is the objective function as shown in the figure below, and a, y > 0 are the learning rates. Assume for iteration t= 0 and t= -1, we set wi") = wo as the initial guess. 8 6 4 Figure 1: Graph of objective func- tion L 2 -15 -10 10 15 Answer the following with justification: a) Assuming that wo = -8 (the flat region). Discuss if the basic SGD (y = 0 ) algorithm terminate at the minimum? Why? (Why not?) b) Assuming that wo = -6 (in a sloped region). Discuss if the basic SGD (y = 0 ) algorithm terminate at the minimum? Why? (Why not?) c) Assuming that wo = -8 (the flat region). Discuss if the SGD with momentum (y > 0 ) algorithm terminate at the minimum? Why? (Why not?) d) Assuming that wo = -6 (in a sloped region). Discuss if the SGD with momentum (Y > 0) terminate at the minimum? Why? (Why not?)