Note that R0 is a (ρ√ σT /√
2θ)-strongly convex on X. Suppose R0 is also a mirror map. Then, pluggingR0 into the above inequality yields, for every u∈X,
RegretT(LOMDR0,ENEMY, u)≤ ρ√
√ T
2σθ(R(u)−min
x∈XR(x)) +ρ√
√θT
2σ ≤ ρ√
√θT
2σ +ρ√
√θT
2σ =ρ r2θT
σ , where in the second inequality we took the supremum overu∈X.
of a mirror map, we know that the Bregman projections onto X w.r.t. R lie in int(domR)∩X.
Therefore, if we haveint(domR)∩X⊆intX, equivalence between EOMD and LOMD holds forC!
In fact, requiring the iterates to lie in the interior of X is excessively strong. For example, the spectraplex is not full-dimensional since it lives inside a hyperplane w.r.t. the trace inner product.
Thus, the above argument does not cover the cases considered in [3]. Still, a very similar argument holds if we require the iterates to lie on ri(X). Indeed, the next theorem proves equivalence of the eager and lazy version of OMD under this assumption. One may note during the proof that a slightly technical issue arises: we need to ensure that both iterates pick the same subgradients whenever they have to choose one, that is, they have to pick a subgradient according to the same well-order on the subdifferentials. Usually the algorithms access the subgradients of a convex function f:E→(−∞,+∞] through a function of the formx∈X 7→g∈∂f(x). Thus, assuming the both algorithms pick the same subgradient given the same functionf and the same pointxis not a strong assumption.
Theorem 5.6.1. LetC:= (X,F) be an OCO instance such that X is a nonempty closed set and each f ∈ F is a proper closed function which is subdifferentiable on X. Let R:E→ (−∞,+∞]
be a mirror map forX. Ifint(domR)∩X⊆ri(X) and bothEOMDXR andLOMDXR use the same well-order on the sets∂f(x) for f ∈ F andx∈X, thenEOMDXR = LOMDXR.
Proof. Suppose int(domR)∩X ⊆ri(X). Let us prove, by induction onT ∈N, that EOMDXR(f) = LOMDXR(f), ∀f ∈ FT.
ForT = 0, we have
{EOMDXR(hi)}= arg min
x∈X
R(x) ={LOMDXR(hi)}.
LetT ∈N\ {0}and f ∈ FT. By induction, we have
xt:= EOMDXR(f1:t−1) = LOMDXR(f1:t−1), ∀t∈[T].
Let us show thatEOMDXR(f) = LOMDXR(f). Letgt∈∂ft(xt)be as in the definition ofEOMDXR(f) for each t∈[T](which are the same asgt in the definition of LOMDXR(f) is both oracles use the same well-order on the subdifferentials to pick the subgradients). By Theorem 5.4.2, there are p1, . . . , pT ∈Ewithpt∈NX(xt) for eacht∈[T]such that
{EOMDXR(f)}= arg min
x∈X
XT
t=1
hgt, xi+R(x) +
T
X
t=1
hpt, xi
By (5.1.ii) from the mirror map definition,ΠRX(z)∈int(domR)∩X ⊆ri(X)for anyz∈int(domR).
In particular,xt= EOMDXR(f1:t−1)∈ri(X) for everyt∈[T]. Let us show that the termshpt, xifor t∈[T]do not affect the point that attains the above minimum. More specifically, let us show
hpt, xi=hpt, xti, ∀x∈ri(X),∀t∈[T]. (5.21) Let t ∈ [T]. Since pt ∈ NX(xt), we have hpt, x−xti ≤ 0 for every x ∈ X. Thus, suppose there is x¯ ∈X such that hpt,x¯−xti <0. Since xt∈ ri(X), by Theorem 3.2.2 there is µ >1 such that xµ:=µxt+ (1−µ)¯x∈X. Since1−µ≤0,
hpt, xµ−xti= (1−µ)hpt,x¯−xti>0,
a contradiction to the fact thatpt∈NX(xt). This proves (5.21). Thus, using thatgt∈∂ft(xt) are the same in the definitions ofEOMDXR(f) andLOMDXR(f) for each t∈[T], and by Theorem 5.5.1 we have
{EOMDXR(f)}= arg min
x∈X
XT
t=1
hgt, xi+R(x) +
T
X
t=1
hpt, xi
(5.21)
= arg min
x∈X
XT
t=1
hgt, xi+R(x)
Thm.5.5.1
= {LOMDXR(f)}.
Chapter 6
Adaptive Regularization
Up to this point, all the algorithms we have seen for OCO problems required us to make a smart choice of regularizer/mirror map strategy based on the parameters of the instance at hand to guarantee good regret bounds. For example, most algorithms seen on Chapters 4 and 5 required previous knowledge of the Lipschitz constant of the functions played by the enemy to properly scale the regularizer and, in this way, guarantee low-regret bounds. Not only that, the strategies seen so far used little to no information from functions previously played by the enemy, usually only looking at most at the round number of the game to compute the new regularizer/mirror map increment.
Moreover, the regret bounds from previous chapters depend on the Lipschitz constant of the functions played by the enemy, since by Theorem 3.8.4 the Lipschitz constant usually upper bounds the dual norms of the subgradients. The problem is that this only reflects a worst-case scenario, not giving much information in the case of enemies who play functions with subgradients whose dual norm is small. Thus, one may wonder if it might be possible to derive regret bounds for some OCO algorithms which still depend on the dual norms of the subgradients (in an informative way) instead of using a crude upper bound on such norms. Intuitively, when playing against “easy” enemies, the player oracle should perform better, and we should be able to have better regret guarantees in such cases.
In this chapter we describe algorithms which use information from the subgradients of the functions played by the enemy to better choose its regularizer increments during the game. In order to derive regret bounds we show that these algorithms are special cases of theAdaReg algorithm, first described in [33], which is a clever application of the Adaptive Online Mirror Descent algorithm from Chapter 5. This enables us to make a unified analysis, first presented in [33], of two known and similar algorithms from the OCO literature: the AdaGrad [31] and Online Newton Step [37]
algorithms. As a warm-up, on Section 6.1 we describe an online mirror descent algorithm (and its lazy version) with step sizes which adapt based on the subgradient of the enemy’s functions. On Section 6.2 we present theAdaReg algorithm, discuss its main ideas, and derive a general regret bound by writing it as an AdaOMD algorithm with a smart choice of mirror map strategy. On Section 6.3 we describe the AdaGrad algorithm and derive a regret bound for it from the regret bound we have for AdaReg. On Section 6.4 we show a more efficient version of AdaGrad which only uses diagonal instead of general matrices to skew the subgradient steps. On Section 6.5 we define exp-concave functions and present the Online Newton Step algorithm, which has a logarithmic regret bound against exp-concave functions. Again, we derive regret bounds for the Online Newton Step algorithm by writing it as an application of the AdaReg algorithm. Finally, on Section 6.6 we show a step size strategy for online gradient descent which attains logarithmic regret against strongly convex functions. Additionally, we show how it can be seen as a “scalar version” of the
Online Newton Step algorithm.
In this chapter we will extensively use norms induced by positive definite matrices. Thus, let us define some notation related to norms based on positive definite matrices. LetA∈Sd++. Abusing the notation of Bregman projection, for every closed convex setX ⊆Rd define
{ΠAX(z)}:= arg min
x∈X
kx−zkA, ∀z∈Rd.
If R:= 12k·k2A, then one may verify that BR(x, y) = 12kx−yk2A. Thus, in this case, for any closed convex set X⊆Rd we have ΠRX = ΠAX. This equation of Bregman projections may be used without reference throughout this chapter. Moreover, throughout this chapter we consider thatSdis equipped with the trace inner product given byhX, Yi:= Tr(XY) for everyX, Y ∈Sd.
6.1 A First Example: Adaptive Online Gradient Descent
In Chapter 4, we have seen different FTRL regularizer strategies for some classes of OCO instances.
Among all the regularizer strategies seen in that chapter, the regret bounds when using them in AdaFTRLdepended mainly on whether the regularizer strategy was proximal or not, that is, whether the regularizer increment at roundt+ 1 was minimized by the iterate from round t (if t >1) or not. In spite of the discussion made about the differences on the regret bounds we have for each of these regularizer strategy classes (see Theorem 4.4.3, 4.4.4, and the discussion which follows them), in all the cases studied in Chapter 4, proximal regularizer strategies did not yield any significant differences on the final regret bounds if compared to the ones given by non-proximal strategies. Thus, even though the regret bounds given by Theorems 4.4.3 and 4.4.4 hint at the possibility of proximal regularizer strategies being able to adapt better to the functions picked by the enemy, we have not yet exploited this difference in this text.
One aspect which also motivates the investigation of the advantages of proximal regularizer strategies are the connections of Online Mirror Descent algorithms to Adaptive FTRL algorithms seen in Chapter 5. Namely, letR be a mirror map strategy for some OCO instanceC:= (X,F). In Section 5.4 we have seen that AdaOMDXR is equal toAdaFTRLR0 when applied to linear functions, where R0 is a proximal regularizer strategy forC mostly based on Bregman divergences w.r.t. the regularizer increments given by R. Additionally, in Section 5.5 we have seen that AdaDAXR is equivalent to AdaFTRLR00 when applied to linear functions, where R00 is equal to Reverywhere but on the empty sequence, where R00(hi) :=R(hi) +δ(· |X). Thus, investigating when and how proximal FTRL regularizer strategies can be advantageous may shed light on the key differences between AdaOMDand AdaDA.
Our main goal in this chapter is to devise mirror map strategies for AdaOMD which take advantage of the adaptiveness present on its regret bound, which is inherited from the regret bound for proximal FTRL regularizer strategies from Theorem 4.4.4. One may build proximal FTRL regularizer strategies similar to the mirror maps seen in this chapter which yield similar regret bounds. Still, in this chapter our focus is on the Adaptive Online Mirror Descent algorithm since looking at adaptive regularizers forAdaOMDyields an unified analysis of two major OCO algorithms:
AdaGrad [31] and Online Newton Step [37].
As a warm-up, let us devise versions of the Online Mirror Descent algorithm (with a static mirror map) with time-varying step sizes based on the subgradients of the enemy’s functions. Our goal is to devise an algorithm with low-regret guarantees regardless of the number of roundsand without knowledge of a bound on the norms of the subgradients. Not only that, we want the regret bound to be better for an enemy which picks “easy” functions, that is, functions whose subgradients
have small (dual) norm. For example, letC:= (X,F) be an OCO instance such that eachf ∈ F is subdifferentiable on X ⊆Rd with1 kgk2 ≤ρ for every x ∈X and g ∈∂f(x) and that there is θ∈R++ such thatθ≥supx,y∈X 12kx−yk22 . On Section 4.7 we have seen how to build a proximal FTRL regularizer strategy which yields a regret bound that holds for any number of rounds. Roughly, Corollary 4.7.3 tells us thatAdaFTRL with (cumulative) regularizer Rt at roundt∈N for t >1 given by
Rt(x) :=
s θ (t−1)ρ2
1
2kx−xt−1k22
, ∀x∈Rd,
where xt−1 is the point chosen by the player at round t−1, yields regret smaller than2ρ√
θT for a game withT ∈N rounds. Letg1, . . . , gT ∈Rd be such that, for everyt∈[T], we havegt∈∂ft(xt) where ft∈ F is the function picked by the enemy on roundt. Observe that Pt−1
j=1kgjk22≤(t−1)ρ2. That is, the denominator of the multiplicative factor on the regularizer at round tis an upper-bound on the sum of the squared norms of the subgradients of the functions played by the enemy so far (i.e., on rounds1 up tot−1). The idea to make this regularizer strategy adaptive is to, instead of using such an upper bound on the multiplicative factor of the regularizer, use the actual sum of the squared norms of the subgradients, without needing the knowledge of the constant ρ in advance.
Interestingly, this implies that in the case where the enemy plays functions whose subgradients have small norm, then the step sizes2 will be bigger, that is, the algorithm will be more aggressive. In the following theorem we describe a mirror map strategy with the same idea. Later we will see how this strategy need not work very well for non-proximal regularizer strategies.
Theorem 6.1.1. Let C := (X,F) be an OCO instance such that X is closed and such that each f ∈ F is a proper closed function which is subdifferentiable onX, and letR:E→(−∞,+∞]be a mirror map for X which is σ-strongly convex onX w.r.t. a norm k·konE. Moreover, suppose there isθ∈R++ such thatθ≥sup{BR(u, x) : u∈X, x∈X∩int(domR)} and define
η(g) :=
q θ
2Pt
j=1kgjk22 if g6∈ {0,hi},
1 otherwise,
for everyg=hg1, . . . , gti ∈Seq(E),and R(f) :=
1
η(g1:t) −[t >0] 1 η(g1:t−1)
1
√σR, for everyf =hf1, . . . , fti ∈Seq(F),where gi ∈E is as in AdaOMDR(f)for i∈[t].
LetENEMY be an enemy oracle forC, letT ∈N, and define
(x,f) := OCOC(AdaOMDXR,ENEMY, T).
Finally, let gt ∈ ∂ft(xt) be the same as in the definition of AdaOMDXR(f) on Algorithm 5.1 for each t∈[T]. Then
Regret(AdaOMDXR,f, X)≤ v u u t
2θ σ
T
X
t=1
kgtk2∗.
Moreover, if R= 12k·k22, then, by setting ηt:=η(g1:t−1) for eacht∈[T]we have
xt= Πk·kX 2([t >1](xt−1−ηtgt−1)), ∀t∈[T]. (6.1)
1We are using the squared`2-norm because for the sake of simplicity and since it yields the online gradient descent algorithm.
2The constant multiplying the regularizer.
Proof. Let us first prove the regret bound. First of all, by the definition of mirror map, it is clear that µRis a mirror map for X for anyµ∈R++ sinceR is a mirror map forX. Thus, Ris a mirror map strategy for C. For eacht∈ {1, . . . , T + 1} definert:=R(f1:t−1). Note that, for everyt∈[T],
t
X
i=1
ri= 1
√σηt
R, which is (√
σ/ηt)-strongly convex w.r.t. k·k. By setting x0 := x1, Theorem 5.4.3 yields, for every u∈X,
Regret(AdaOMDXR,f, u)≤
T+1
X
t=1
Brt(u, xt−1) +1 2
T
X
t=1
ηt+1
√σ kgtk2∗
= 1 2
T+1
X
t=1
1 ηt
−[t >1] 1 ηt−1
1
√σBR(u, xt−1) + 1 2√
σ
T
X
t=1
ηt+1kgtk2∗
≤ θ
2ηT+1√
σ + 1 2√ σ
T
X
t=1
ηt+1kgtk2∗
≤ 1 2
v u u t
2θ σ
T
X
t=1
kgtk2∗+1 2
r θ 2σ
T
X
t=1
kgtk2∗ q
Pt
j=1kgjk2∗
Le. 4.6.2
≤ 1 2
v u u t2θ
σ
T
X
t=1
kgtk2∗+1 2
v u u t2θ
σ
T
X
t=1
kgtk2∗= v u u t2θ
σ
T
X
t=1
kgtk2∗.
Let us show that (6.1) holds for R:= 12k·k22. First, recall that, by Lemma 5.2.1, 12k·k22 is a mirror map forXwhich is1-strongly convex w.r.t. the`2-norm. Let us proceed with the proof by induction on t∈[T]. Fort= 1, by the definition ofAdaOMDXR(hi)we have{x1}= arg minx∈Xkxk2 ={Πk·kX 2(0)}.
Thus, let t ∈ {2, . . . , T}, and let Rt and yt be defined as in the definition of AdaOMDXR(f) in Algorithm 5.1, that is,
Rt:=
t
X
i=1
rt= 1 2ηt
k·k22 and yt:=∇Rt(xt−1)−gt= 1 ηt
xt−1−gt.
So, since the`2-norm is self-dual, by Theorem 3.8.2 we have(12k·k22)∗ = 12k·k22. Thus, by Theorem 3.4.3 we have R∗t(y) = 2η1
tkηtyk22 = η2tkyk22 for every y∈E. Finally, the definition ofxtin AdaOMDXR(f) yields
xt= ΠRXt(∇R∗t(yt)) = Πk·kX 2(xt−1−ηtgt−1).
Note that the above regret bound is at least as good as the one for EOMD from Corollary 5.4.4 (or as the one for the proximal FTRL example from Corollary 4.7.3). Indeed, let C:= (X,F) be an OCO instance,R be a mirror map forX which is 1-strongly convex onX w.r.t. a norm k·kon E, T ∈N, and letθ andgt∈Rd be as in the above theorem for everyt∈[T]. If we know ρ∈Rsuch thatkgtk∗≤ρ for everyt∈N, then
v u u t2θ
T
X
t=1
kgtk2∗ ≤ρ
√ 2θT ,
and the latter exactly matches the bound from Corollary 5.4.4. However, the bound given by Theorem 6.1.1 can be much better since, if the enemy picks many functions whose subgradients norm are way smaller than ρ, then the algorithm will be able to exploit this fact (using bigger step sizes) and attain smaller regret.
One may be wondering if a mirror map strategy similar to the one from the above theorem may work for the Adaptive Dual Averaging algorithm. As we have discussed on Section 5.5, Dual Averaging is practically equivalent to the general FTRL algorithm applied to the subgradients of the functions picked by the enemy. Thus, not surprisingly, the regret bound for AdaDAfrom Corollary 5.5.2 is identical to the regret bound for AdaFTRL with general regularizer strategies given by Theorem 4.4.3. That is, when usingAdaDA we do not have the additional adaptiveness present in the proximal FTRL regret bound from Theorem 4.4.4. Even though we can adjust the multiplicative factors ofAdaDA with a mirror map similar to the one from the last theorem so that the final regret bound is similar as well, we still need in this case knowledge of the Lipschitz constant (or an upper-bound on the subgradients’ norms) of the functions played by the enemy to properly adjust the multiplicative factor of the mirror map strategy. This is one case where the intuition that proximal regularizer strategies can “measure the subgradient from roundtwith the norm related to the regularizer increment from roundt+ 1”, as discussed on Section 4.4, yields a relevant difference between regret bounds.
Theorem 6.1.2. Let C := (X,F) be an OCO instance such that X is closed and such that each f ∈ F is a proper closed function which is subdifferentiable onX and let R:E→(−∞,+∞]be a mirror map forX which isσ-strongly convex on X w.r.t. a norm k·kon E. Suppose eachf ∈ F is ρ-Lipschitz continuous w.r.t.k·kon a convex setD⊆Ewith nonempty interior such that3 intD⊇X, and suppose there is θ∈R++ such thatθ≥sup{R(u)−R(x) : u, x∈X}. Define
η(g) :=
s θ 2(ρ+Pt
j=1kgjk22) for everyg=hg1, . . . , gti ∈Seq(Rd), R(f) :=
1
η(g1:t) −[t >0] 1 η(g1:t−1)
1
√σR, for everyf =hf1, . . . , fti ∈Seq(F),where gi ∈Rd is as in AdaDAR(f) for i∈[t].
LetENEMY be an enemy oracle forC, letT ∈N, and define
(x,f) := OCOC(AdaDAXR,ENEMY, T).
Moreover, let gt ∈ ∂ft(xt) be the same as in the definition of AdaDAXR(f) on Algorithm 5.3 for each t∈[T]and setηt:=η(g1:t−1) for eacht∈[T]. Then
Regret(AdaDAXR,f, X)≤ v u u t2θ
σ (ρ+
T−1
X
t=1
kgtk2∗).
Proof. Since µRis a mirror map forX for anyµ∈R++, we have thatRis a mirror map strategy for C. For each t∈ {1, . . . , T + 1}definert:=R(f1:t−1). Note that, for everyt∈[T],
t
X
i=1
ri= 1 ηt
√σR,
3Here we assume this so that, for anyf ∈ F, any subgradient of f at a point ofF has small dual norm by Theorem 3.8.4. Still, if one can control the choice of subgradients ofAdaDA, one may only requireX⊆D.
which is (√
σ/ηt)-strongly convex w.r.t. k·k. By settingx0 := x1, Corollary 5.5.2 yields, for every u∈X,
Regret(AdaDAXR,f, u)≤
T
X
t=1
(rt(u)−rt(xt)) +1 2
T
X
t=1
ηt
√σkgtk2∗
= 1 2
T
X
t=1
1
ηt −[t >1] 1 ηt−1
1
√σ(R(u)−R(xt)) + 1 2√ σ
T
X
t=1
ηtkgtk2∗
≤ θ 2ηT
√σ + 1 2√
σ
T
X
t=1
ηtkgtk2∗
≤ 1 2
v u u t2θ
σ
ρ+
T−1
X
t=1
kgtk2∗ +1
2 r θ
2σ
T
X
t=1
kgtk2∗ q
ρ+Pt−1 j=1kgjk2∗
≤ 1 2
v u u t
2θ σ
ρ+
T−1
X
t=1
kgtk2∗ +1
2 r θ
2σ
T
X
t=1
kgtk2∗ q
Pt
j=1kgjk2∗
Le. 4.6.2
≤ 1 2
v u u t2θ
σ
ρ+
T−1
X
t=1
kgtk2∗ +1
2 v u u t2θ
σ
T
X
t=1
kgtk2∗
≤ v u u t
2θ σ
ρ+
T−1
X
t=1
kgtk2∗ .