2 Multivariable mMathematics
1
Multivariate Spectral Gradient Algorithm fo
m m m m
r Nonsmooth Convex Optimization Problem
m m m m
s
INTRODUCTION
Considermthemunconstrained mminimization mproblem
(1)
wherem m isma mnonsmooth mconvex mfunction. mThe mMoreau-
Yosidamregularization m[1]mof m m associated mwith m
m ismdefined mby
(2)
wherem m⋅ m
m ismthe mEuclidean mnorm mand m isma mpositive mparameter. mThe mfunction mmini
mized mon mthemright-
hand msidemismstrongly mconvex mand mdifferentiable, mso mitmhasmamunique mminim
izermformevery m.m Underm somemreasonable mconditions, mthemgradientmfunctio
n mof m(m)mcan mbemproved mto mbemsemismooth m[2,
3],mthough mgenerally m(m )mismnotmtwicemdifferentiable. mItmismwidely mknown mt
hatmthemproblem
(3)
and mthemoriginalmproblemm(1)maremequivalentmin mthemsensemthatmthemtwo mcorr
espondingmsolution msetsmcoincidentally maremthemsame.The mfollowingmpropos
ition mshowsmsomempropertiesmof mthemMoreau-
Yosidamregularization mfunction m (m ).
Proposition m1 m(seemChaptermXV,mTheorem m,m[1]).mThemMoreau-
Yosida mregularization mfunction m ismconvex,mfinitevalued, mand mdifferentiable m
everywhere mwith mgradient
,Multivariate mSpectralmGradientmAlgorithm mformNonsmoothmConvexm 3
...
(4)
,4 Multivariable mMathematics
Where
(5)
ismthemunique mminimizermin m(2).mMoreover,mformallm ,m ∈m Rm,monemhas
(6)
Thism proposition m showsm thatm them gradientm function :
mismLipschitz mcontinuousmwith mmodulusm1/m .mIn mthismcase,mthemgradientmfun
ction
ismdifferentiable malmostmeverywhere mbymthemRademachermtheorem;mthen mt
hemBsubdifferentialm[4]mof at ismdefined mby
(7)
, Multivariate mSpectralmGradientmAlgorithm mformNonsmoothmConvexm 5
...
where =m{m m:
ismdifferentiable matm },mand mthemnextmproperty mof mBD-
mregularity mholds m[4–6].
Proposition m2.mIf ismBD-regularmatm ,mthen
(i) almmatricesm ∈ (m)maremnonsingular;
(ii) theremexistsma mneighborhood mNmof
m, m 1>m0, mand m 2>m0;mformallm ∈m N, mone mhas
(8)
Instead mof mthemcorrespondingmexactmvalues,mwemoften musemthemapproximate m
valuemof mfunction m(m )mand mgradientm (m )min mthempracticalmcomputation, mbecause
(m )mismdifficultmand msometimesmimpossible mto mbemsolved mprecisely. mSuppos
emthat,mformany m >m0 mand mformeach
m , mthere mexists man mapproximate mvector
(m ,m )m∈ of mthe mun ique mmin imizerm ( m )in m(2)such mthat
m
(9)
Themimplementable malgorithmsmto mfind msuch mapproximate mvector (m ,
)m∈
m can mbe mfound, mformexample, min m[7, m8]. mThemexistence mtheorem mofmthe mappr
oximate mvector (m ,m )mismpresented masmfollows.
Proposition m3 m(seemLemmam in m[7]).mLetm{
}mbemgenerated maccording mt
o mthemformula
(10)
where >m0 mismamstepsize mand isman mapproximate msubgradientmat ;mthatmis,
(11)
(i) If satisfies
(12)
then m(11)mholdsmwith
(13)
(ii) Conversely,m if(11)mholdsmwith given m by m (13),m then m (12)m holds:
= ( , ).
+1
1
Multivariate Spectral Gradient Algorithm fo
m m m m
r Nonsmooth Convex Optimization Problem
m m m m
s
INTRODUCTION
Considermthemunconstrained mminimization mproblem
(1)
wherem m isma mnonsmooth mconvex mfunction. mThe mMoreau-
Yosidamregularization m[1]mof m m associated mwith m
m ismdefined mby
(2)
wherem m⋅ m
m ismthe mEuclidean mnorm mand m isma mpositive mparameter. mThe mfunction mmini
mized mon mthemright-
hand msidemismstrongly mconvex mand mdifferentiable, mso mitmhasmamunique mminim
izermformevery m.m Underm somemreasonable mconditions, mthemgradientmfunctio
n mof m(m)mcan mbemproved mto mbemsemismooth m[2,
3],mthough mgenerally m(m )mismnotmtwicemdifferentiable. mItmismwidely mknown mt
hatmthemproblem
(3)
and mthemoriginalmproblemm(1)maremequivalentmin mthemsensemthatmthemtwo mcorr
espondingmsolution msetsmcoincidentally maremthemsame.The mfollowingmpropos
ition mshowsmsomempropertiesmof mthemMoreau-
Yosidamregularization mfunction m (m ).
Proposition m1 m(seemChaptermXV,mTheorem m,m[1]).mThemMoreau-
Yosida mregularization mfunction m ismconvex,mfinitevalued, mand mdifferentiable m
everywhere mwith mgradient
,Multivariate mSpectralmGradientmAlgorithm mformNonsmoothmConvexm 3
...
(4)
,4 Multivariable mMathematics
Where
(5)
ismthemunique mminimizermin m(2).mMoreover,mformallm ,m ∈m Rm,monemhas
(6)
Thism proposition m showsm thatm them gradientm function :
mismLipschitz mcontinuousmwith mmodulusm1/m .mIn mthismcase,mthemgradientmfun
ction
ismdifferentiable malmostmeverywhere mbymthemRademachermtheorem;mthen mt
hemBsubdifferentialm[4]mof at ismdefined mby
(7)
, Multivariate mSpectralmGradientmAlgorithm mformNonsmoothmConvexm 5
...
where =m{m m:
ismdifferentiable matm },mand mthemnextmproperty mof mBD-
mregularity mholds m[4–6].
Proposition m2.mIf ismBD-regularmatm ,mthen
(i) almmatricesm ∈ (m)maremnonsingular;
(ii) theremexistsma mneighborhood mNmof
m, m 1>m0, mand m 2>m0;mformallm ∈m N, mone mhas
(8)
Instead mof mthemcorrespondingmexactmvalues,mwemoften musemthemapproximate m
valuemof mfunction m(m )mand mgradientm (m )min mthempracticalmcomputation, mbecause
(m )mismdifficultmand msometimesmimpossible mto mbemsolved mprecisely. mSuppos
emthat,mformany m >m0 mand mformeach
m , mthere mexists man mapproximate mvector
(m ,m )m∈ of mthe mun ique mmin imizerm ( m )in m(2)such mthat
m
(9)
Themimplementable malgorithmsmto mfind msuch mapproximate mvector (m ,
)m∈
m can mbe mfound, mformexample, min m[7, m8]. mThemexistence mtheorem mofmthe mappr
oximate mvector (m ,m )mismpresented masmfollows.
Proposition m3 m(seemLemmam in m[7]).mLetm{
}mbemgenerated maccording mt
o mthemformula
(10)
where >m0 mismamstepsize mand isman mapproximate msubgradientmat ;mthatmis,
(11)
(i) If satisfies
(12)
then m(11)mholdsmwith
(13)
(ii) Conversely,m if(11)mholdsmwith given m by m (13),m then m (12)m holds:
= ( , ).
+1
