Estimation and accuracy of the stationary solution in the
dynamic programming problem: New results
Wilfredo L. Maldonado 29 de outubro de 2002
D FRQWUDFWLYH PHWKRG IRU FRPSXWLQJ WKH
VWDWLRQDU\ VROXWLRQ RI WKH HXOHU HTXDWLRQ
"
b,uijaN w B,aNABaN |
#Q6Oji|N NijiB }
"Ghsduwphqw ri Hfrqrplfv
Xqlyhuvlgdgh Ihghudo Ioxplqhqvh
Uxd Wludghqwhv 4:/ Lqjd/ Qlwhurl
FHS 575430843/ UM/ Eudvlo
Hpdlo= zloo|Clpsd1eu
#HSJH2IJY
Sudld gh Erwdirjr/ 4<3
FHS 555860<33/ UM/ Eudvlo
H0pdlo= kxpehuwrCijy1eu
Ot|iBW| UL?|h@U|i 4i|L_ uLh UL4T|?} t|@|L?@h) tL*|L?t Lu ?|ih|i4TLh@* i^ *Mh4 4L_i*t t ThL_i_ Ai 4i|L_ t 4T*i4i?|i_ t?} @ UL?|h@U|L? 4@TT?} _ihi_ uhL4 |i ht|Lh_ih UL?_|L?t Ai _i|ih4?t|U _)?@4U ThL}h@44?} ThLM*i4 t ti_ |L **t|h@|i |i 4i|L_ 5L4i ?4ihU@* i @4T*it @hi TihuLh4i_
41 Lqwurgxfwlrq
Lq lqwhuwhpsrudo hfrqrplf prghov rqh ri wkh pdlq glfxowlhv lv wr qg dffxudwh hvwlpdwlyhv
ri wkh vwdwlrqdu| vroxwlrqv1 Iru lqvwdqfh/ lq g|qdplf surjudpplqj prghov wkh wudglwlrqdo
phwkrg lv edvhg rq wkh Ehoopdq dssurdfk1 Wklv frqvlvwv lq hvwlpdwlqj wkh fruuhvsrqglqj
ydoxh ixqfwlrq xvlqj d frqwudfwlrq pdsslqj +vhh Vwrnh| dqg Oxfdv zlwk Suhvfrww +4<;<,,
dqg wkhq frpsxwlqj wkh srolf| ixqfwlrq iurp wkh dssur{lpdwhg ydoxh ixqfwlrq1
Wd|oru dqg Xkolj +4<<3, ghvfulehg vrph qxphulfdo phwkrgv edvhg rq wkh Ehoopdq lwhud0
wlrqv +l1h1/ Ydoxh0Ixqfwlrq Julg dqg Txdgudwxuh Ydoxh0Ixqfwlrq Julg phwkrgv, dqg Vdqwrv
dqg Yljr0Djxldu +4<<;, dqg Pdogrqdgr dqg Vydlwhu +5334, surylghg dq hvwlpdwlrq huuru
iru wkh dssur{lpdwh srolf|1 Krzhyhu/ Ehoopdq*v phwkrg kdv wzr pdlq glvdgydqwdjhv= lw lv
rqo| xvhixo zkhq wkh prgho fdq eh h{suhvvhg dv d uhsuhvhqwdwlyh djhqw prgho dqg/ ehvlghv
wkdw/ wkh vshhg ri frqyhujhqfh lv vorz1 Rq wkh rwkhu kdqg/ qxphulfdo phwkrgv edvhg rq wkh
W At T@Tih @t Thiti?|i_ @| |i H| W?|ih?@|L?@* L?uihi?Ui Lu |i 5LUi|) uLh L4T|@|L?@* ,UL?L4Ut ? a?ic 2ff2 Ai @|Lht }h@|iu**) @U!?L*i_}i |i UL44i?|t Lu k a__ `i @*tL |@?! @L WMti? @?_ #@_ Lh@ uLh i*T?} t | |i ?4ihU@* 4T*i4i?|@|L?2 A 2 , 3 2 ( , ' 2)
, (2 2 2) (2 A
; 2 5 % 5009::+
2 ,7( 2 , 2 A A
(22 () 2 ) + 2 #
2 &; 500": B 2 ,+ <
6 / #< 5 """: ( 7 ( 2 2 ,( 2
2 B , 2 2 2
2+ 500" 00: 2 2 ( 2 B #
) + @2 2 ,
A 2 C 2 5% 500 :: 2 B#;
2 5= 5099: = 8 B 5009::+
2 2 2 , A
) + 2 ( 4
(22 2 2 ) 4; + = 2 2
( 2 F
4+ < 2 2
2 2 2 ) 2 , +
2 ( 7 2 , 2 A +
2 2 A (22 2) 2
2 ) 2 ) + 2 ) ,
2 2 + 6 ;
(2 2 4 2 +
2 (+ ( 2
2) 2+ 1 2 , 2 #A
+ ) ! 2 2 2 2
+ < 2 ;+
!" !#$%&
2 A, 2 ( (2 , ,) 2
( ' [ ?
G2 G [[[ 2
,) H ' G $ ?
G2 (2 B 4 2 ) A,+
D ( 2 H ( ) E, 2 2 H
54 2 G+
48 m=m , 2 A ?
G+ 2 2 A
q 5P
G: ( , n = n+
5) E
K5{: F i| 5 ?
GG m| {m ? uj+
< A, 2 {
35 [ A 5{
M:
M32 2
H5{
M4> {
M> {
M.4: F "
) 5[> G> H: j' [ $ [ 2 2'
H5{> j5{:> j
55{:: F "
Vlqfh.lv d ixqfwlrq ri 3%c %2c %5 M (/.
lv wkh yhfwru ri sduwldo ghulydwlyhv ri. zlwk uhvshfw wr%1
28f8 7
t%M^G( %'f%c f M }?
+
D ) 2 H
q5H: F H+ D ( 7 2 ('
""#' Wkhuh h{lvwv dq lqwhulru vwhdg| vwdwh H dqg vxfk wkdw
+l, 5
:
I 5
:
+ll, 5
:
$#!&( 2 2 ; ) A ) A,
2 2 ) , 2 B F "+ < )
3 2 '
5
:
I 5
:
(22 (7 2 +
< , ; 2 2 ) ,+ ( 2
6 7) 8 (2 5090:
5
w w w: F
5
w w: I
5
w w:
(2 2 2 2 4
2 + 2 2 ) + ) )
2 2 # )* 2
2 % , 2 ) +
+ ! $",
2 ( ( 2 )
2, 2 ) 2 5 :+ 2 2 4
2 ) A, A 2 ( 2 2
(2 2 ) 2 4; +
- " " H
qF
5
u
5H:: G 5H: F H 5:
5:
u5H:
(2
o5
u
5H:: 2 #2 ) E,
u5H:
+
4 2
F
{#EK{
5:
8 H
, 2 (2 2 + 2 5 H
:
+ @, 2 ,) 2 4 2 5 2
4 : ) 2 H
,
5
u
5H::+
38 8 85 ! 7 6
9 9 7 %
$##! + Xqghu / wkhuh h{lvw "/ " dqg ' vxfk wkdw
5
5:
5:: F "
5J:
iru doo
5H: dqg 1
-$$# + Dvvxph 1 Wkhq wkhuh h{lvw " dqg " vxfk wkdw ' ghqhg
lq Ohppd 614 lv d 0frqwudfwlrq/ iru vrph 5" :1
2 2 1+ 2 ( 2 2 ' H
H
A) 2 4; + ) 2 2 4; )
5 :+ 2 2 2
2 2, 2 2 ) +
2 ( ) 2 ( 2 2 2 , 2 1+
2 ) 2, 2 (2 4+
,,!. ++ Li wkh dvvxpswlrq lv vdwlvhg lq d frqyh{ qhljkerukrrg ri H/ wkhq
wkhuh h{lvw " dqg ' vdwlvi|lqj 5: +zkhuh
5H: lv uhsodfhg e| , dqg wkhuh
h{lvw dqg vxfk wkdw
lv d 0frqwudfwlrq zlwk {hg srlqw 1
/ -$ ,0-# ! #$!, 1!#',$"
2 ( ( , 2 2 2 1+ + & 4 ( (
2 2 2 +
8 2 ) ,+ 2 4 2 4#
2 ( 2 A ' , '
4 ' ) ,)
5 5:: I
5 5: 5 5::: F "
2 ) 4; 2 A 5
5
::
( (2
+
"( 2 2 2 A
&? 2 + 2 4 2 ( A '
5: F
;
i # 5j
5 : I
5 :
(2 2 , (2 2 ,
5: F 5
5::+
2 4 2 2 '
5
5:: I
5
5:
5
5::: F "
# #
(2
5: F ;
i # 5j
5 : I
5 :+ 2 4 2
A ) 2
F
5
:+ > 2 & 2 2
+ 2 2 A &? +
&; 500":: 2 2 7 B 2
(2 500" 00: 2 +
8 5009: 2 ' 2 4 2 2 1+
2 2
)+ ; , ( ( 2 ( 2
2 2 2 ( ' 5: 2 (2 2
) 2 2 2G 5: 2 2
)
2 )
+
D ( , 2 2 2 (22
2 +
$2 E (2 2 2 ,
2 ) 7 B 2 + = ) (
; 2 ) 2 +
8
' , 5
H 2 ) :+ ;
+
'
5
5:: F "
5:
4
5: ++
5
5:
5
5::: F "
5
5:: F "
4
5:+
( 2 ('
5:
4
+ 2 ( 2 2 A,
A 2 ( A
(2 ( 2 +
5
5:: F "
4
5:+
2 ( 2 2
5: )
5: 2
(2 7 ) B 2 +
!
2 (2 (2 ) '
5: F
5: F
(2 " 5 F 5: F : " +
2 A ,)'
5 : F 5
:
I
5
:
F "
F F , 2 ;)
5: F
5: I
I
2 ) 5: F
(22 4; +
2 2 ; , 2 , (
2 2 C4 2 2 2+
2 = / 2 ( 2 2 ( '
" #
I
I
(2 2 + 7 2
2 2 !
2 + 2
#
,,) +
4 ( 2 ( 2 2 (
4 2 ' F 0$ K"$G L+ 2
5: F H +
4 5: F 5: F 1! 2 ) 2
2
" 5(22 : 2 52
,(
%1!0 "
&:+ 4 F F $ 2 2
) ;) 7 ( ( ) 2 ( 2
2 2
%52 ,(
"9 "
%:+
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.4
0.6 0.8 1 1.2 1.4 1.6 1.8 2
k
t kt+1
Growth Model with Logarithmic Utility Function
'()*+' (2 (2 5: F F $
F 1! F 0$ F "+
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.12
0.14 0.16 0.18 0.2 0.22 0.24 0.26 0.28 0.3 0.32
k
t kt+1
Growth Model with Power Utility Function
" ! ! #$ #%%&''
8 5009: ) (2 ,) "5: F
5 :
5 " F : " + 2 A, A
2 ,)'
5#
#
#
: F #
I $
#
#
#
F "
(2 #
2 5: 2 ) % $5: F
4(,
4(,+,4,(,
< ) 4 ,) ' K" L K" L 2 2
$
5:
5:
F 5:
(22 ) A, ' H# F 5 I $5::
@ 2 ( , 2 '
" $5: $
5:
$
5: 5 ":
" $5:
$
5: "
2 2 A 8 5009: )
) 27+ = 2 2
# ) 5 2
#
,) 8 5009::+
1 2 ( ) ; 2 ( '
F ! F 00 2 K"$G L+ D (2
2 2 2 52 2 2 1M "
,:+
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1
pt pt+1
Monetary Model of Li (1998)
'()*+1' 6 ) 2 ) 8 5009:
(2 F ! F 00 F "+
( ! #) * #%%+''
& 2 500!: )B ( # (2 (2 , ;)+
2 ) 2 2 2B ,)'
5
: F 5
:
5
:
(2 2
2 ; 2
7 (22 ;) 2 ,
, 5" :+ 2 2 A, A ,)'
5
: F
5
: I
5
: F "
2 A )
H F
5 :5 :
5 :5 : I 5 :
2 '
I 5
I
:
I
I
(22 2 2 2 ; ) 5 2
7 :+
! 2 ( 2 ; 2 ) 2 )
2 ) + 2 ' F $ F 0$
F $ F $ 2 K"$G L+ D (2 2
2 2 52 2 2 11 "
&:+
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.06
0.08 0.1 0.12 0.14 0.16 0.18 0.2 0.22 0.24
xt
xt+1
Two−Sector Growth Model of Boldrin and Rustichini (1994)
'()*+!' 6 ) 2 ( # (2
2 ,""
2 ( 2 ; 2 ) 2
A + C E
2 2 2
# ) #
, + 2 2 ) E,) 2
) +
2 A 2) 2 (2 2 4 2
A 2 ) + 2 2) 2 2)
2 2 ) 2 ) +
< 2 2 2 2 2 2 2
2 B 2 ( ;
2 + @ 2 ,) ;
2 +
3
4 4 $##! +' < 2 2 ; " 2 2
5H H H:
u
I
5&:
u
5&&:
8 , 2 2 5
: 5
5
:: 4
2, 2 5H H: ( ) E,
5H H
5H:: F 5H H H: F "
5
5
:: 4 ; 5
: 2, 2
5H H:+ &) 2 2 2 ; '
'
5" : )
E, '
"5H:
#5H: 5 F
: 2 2
5
5
:: F "
l5H: ( F
F 5
:
= 2
"5H:
5: F
5 5:
5::K
5 5:
5:: I
5 5:
5::55::5:L
5:
u
I
"
5H:
, 5&: 2 2 5:
5H:+
@, 2
5: F H I
5: H
-.
5H I %5 H:: H '
2 ( 2 ' F '
F '
+
D 2 ( 7 ' F + 8
F ' "G 4
5H:+ 6
2
+ 2
5H: 2 ; A
5H: 2 2
5
5
:: F "+ @2( 2 ;
N
F N
) 2
A)+ 4 ' K" L ,)
5%: F 5
N
:
5
I %5
N
:
5
::
2 5": F 5: F "
5%: F 5 N
:
5
N
: "+ 6
5
5
I %5
N
:
5
::: 4 ; 2 +
8 , 2 ,
5H:+ 8 5
:
, A
5H:
+ 6 55
::
A
H
5H: 5 : 2 ; ,A
5H:+ &) 2 ) 5
5 :: 5
5:: F "+
> ( A) ) 2 2 A 55
::
+ D ) 2 2 5
5:: 2
A + 2 2 ,
5H: ( ; ,
H (22 5):
5H:+ 2 2 4
+
( ( 2 2 "+ 8 2 '
5: F K5
: I 5
:55::5:L
K
55::55::
I 55::
5:L
5 5 5:
5::+ 7 2 2 2 5: (
2
&5:
I 5
I :
K
I
(2
" "
5,) 5&&::+ 2 3) 2 2
I
+
4 4 -$$# +' 8
, 2 ,+ 6
2 ; "
2 2 2
O '
5H H H:
4 ,)
O5
!
!
: F
5
:K
5
: I
5
:!
!
L
82B ++ 2 ; * " 2 2'
O5
!
: O5
!
: *
/
I +
K !
!
I !
!
L
5&&&:
!
(2 (
F 5
:
5H:
!
F 5!
!
:
( F
+
F
u
+
2 2
5: 2 ( 2 " 2
2 +
+ ) ( 2 2 ) 8 1+ '
5* I +
:5 I :
*
F
5 2
+
" (2 ":+
D ( 2 2 ' 4 ,) 5: F
#
5
4 8 1+:+
-
,) 8 1+ ( F
5H:
l5: F O5
l5:
5:
5
5::
5::
@, 2
F
5
:
( F
5
:
5
:
F5
5
:
5
::
I 5
5
:
5
::
I
5
:5
:
2
I
I
5/:
,) 2 4
2 +
2 5&&&: 5&, :
" #
*5
" #I 5 I :
: I 5 I :+
" #
6 '
+ ) 2 ( ; )
H
+ 8 2 ; 2 ( ,
H
+ &) 2
&2 ; 2 2 ; H
2 2
F
5:
5
:
4 4 ,,!. ++' ; " - " , 2 2 5 I -: 5
-
: 2
5H: + 5(: 2 2
- 2 2 H
5:
5H:
5H:+ 8 F
5H: F
5H:
2
# ( F " )+
5H: ( 2'
*-
" #I
" #I -
I
I+
K
I
L I +
4
( 2 2 ( A'
I K-
I
L I
G
5 I :
G
I -
G
" # " #2 2 A 2 4 ( '
5 *-:
" #K*- I +
L5 I -:
IK*-5 I : I *5
I I : I +
5 I :L
4
F K*- I +
L5 I -:5 *-:
F K*-5 I : I *5
I I : I +
5 I :L5 *-:
( 2 2
" #I
&)
" #I
/
(2
F =;
5=;
:
6
+
(2 - "
2 - " 2 ( 4
2 2
# +
!1$5 5 6 ! 7 %$-" )889* P6 2 6 #
2 - (2 = ,) #6#6 #A < 2Q %
& R 6 9 0# +
-"!5 6 )889* P6 2 2 (2 ,) #A
; ,) Q % & 6#
9 1# M+
,$#!5 : 6 )889* P6 2 6 2 - (2 = ,) )#
Q % & R 6 9 # 0+
,$#!5 : 6 )88* PA, ) (2
;Q $0 "0#"!+
65 7 )88* P C =2 6 < - (2 = Q
% 2 ) $9 !"#$ +
65 7 )88;* P =2 Q 2 = ,
=2+
5 3 6 )88;* P <) @ A <
= ) = Q % ) 11$#$+
!,!5 : ! <!$ )99* P@ 2 ) 2 )
2 & 2 Q & / + 1 + $ #9+
!$5 )8;;* P6 # 2 ,) B ;#
Q D 7 = )+
!$5 ! $= )88;* PB ; < 2G
6 Q ( 7 +
!"5 ! 6 0>0! )88;* P<) )
< 2 < = Q MM !"0# M+
&$.5 ! !" )%- $"* )8;8* P =2
)Q > ) , =+ 8 +
!.,5 6 ! -,0 )889* P6 6 2 - (2 = ' <
< 6 =2 Q % & R 6
9 #+
On the accuracy of the estimated policy function
using the Bellman equation
Wilfredo L. Maldonado∗
Department of Economics, Universidade Federal Fluminense
Rua Tiradentes 17, Ing´a, Niter´oi
CEP 24210-510, RJ, BRAZIL Email: willy@impa.br
B. F. Svaiter†
Instituto de Matem´atica Pura e Aplicada
Est. Dona Castorina 110, Jardim Botˆanico, Rio de Janeiro
CEP 22460-320, RJ, BRAZIL E-mail: benar@impa.br
September 17, 2002
Abstract
In this paper we give explicit error bounds for approximations of the optimal policy function in the stochastic dynamic programming problem. The approximated policy function is obtained by using the Bellman equation with an approximated value function and the error bounds depend on the primitive data of the problem. Neither differen-tiability of the return function nor interiority of solutions is required. Furthermore, similar error bounds are obtained when the maximiza-tion in the Bellman equamaximiza-tion and the computamaximiza-tion of the associated policy function are performed inexactly. This shows the robustness of the method and provides a stopping criterium for computational implementations.
Keywords: Stochastic dynamic programing problem, estimation of the policy function
JEL Classification Numbers: C61, D90
∗I would like to thank the CNPq of Brazil for financial support 300059/99-0(RE). †Partially supported by CNPq Grant 301200/93-9(RN) and by PRONEX– Optimization. Finally, we would like to thank the valuable comments and suggestions of Kenneth Judd.
1
Introduction
It is well known that the Bellman Principle of Optimality allows to de-fine an iterative method to estimate the value function and the optimal policy function in stochastic dynamic programming problems (Christiano [4], Tauchen [11], Santos and Vigo-Aguiar [9]). This method is based on a contraction mapping and provides an efficient algorithm to estimate the value function with high precision. However, until now, only asymptotic convergence results, without numerical error bounds, were obtained for the estimated policy function with this method. For example, Stokey and Lucas with Prescott ([10], theorem 9.9) established the pointwise convergence (and uniform convergence if the domain is a compact set) to the optimal policy function under the assumption of strict concavity of the return function.
We will prove that the error of the estimated policy function in then-th step of the contractive method is bounded by
2
η1
F∞
1−β
1/2
βn/2.
whereF is the return function,β is the discount factor, η1 is the parameter
of strong concavity of F and n is the number of iterations. This estimate holds for a constant function equal to zero as an initial guess for the value function.
There exist other approaches for obtaining good estimates for the optimal policy function. For example, the Euler equation grid method (Baxteret al. [1], Coleman [2, 3]), the parameterized expectations method (Marcet and Marshall [6]) and projection methods (Judd [5]). Again, only asymptotic convergence to the optimal policy function was proved for these methods.
Bounds for the distance between the optimal policy function (of the original problem) and the exact optimal policy function of a discretized (piecewise linear) version of the problem were obtained by Santos and Vigo-Aguiar [9].
In Santos [8], Euler equation residuals were used to obtain error bounds for an approximated policy function. Either an assumption on the repeated iterations of the approximated policy function (condition NDIV) or a bound on the second derivative of the return function evaluated at the optimal policy function was necessary. Existence of interior solutions and twice differentiablity of the return function were also required.
The error bounds presented in this paper only require boundedness and strong concavity of the return function. These assumptions are quite general and were used by Santos and Vigo-Aguiar [9] and Santos [8]. We require
neither differentiability of the return function nor existence of interior solu-tions.
We also prove the robustness of the contractive method by considering the use of an inexact operator defining the contractive method or inexact solutions in each maximization process. In this case we again obtain error bounds.
Our result has a direct consequence for the use of the contractive method to estimate the policy function: the number of iterations required to attain a given precision is computed in advance, using only some primitive data of the problem.
This paper is organized into five sections. Section 2 describes the frame-work we will consider and the hypotheses assumed. Section 3 states the main theorem, which provides an error bound for the estimated policy function when the contraction method based on Bellman’s principle is used. Sec-tion 4 shows the robustness of this method under small numerical errors. Conclusions are given in Section 5 and the proofs are given in the appendix.
2
The framework
The stochastic dynamic programming problem is defined using the following elements: the set of values for the endogenous state variablesX ⊂Rl(which is a convex Borel set), the set of values for the exogenous shocks Z ⊂ Rk (which is a compact set); both are measurable spaces with theirσ-algebras denoted by Xand Zrespectively. The evolution of the stochastic shocks is given by the transition functionQdefined on (Z,Z) with the Feller property. A (measurable) set Ω⊂X×X×Z describing the feasibility of decisions,i.e. if (x, z)∈X×Z are the current values of the state variable and the shock theny ∈X is feasible for the next period if and only if (x, y, z)∈Ω. From this we can define the correspondence Γ : X×Z → Rl by Γ(x, z) = {y ∈
X; (x, y, z) ∈ Ω}. The one-period return function F : Ω→ R is such that
F(x, y, z) is the current return ify is chosen for the next period from (x, z). The discount factor is β ∈ (0,1). With all these elements, the stochastic dynamic programming problem is to find a sequence of contingent plans (ˆxt)t≥1(where for allt≥1, ˆxt:Zt→X is a measurable function) such that
it solves the following maximization:
v(x0, z0) = Max
∞
t=0
Zt
βtF(xt, xt+1, zt)Qt(z0, dzt)
subject to (xt, xt+1, zt)∈Ω for allt≥0
(x0, z0)∈X×Z given
The following hypotheses will be used in this work.
Hypothesis 1. The correspondence Γ is nonempty, compact-valued,
con-tinuous and for allx, x´∈X,z∈Z and α∈[0,1] it satisfies:
αΓ(x, z) + (1−α)Γ(x´, z)⊂Γ(αx+ (1−α)x´, z).
Hypothesis 2. The function F is bounded, continuous and there exists
η1>0 such thatF(x, y, z) + (η1/2)|x|2 is a concave function in (x, y).
Hipothesis 1 as well as boundedness and continuity of F are quite gen-eral in models with bounded returns and technologies with non-increasing returns. The second part of hypothesis 2 (also called the strong concav-ity1 hypothesis of F(., ., z)) was also used by Santos and Vigo-Aguiar [9] for obtaining error bounds on their estimations. Also, Montrucchio [7] used that hypothesis for obtaining Lipschitz continuity of the policy functions. Conditions on the utility functions and technologies that ensures the strong concavity in optimal growth models are given by Venditti [12].
Under these assumptions, the value functionvis well-defined and satisfies
v∞≤ F∞/(1−β). From now on, · stands for · ∞.
3
The main result
In this section we will state the main theorem that estimates the approxima-tion error in the optimal policy funcapproxima-tion computed through the contractive method defined below. This estimate is obtained using primitive data of the problem. Let T be the operator on C(X ×Z) (the set of continuous and bounded functions defined inX×Z with the topology induced by the supremum norm) defined by:
T V(x, z) = Max
{y∈X; (x,y,z)∈Ω}
F(x, y, z) +β
Z
V(y, z´)Q(z, dz´).
It is well known (see Stokey and Lucas with Prescott [10]) that under hy-potheses 1 and 2 this operator is a contraction mapping with modulus β
and fixed point v (the value function). The contractive method based on this operator is defined as follows: letv0 ∈C(X×Z) be a concave function
inx and define the sequence (vn)n≥0 by:
vn+1(x, z) =T vn(x, z), ∀(x, z)∈X×Z,∀n≥0 (1)
1A function f : C ⊂
Rn
→ R is strongly concave if there exists η > 0 such that
f(x) + (η/2)|x|2 is a concave function. The greatest η with that property is called the
parameter of strong concavity.
Lemma 3.1 With hypotheses 1 and 2, each vn defined by (1) is strongly
concave inxwith parameterη1for alln≥1. In particular the value function
v is also strongly concave in x with the same parameter.
Since for each n≥1, vn is strongly concave, we can define:
gn(x, z) = Argmax
{y∈X; (x,y,z)∈Ω}
F(x, y, z) +β
Z
vn(y, z´)Q(z, dz´), (2)
and the optimal policy is a function given by:
g(x, z) = Argmax
{y∈X; (x,y,z)∈Ω}
F(x, y, z) +β
Z
v(y, z´)Q(z, dz´).
The main theorem With hypotheses 1 and 2, the sequence (vn, gn)n≥1,
defined by (1) and (2) satisfies:
g−gn ≤
2
η1
v−vn
1/2
.
In particular,
g−gn ≤
2
η1
F
1−β +v0
1/2
βn/2.
A similar result can be proved substituting Hypothesis 2 by the following:
Hypothesis 3 The function F is bounded, continuous and there exists
η2>0 such thatF(x, y, z) + (η2/2)|y|2 is a concave function in (x, y).
Theorem 3.2 With hypotheses 1 and 3, the sequence(vn, gn)n≥0 defined by
(1) and (2) satisfies:
g−gn ≤
2β η2
v−vn
1/2
.
In particular,
g−gn ≤
2
η2
F
1−β +v0
1/2
β(n+1)/2.
4
Robustness of the approximation method
In this section we will show that the approximation method is robust by jointly considering errors in computing the T operator and errors in com-puting the maximizer in each iteration.
Suppose thatT is performed using a numerical method and thatT, an “approximated” operator, is computed.
Hypothesis 4 LetT :C(X×Z) →C(X×Z). Assume that there exists
ε≥0 such that for all f ∈C(X×Z)
T(f)−T(f) ≤ε.
Now let (˜vn)n≥0 be a sequence generated by the rule
˜
vn+1 =T(˜vn).
Proposition 4.1 If the correspondenceΓis nonempty, compact-valued,and
continuous, the function F is bounded and continuous, and T satisfies
Hy-pothesis 4, then the sequence(˜vn)n≥0 satisfies
v˜n−v ≤ ε
1−β +β n
F
1−β +˜v0
.
Remark The aplicationTdoes not have to satify the usual assumptions of
monotonicity (f ≤g⇒T f ≤T g ) and discounting (T(f+a) =T f +βa, a∈
R) (see Santos and Vigo [9]). Since T represents the “inexact” operator T
these assumptions are hard to check (and may not hold) when rounding and chopping errors are embedded into the analysis.
Proposition 4.1 also says that if
βn
F
1−β +v˜0
≪ ε
1−β
then more than n iterations may not appreciably improve the accuracy of the estimated value function.
In addition, suppose that an inexact maximization is used to compute the policy associated to ˜vn. That is, take a tolerance τ ≥ 0 and define
Gn(x, z) as those ˜y∈Γ(x, z) such that∀y ∈Γ(x, z)
F(x,y, z˜ ) +β
Z
˜
vn(˜y, z′)Q(z, dz′)≥F(x, y, z) +β
Z
˜
vn(y, z′)Q(z, dz′)−τ.
(3)
Observe thatGncan be a correspondence. In general, practical computation
does not providethe whole set Gn(x, z). Instead, the inexact maximization
will provide some ˜yn∈Gn (x, z). Even so, we have the following estimation.
Theorem 4.2 Suppose that hypotheses 1, 2 and 4 are satisfied. Let ˜gn :
X×Z → X be a selection of Gn, that is, ˜gn(x, z) ∈Gn(x, z) for all (x, z).
Then,
g−g˜n ≤
4
η1
v−˜vn+τ
1/2
.
In particular,
g−g˜n ≤
4
η1
ε
1−β +β n
F
1−β +˜v0
+τ
1/2
.
Remark Theorem 4.2 says that ifn is such that
βn
F
1−β +v˜0
≪ ε
1−β + η1τ
4
then more than n iterations may not appreciably improve the accuracy of the estimated policy.
5
Conclusions
In this paper we show that the contraction method defined from the Bell-man equation provides accurate estimates for the optimal policy function of the stochastic dynamic programming problem. An error bound for the estimate using the n-th iteration is explicitly constructed. It only depends on the norm and the parameter of strong concavity of the return function and the discount factor. Neither differentiability of the return function nor interiority of the solution are required. This can be useful when the model involves piece-wise linear taxation/subsidies or short-run Leontief technolo-gies because in these cases the return function may not be differentiable. Also interiority of solutions can not be guaranteed if (for example) Inada’s condition is not considered.
When inexact computations are performed in the calculation ofT (mak-ing a discretization of the state space, for example) or in the maximizer of each iteration we also provide an error bound for the estimation of the value and policy functions. The intuition is quite simple: Perturbations of a con-traction mapping are stable even though a fixed point for the perturbated mapping may not exist.
The error bounds presented in this paper can be used for evaluating
a priori the number of iterations needed in practical computations of the
Bellman method. For example, following the notation of Section 4, let ε
be the error on the T operator and τ be the error on the maximization procedure used to compute the associated policy. Ifnis such that
βn
F
1−β +v˜0
≪ ε
1−β + η1τ
4
then more than n iterations may not appreciably improve the accuracy of the estimated policy.
Finally, it is important to note that the error bounds found in this work can be used combining the Bellman method with other methods (which in general are faster than the Bellman one). Suppose that it is found an approximation for the value function ˆv0 by any method and let ˆv1 =Tvˆ0.
Using the inequalityv−vˆ0 ≤ v−vˆ1+ˆv1−vˆ0, the contraction property
ofT and the main theorem we will obtain:
g−ˆg0 ≤
2
η1(1−β)
ˆv1−ˆv0
1/2
,
where
ˆ
g0(x, z) = Argmax
{y∈X; (x,y,z)∈Ω}
F(x, y, z) +β
Z
ˆ
v0(y, z´)Q(z, dz´).
A
Appendix
To prove Lemma 3.1, let us show the following:
Lemma A.1 F(x, y, z) + (η/2)|x|2 is a concave function in (x, y) if and
only if for all (xi, yi, z)∈Ω, i= 1,2 and for all α∈[0,1] it holds that:
F(xα, yα, z)≥αF(x1, y1, z) + (1−α)F(x2, y2, z) +
η
2α(1−α)|x1−x2|
2,
where xα =αx1+ (1−α)x2 andyα =αy1+ (1−α)y2.
Proof:
(⇒) By hypothesis:
F(xα, yα, z)+η 2|x
α|2≥α[F(x
1, y1, z)+
η
2|x1|
2]+(1−α)[F(x
2, y2, z)+
η
2|x2|
2],
expanding the square of the left side and simplifying it results in:
F(xα, yα, z)≥αF(x1, y1, z) + (1−α)F(x2, y2, z) +
η
2α(1−α)|x1−x2|
2.
(⇐) Completely analogous.
Proof of Lemma 3.1It will be sufficient to prove that ifV(., z) is a concave
function thenT V(., z) is a strongly concave function with parameterη1. Let
x1, x2∈X,α∈[0,1], xα=αx1+ (1−α)x2 and fori= 1,2 letyi∈Γ(xi, z)
be such that:
T V(xi, z) =F(xi, yi, z) +β
Z
V(yi, z´)Q(z, dz´).
Then, using hypotheses 1, 2 and Lemma A.1 we have that:
T V(xα) ≥ F(xα, αy1+ (1−α)y2, z) +β
Z
V(αy1+ (1−α)y2, z′)Q(z, dz′)
≥ αF(x1, y1, z) + (1−α)F(x2, y2, z) +
η
2α(1−α)|x1−x2|
2+
β
Z
[αV(y1, z´) + (1−α)V(y2, z´)]Q(z, dz´)
= αT V(x1, z) + (1−α)T V(x2, z) +
η
2α(1−α)|x1−x2|
2.
Since the set of strongly concave functions is a closed set with the topology induced by the sup norm it follows that the fixed point of T is strongly concave.
To prove the main theorem, we will need the following lemmata.
Lemma A.2 f :C ⊂Rn→R (C is a convex set) is strongly concave with
modulusη if and only if for allx1, x2∈C it holds that:
f(αx1+ (1−α)x2)≥αf(x1) + (1−α)f(x2) + (η/2)α(1−α)|x1−x2|2.
Proof: Analogous to the proof of lemma A.1.
Lemma A.3 Let f :C ⊂Rn→R(C is a convex set) be a strongly concave
function with modulusη. If x∗ =Argmaxx∈Cf(x) then
f(x)≤f(x∗)−η
2|x−x
∗|2, ∀x∈C.
Proof: Letx∈Candα∈(0,1). By definition ofx∗and the characterization of strong concavity given in Lemma A.2 we have:
f(x∗)≥f(αx∗+ (1−α)x)≥αf(x∗) + (1−α)f(x) +η2α(1−α)|x−x∗|2
⇒f(x∗)≥f(x) + η
2α|x−x∗|2,
makingα →1 we obtain the result.
Proof of the main theorem.
Let us fix some notations. Let vn ∈ C(X ×Z) be the n-th iteration
(n≥1) of theT operator from some initial concave functionv0 ∈C(X×Z).
Let
gn(x, z) = Argmax{y∈X; (x,y,z)∈Ω}F(x, y, z) +β
Z
vn(y, z´)Q(z, dz´),
φn(x, y, z) =F(x, y, z) +β
Z
vn(y, z´,)Q(z, dz´),
φ(x, y, z) =F(x, y, z) +β
Z
v(y, z´)Q(z, dz´).
By lemma 3.1 the functions φ(x, ., z) and φn(x, ., z) are strongly concave
with parameterβη1. Then by lemma A.3 we have that:
φ(x, g(x, z), z)≥φ(x, gn(x, z), z) +βη1
2 |g(x, z)−gn(x, z)|
2,
φn(x, gn(x, z), z)≥φn(x, g(x, z), z) + βη1
2 |g(x, z)−gn(x, z)|
2.
Summing up the above inequalities we obtain that:
β{
Z
[(vn−v)(gn(x, z), z´)+(v−vn)(g(x, z), z´)]Q(z, dz´)} ≥βη1|g(x, z)−gn(x, z)|2
⇒2v−vn ≥η1|g(x, z)−gn(x, z)|2
this inequality holds for all (x, z)∈X×Z, so we conclude:
g−gn ≤[
2
η1
v−vn]1/2.
Finally, using thatT is a contraction it results thatv−vn ≤βnv−v0.
The bound is obtained sincev ≤ F/(1−β).
Proof of theorem 3.2
Under hypothesis 3, φn(x, ., z) and φ(x, ., z) are strongly concave with
parameterη2. Then by lemma A.3 we have that:
φ(x, g(x, z), z)≥φ(x, gn(x, z), z) + η2
2|g(x, z)−gn(x, z)|
2,
φn(x, gn(x, z), z)≥φn(x, g(x, z), z) + η2
2 |g(x, z)−gn(x, z)|
2.
Using the same reasoning as in the proof of the main theorem we conclude that:
g−gn ≤[
2β η2
v−vn]1/2≤
2
η2
F
1−β +v0
1/2
β(n+1)/2.
Proof of Proposition 4.1
By our assumptions,Tis aβ-contraction onC(X×Z) with fixed pointv. Using also the definition of the sequence (˜vn)n≥0, the triangular inequality
and Hypothesis 4 we get
˜vn+1−v = T(˜vn)−v
≤ T(˜vn)−T(˜vn)+T(˜vn)−v
≤ ε+β˜vn−v.
Hence
˜vn−v ≤
n−1
j=0
εβj+βnv˜0−v ≤ε/(1−β) +βn˜v0−v.
Proof of theorem 4.2
Let
˜
φn(x, y, z) =F(x, y, z) +β
Z
˜
vn(y, z´)Q(z, dz´).
Then
˜
φn(x,g˜n(x, z), z)≥φ˜n(x, g(x, z), z)−τ.
With hypotheses 1 and 2 we have:
φ(x, g(x, z), z)≥φ(x,˜gn(x, z), z) + βη1
2 |g(x, z)−g˜n(x, z)|
2.
Adding up these inequalities and following the same procedure as in the proof of the main theorem we will obtain
g−g˜n ≤
4
η1
v−˜vn+τ
1/2
.
Finally, using Proposition 4.1 it will result that
g−g˜n ≤
4
η1
ε
1−β +β n
F
1−β +˜v0
+τ
1/2
.
References
[1] Baxter, M., M. J. Crucini and K. G. RouwenhorstSolving the Stochas-tic Growth Model by a Discrete-State-Space, Euler-Equation Approach, Journal of Business & Economic Statistics Vol. 8-1, pp 19–21, 1990.
[2] Coleman, W. J.Solving the Stochastic Growth Model by Policy-Function
Iteration,Journal of Business & Economic Statistics Vol. 8-1, pp 27–29,
1990.
[3] Coleman, W. J.Equilibrium in a Production Economy with an Income Tax, Econometrica Vol 59, pp 1091–1104, 1991.
[4] Christiano L.Solving the Stochastic Growth Model by Linear-Quadratic
Approximation and by Value Function Iteration,Journal of Business &
Economic Statistics Vol. 8-1, pp. 23–26, 1990.
[5] Judd, K. L. Projection Methods for Solving Aggregate Growth Models, Journal of Economic Theory Vol. 58, pp. 410-452, 1992.
[6] Marcet A. and D. Marshall Solving Nonlinear Rational Expectations Models by Parameterized Expectations: Convergence to Stationary
So-lutions, Working Paper No. 76, Department of Economics, Universitat
Pompeu Fabra, 1994.
[7] Montrucchio L.Lipschitz continuous policy functions for strongly
con-cave optimization problems, Journal of Mathematical Economics Vol.
16, pp 259-273, 1978.
[8] Santos M. Accuracy of Numerical Solutions using the Euler Equation
Residuals, Econometrica Vol. 68-6, pp 1377-1402, 2000.
[9] Santos M. and J. Vigo-AguiarAnalysis of a Numerical Dynamic
Pro-gramming Algorithm Applied to Economic Models, Econometrica Vol.
66-2, pp 409-426, 1998.
[10] Stokey N. and R. Lucas with E. Prescott Recursive Methods in
Economic Dynamics,M.A.: Harvard University Press, 1989.
[11] Tauchen G. Solving the Stochastic Growth Problem by Using
Quadra-ture Methods and Value-Function Iterations, Journal of Business and
Economic Statistics vol. 8, pp 49–51, 1990.
[12] Venditti A.Strongly concavity properties of indirect utility functions in
multisector optimal growth models,Journal of Economic Theory vol. 74,
pp 349-367, 1997.
