EX-DRL: Hedging Against Heavy Losses with EXtreme Distributional Reinforcement Learning

Parvin Malekzadeh1, Zissis Poulos2, Jacky Chen1, Zeyu Wang1, Konstantinos N. Plataniotis1
1University of Toronto, 2 York University
arXiv Code
MY ALT TEXT

Visual illustration of the proposed EX-DRL framework.

Abstract

Recent advancements in Distributional Reinforcement Learning (DRL) for modeling loss distributions have shown promise in developing hedging strategies in derivatives markets. A common approach in DRL involves learning the quantiles of loss distributions at specified levels using Quantile Regression (QR). This method is particularly effective in option hedging due to its direct quantilebased risk assessment, such as Value at Risk (VaR) and Conditional Value at Risk (CVaR). However, these risk measures depend on the accurate estimation of extreme quantiles in the loss distribution’s tail, which can be imprecise in QR-based DRL due to the rarity and extremity of tail data, as highlighted in the literature.

To address this issue, we propose EXtreme DRL (EX-DRL), which enhances extreme quantile prediction by modeling the tail of the loss distribution with a Generalized Pareto Distribution (GPD). This method introduces supplementary data to mitigate the scarcity of extreme quantile observations, thereby improving estimation accuracy through QR.

Comprehensive experiments on gamma hedging options demonstrate that EX-DRL improves existing QR-based models by providing more precise estimates of extreme quantiles, thereby improving the computation and reliability of risk metrics for complex financial risk management.

Hedging with EX-DRL

Our proposed EX-DRL approach is compatible with any QR-based method.

Drawing inspiration from the effectiveness of QR-D4PG, a recent actor-critic QR-based DRL method, we propose an extension, EX-D4PG, which replaces the quantile distribution used for the target distribution in QR-D4PG with our mixture model target distribution. Unlike QR-D4PG, EX-D4PG’s critic not only reflects the usual dynamics of the return distribution but also emphasizes extreme event modeling.

MY ALT TEXT
Figure 1: The network graph of the proposed EX-D4PG framework.

Experimental Results

We conducted a series of hedging experiments to compare the performance of EX-D4PG with QR-D4PG.

First Network Image Second Network Image
Figure 2: VaR and CVaR values when volatility = 0.3 (left) and volatility = 0.5 (right).

MY ALT TEXT

First Network Image Second Network Image
Figure 3: Comparison of VaR95 (left) and CVaR95 (right) at different volatility levels.

First Network Image Second Network Image
Figure 4: Comparison of gamma hedge ratio using VaR95 (left) and CVaR95 (right) at various volatility levels.

BibTeX


            @article{malekzadeh2024ex,
            title={EX-DRL: Hedging Against Heavy Losses with EXtreme Distributional Reinforcement Learning},
            author={Malekzadeh, Parvin and Poulos, Zissis and Chen, Jacky and Wang, Zeyu and Plataniotis, Konstantinos N},
            journal={arXiv preprint arXiv:2408.12446},
            year={2024}
            }