Our main contributions can be summarized as follows:

1. (i)

   We show that under model-class symmetries that are natural for learning via embedding (unitary invariance and closure under pinching operation), the QMLP objective reduces to a classical KL divergence objective on eigenvalues (Proposition 1), and provide a geometric interpretation of this equivalence in terms of a refined version of the quantum Pythagorean theorem (Theorem 1).

2. (ii)

   We obtain non-asymptotic upper bounds (Theorem 2) quantifying the expected rate of convergence of QMLP (and MLP) to the target state both in trace norm and quantum relative entropy. We also obtain the corresponding concentration inequalities. These bounds depend on the dimension $d$ of the Hilbert space, the prompt length $n$, and the minimal positive eigenvalue of the embedded quantum state corresponding to the true data distribution. For instance, our bounds show that the rate of convergence in trace norm is $\tilde{O}(d/\sqrt{n})$ (up to an additive approximation slack factor) in general and $\tilde{O}(d^{3}/n)$ when the class of quantum models is sufficiently expressive, where $\tilde{O}(\cdot)$ hides logarithmic factors and dependence on minimal eigenvalue. Under additional conditions on the expressivity of quantum models measured in terms of quantum relative entropy, the corresponding rates of convergence under quantum relative entropy are $\tilde{O}(d/\sqrt{n})$ and $\tilde{O}(d^{2}/n)$, respectively.

The proof of Proposition 1 relies on elementary arguments involving quantum measurements and data-processing inequality. The proof of quantum Pythagorean theorem involves adapting the proof of the corresponding classical result given in Csiszár and Shields 2004 to the non-commutative setting. The essential ingredients in the proof of Theorem 2 are the variational expression for quantum relative entropy (see (2)), matrix Hoeffding and Bernstein inequalities, a second order Taylor expansion for quantum relative entropy and Proposition 2, which relates trace norm between QMLP and embedded state (corresponding to true distribution) to terms involving quantum relative entropy. Our bounds are tailored to embedded empirical states and QMLP, giving explicit dependence on embedding dimension and spectral conditioning. This yields prediction-oriented bounds relating the prompt length and the embedding dimension, which differs from existing tomography-oriented bounds, e.g., (Koltchinskii and Xia 2015; Ferrie and Blume-Kohout 2016).

Several recent studies have attempted to unlock the learning phenomenon in LLMs from different viewpoints, e.g., modeling the transformer as a finite state machine and quantifying the ability of transformers for learning via approximation of linear functions. For instance, Basu et al. 2023 considers Markov models for both data and transformer and shows that the average empirical loss as measured by the log loss function converges to the conditional entropy asymptotically, or equivalently that the asymptotic regret in prediction vanishes. Edelman et al. 2024 show that ICL can be explained by the emergence of statistical induction heads in transformers, which implement simple Markovian predictors over token sequences, revealing ICL as the activation of low-complexity statistical rules encoded during training. Jeon et al. 2024 provide an information-theoretic characterization of ICL, showing that in-context predictions in transformers correspond to optimal inference under implicit information constraints, where the prompt supplies limited mutual information about the task that is efficiently exploited by the pretrained model. Sander and Peyré 2025 investigate the expressiveness of transformers for next token prediction by considering embedding of tokens into a reproducing kernel Hilbert space (RKHS). Pan et al. 2025 proposed a theoretical framework for LLM scaling laws and knowledge acquisition by modeling training as an information-theoretic compression process governed by a hierarchical generative model of syntax and knowledge.

Embedding probability distributions into Hilbert spaces for inference has been explored in a variety of other contexts. For instance, the mean embedding (Berlinet and Thomas-Agnan 2004; Muandet et al. 2017) and the associated maximum mean discrepancy (MMD) metric has been used for designing kernel based two-sample tests (Gretton et al. 2006), defining independence/dependence measures (Fukumizu et al. 2004), feature extraction (Smola et al. 2007) and graphical models (Song et al. 2010) among other applications. The covariance embedding (Bach 2023) of probability distributions into density operators has also found several applications. To mention a few, Hoyos-Osorio and Sanchez-Giraldo 2024 considered a representation Jensen-Shannon divergence defined in terms of von-Neumann entropy of embedded density operators, Kachaiev and Recanatesi 2024 proposed unsupervised learning of the kernel via maximum von-Neumann entropy, and Santoro and Panaretos 2025 proposed a kernel-based likelihood test utilizing both the mean and covariance embedding. The covariance embedding of probability distributions and the associated QMLP also captures at a high-level the problem of prediction in quantum LLMs (Basile and Tamburini 2017), where the expressivity of quantum architecture used for prediction is reflected in the class of models over which QMLP is optimized. Another related topic corresponds to quantum generative modeling and simulation of probability distributions, in which the hope is to use quantum models to efficiently simulate probability distributions which are difficult to model classically. Two popular frameworks used in this regard pertains to quantum circuit Born machines (Liu and Wang 2018; Benedetti et al. 2019) and quantum Boltzmann machines (Benedetti et al. 2017; Kieferová and Wiebe 2017; Amin et al. 2018), which uses quantum circuits and thermal (or Gibbs) states, respectively, to achieve the modeling objective.

Let $\mathcal{X}$ be a discrete set or a subset of $\mathbb{R}^{d}$. Denote the set of all Borel probability measures whose support is contained in $\mathcal{X}$ by $\mathcal{P}(\mathcal{X})$. Let $\mu$ denote a sigma-finite measure on $\mathcal{X}$. For $P\ll\mu$ (i.e., $P$ absolutely continuous with respect to $\mu$), let $p$ denote its Radon-Nikodym derivative with respect to $\mu$. Unless specified otherwise, we will take $\mu$ to be the counting measure when $\mathcal{X}$ is discrete or Lebesgue measure when $\mathcal{X}\subseteq\mathbb{R}^{d}$, in which case $p$ becomes the usual probability mass function (pmf) or probability density function (pdf).

Let $\mathbb{H}$ denote a complex (or real) separable Hilbert space. Unless specified otherwise, we will assume that $\mathbb{H}$ is complex. When $\mathbb{H}$ is of dimension $d$, we will denote it as $\mathbb{H}_{d}$. For an inner product $\langle\cdot,\cdot\rangle_{\mathbb{H}}$ on $\mathbb{H}$, we will use the physics inspired convention in which the inner product is anti-linear (or conjugate) in the first argument and linear in the second. Whenever there is no confusion, we will denote $\langle\cdot,\cdot\rangle_{\mathbb{H}}$ simply by $\langle\cdot,\cdot\rangle$. The space of continuous linear operators on $\mathbb{H}$ is denoted by $\mathcal{L}(\mathbb{H})$, equipped with operator norm topology. The set of density operators or quantum states is denoted as $\mathcal{S}(\mathbb{H})$, i.e., a self adjoint operator $\rho$ such that $\rho\geq 0$ and $\operatorname{Tr}{\left[\rho\right]}=1$. We will employ the bra-ket notation from quantum information theory (see e.g. Hayashi 2016). To mention it briefly, let $h$ and $\tilde{h}$ be any two elements of some Hilbert space $\mathbb{H}$. Then, $|h\rangle$, $\langle\tilde{h}|$ and $|h\rangle\langle\tilde{h}|$ corresponds, respectively, to an element $h\in\mathbb{H}$, a linear functional on $\mathbb{H}$ with action $\langle\tilde{h}||h\rangle=\langle\tilde{h},h\rangle$, and a linear operator with action $|h\rangle\langle\tilde{h}||\bar{h}\rangle=\langle\tilde{h},\bar{h}\rangle|h\rangle$ for every $\bar{h}\in\mathbb{H}$, respectively. For $L\in\mathcal{L}(\mathbb{H})$, $\lambda_{L}$ denotes the vector composed of eigenvalues of $L$. When $\rho\in\mathcal{S}(\mathbb{H})$, $\lambda_{\rho}$ is a pmf. The support of $L\in\mathcal{L}(\mathbb{H})$, which is defined as the orthogonal complement of kernel of $L$ in $\mathbb{H}$, is denoted by $\mathrm{spt}(L)$. $L_{1}\ll L_{2}$ denotes that $\mathrm{spt}(L_{1})\subseteq\mathrm{spt}(L_{2})$ and $L_{1}\mathchoice{\mathrel{\hbox to0.0pt{\kern 5.0pt\kern-5.27776pt$\displaystyle\not$\hss}{\ll}}}{\mathrel{\hbox to0.0pt{\kern 5.0pt\kern-5.27776pt$\textstyle\not$\hss}{\ll}}}{\mathrel{\hbox to0.0pt{\kern 3.98611pt\kern-4.45831pt$\scriptstyle\not$\hss}{\ll}}}{\mathrel{\hbox to0.0pt{\kern 3.40282pt\kern-3.95834pt$\scriptscriptstyle\not$\hss}{\ll}}}L_{2}$ means that $\mathrm{spt}(L_{1})\subsetneq\mathrm{spt}(L_{2})$. For $a,b\in\mathbb{R}$, $a\vee b$ and $a\wedge b$ denote $\max\{a,b\}$ and $\min\{a,b\}$, respectively.

Next, we introduce the preliminary notions used in the paper.

The quantum relative entropy between $\rho,\sigma\in\mathcal{S}(\mathbb{H})$ is

The quantum relative entropy admits the variational expression (Petz 1988; Berta et al. 2015):

The measured relative entropy (Donald 1986; Piani 2009) between $\rho,\sigma\in\mathcal{S}(\mathbb{H})$ is defined as

where the supremum is over all finite sets $\mathcal{Z}$ and positive operator-valued measures¹¹ 1 A POVM $\left\{M_{z}\right\}_{z\in\mathcal{Z}}$ on a separable Hilbert space $\mathbb{H}$, indexed by a discrete $\mathcal{Z}$, is a set of positive semi-definite operators $M_{z}$ such that $\sum_{z\in\mathcal{Z}}M_{z}$ equals the identity operator on $\mathbb{H}$. (POVMs) $\left\{M_{z}\right\}_{z\in\mathcal{Z}}$ indexed by $\mathcal{Z}$. By data-processing inequality for quantum relative entropy, $\mathsf{D}\left(\rho\middle\|\sigma\right)\geq\mathsf{D}_{\mathsf{\mathsf{M}}}\left(\rho\middle\|\sigma\right)$. This shows that if two density operators $\rho$ and $\sigma$ are close in the quantum relative entropy, then the KL divergence between any two measured probability distributions is only lower (or equal). Moreover, when $\rho,\sigma>0$, equality holds if and only if $\rho$ and $\sigma$ commute, and then this common value also coincides with $\mathsf{D}_{\mathsf{KL}}\left(\lambda_{\rho}\middle\|\lambda_{\sigma}\right)$, where $\lambda_{\rho}$ denotes the probability mass function (pmf) composed of eigenvalues of $\rho$ and the KL divergence between probability measures $P,Q\in\mathcal{P}(\mathcal{X})$ is

When considering an embedding of probability distributions into Hilbert spaces, a natural possibility, which has been extensively studied in the context of RKHSs, is the mean embedding $P\mapsto\mathbb{E}_{P}[\varphi(X)]$, where $\varphi$ is the feature map corresponding to an RKHS (see Appendix B). However, as noted in Bach 2023, the mean embedding induces an MMD metric between probability distributions in the RKHS, which does not have a direct connection to log loss (and information theory). Accordingly, Bach 2023 proposed a second-order covariance embedding as a suitable framework to study interaction between probability distributions and perform sample-based inference. To define it, let $\mathbb{H}$ be a Hilbert space and $\varphi:\mathcal{X}\rightarrow\mathbb{H}$ be a Borel-measurable map such that $\left\|\varphi(x)\right\|_{\mathbb{H}}=1$ for all $x\in\mathcal{X}$. For a probability measure $P$ with density $p$ with respect to a dominating positive $\sigma$-finite measure $\mu$ (counting measure in the discrete case and Lebesgue measure in the continuous case), set

The covariance embedding $\phi:\mathcal{P}(\mathcal{X})\rightarrow\mathcal{L}(\mathbb{H})$ is then the linear map induced by (8). Here, we explore the utility of covariance embeddings for prediction. Then, $\rho_{p}\in\mathcal{S}(\mathbb{H})$ since it is positive semidefinite and has unit trace (see (74) in Appendix B). Hence, $\phi$ is a map of $\mathcal{P}(\mathcal{X})$ into $\mathcal{S}(\mathbb{H})$. Also, recall that $\mathcal{S}(\mathbb{H})$ is a subset of trace-class operators on $\mathbb{H}$, and hence lies in a Hilbert space within²² 2 For $\mathbb{H}_{d}$, the whole $\mathcal{L}(\mathbb{H}_{d})$ becomes a Hilbert space equipped with the Hilbert-Schmidt inner product. $\mathcal{L}(\mathbb{H})$ equipped with the Hilbert-Schmidt inner product $\langle\rho,\sigma\rangle\coloneqq\operatorname{Tr}{\left[\rho^{*}\sigma\right]}$, where $\rho^{*}$ denotes the adjoint of $\rho$. Several interesting properties of the map $\phi$ when $\varphi$ is the feature map corresponding to an RKHS with kernel $K$ were studied in Bach 2023. As noted therein, the image of $\phi$ lies within a Hilbert space that is isomorphic to the RKHS generated by the kernel $K^{2}$. Also, the quantum data-processing inequality yields $\mathsf{D}_{\mathsf{KL}}\left(P\middle\|Q\right)\geq\mathsf{D}\left(\rho_{p}\middle\|\rho_{q}\right)$.

Given the embedding of probability distributions as density operators in a Hilbert space (which we interpret as an internal representation), we model ICL as selecting a density operator $\sigma$ for prediction from a model class $\Sigma$ determined by the QMLP. To obtain a classical distribution on the output vocabulary, we apply a measurement channel (or POVM) $\mathcal{M}_{n}$, viewed as a quantum-to-classical channel. The appropriate embedding $\varphi$ and measurement are selected during training and then held fixed during ICL. Let

denote the embedding of the empirical distribution $\hat{p}_{n}$ of $x^{n}$, i.e., $\hat{\rho}_{n}(x^{n})=\rho_{\hat{p}_{n}}\coloneqq\phi(\hat{p}_{n})$. Let $\hat{P}_{n}$ and $\hat{Q}_{n}$ denote the measurement-output distributions corresponding to $\hat{\rho}_{n}(x^{n})$ and $\sigma$. Prediction accuracy is quantified by the minimal average log loss, which is equivalent to $\min_{Q\in\mathcal{Q}}\mathsf{D}_{\mathsf{KL}}\big(\hat{P}_{n}\|Q\big)$. Here, $\mathcal{Q}\coloneqq\{\mathcal{M}_{n}(\sigma):\sigma\in\Sigma\}$ and $\mathcal{M}_{n}(\sigma)$ denotes the classical distribution obtained as the output of the measurement channel with input $\sigma$. Since the measurement $\mathcal{M}_{n}$ is unspecified, one may consider the above optimization with measured relative entropy in place of KL divergence and replacing $\hat{P}_{n}$, $Q$ and $\mathcal{Q}$ by $\hat{\rho}_{n}(x^{n}),\sigma$ and $\Sigma$, respectively. However, the measured relative entropy is typically intractable, as it requires optimizing over all possible measurements, leading to a nonconvex inner problem without a closed form in general. We therefore define the QMLP given samples $x^{n}\in\mathcal{X}^{n}$ as any minimizer of

whenever the infimum is attained. This formulation has the additional advantage that it leads to convergence guarantees for the MLP in terms of quantum relative entropy, which is a stronger discrepancy measure than measured relative entropy.

When $\Sigma$ is a compact³³ 3 In what follows, the compactness assumption is used only to guarantee that the relevant infimum is achieved, and hence can be dispensed with if the minimum is known to exist. This is a useful relaxation, especially, in infinite dimensional Hilbert space settings, where compactness could be a restrictive assumption. set, the infimum in (10) is achieved. Moreover, when $\Sigma$ is also convex and⁴⁴ 4 Since $\Sigma$ is a compact convex set of positive semi-definite operators, it is easy to see that there exists an element $\sigma\in\Sigma$ such that $\sigma^{\prime}\ll\sigma$ for every $\sigma^{\prime}\in\Sigma$. Define $\mathrm{spt}(\Sigma)\coloneqq\mathrm{spt}(\sigma)$ for such a $\sigma$. To see that such a $\sigma$ exists, note that $\mathrm{spt}(\sigma_{1})\cup\mathrm{spt}(\sigma_{2})=\mathrm{spt}(\alpha\sigma_{1}+(1-\alpha)\sigma_{2})$ for any $\sigma_{1},\sigma_{2}\in\mathcal{S}(\mathbb{H})$ and $\alpha\in(0,1)$. Using this, it is possible to construct either a finite or infinite sequence of density operators $\sigma_{n}\in\Sigma$ with strictly increasing support. In the finite case, the last element of the sequence gives the desired $\sigma$ and in the infinite case, take $\sigma$ to be any limit point. $\mathrm{spt}\left(\hat{\rho}_{n}(x^{n})\right)=\mathrm{spt}(\Sigma)$, this minimum is unique. However, the last condition is restrictive and hard to ensure for all $x^{n}\in\mathcal{X}^{n}$. Accordingly, we call the ”QMLP” an arbitrary minimizer of (10), when the minimizer is not unique. The motivation behind considering QMLP stems from the possibility that with an appropriately chosen embedding $\varphi$ into a Hilbert space $\mathbb{H}_{d}$ of dimension $d\ll|\mathcal{X}|$, $\hat{\rho}_{n}(x^{n})$ is a good proxy of $\hat{p}_{n}$ for the prediction task while simultaneously enabling optimization over a subset of a much smaller Hilbert space $\mathcal{L}(\mathbb{H}_{d})$, i.e., over the space of $d\times d$ matrices. This occurs, for instance, when “neighboring symbols” which should lead to similar prediction outcomes are embedded to closely aligned elements in the Hilbert space, so that the span of $\{\varphi(x):x\in\mathcal{X}\}$ has a low effective dimension compared to orthogonal embedding. Another rationale, as will become evident in Section 4.2, is that the minimal eigenvalue of $\rho_{p}$ increases in general when multiple elements of $\mathcal{X}$ are mapped to non-orthogonal elements. From an optimization standpoint, (10) can be solved by optimization algorithms over Hilbert spaces that can exploit its rich structure such as orthogonality and inner products, and lack in $\mathcal{P}(\mathcal{X})$. In fact, there is a rich theory of optimization in Hilbert spaces (e.g., Houska and Chachuat 2017), and gradient-based algorithms can exploit the aforementioned structure while performing optimization.

Solving (11) involves optimization of quantum relative entropy over a class of density operators. It is of interest to determine conditions under which this optimization reduces to its relatively simpler classical analogue given in (7). We show that this indeed happens when $\Sigma$ has sufficiently rich structure. To state the condition precisely, let $\mathcal{E}_{\rho}\coloneqq\{|e_{i}(\rho)\rangle\}_{i=1}^{\infty}$ denote an orthonormal eigenbasis of $\rho$ and $\mathsf{P}_{i}(\rho)\coloneqq|e_{i}(\rho)\rangle\langle e_{i}(\rho)|$ denote the orthogonal projection corresponding to $|e_{i}(\rho)\rangle$. For $\Sigma\subseteq\mathcal{S}(\mathbb{H})$, define the following set obtained by pinching⁵⁵ 5 This is slightly different from standard pinching, where projectors of similar eigenvalues are combined, (e.g., Hayashi 2002). the elements of $\Sigma$ by the rank-one orthogonal projections $\{\mathsf{P}_{i}(\rho)\}_{i=1}^{\infty}$:

It is easy to verify that the operators in $\mathcal{J}\left(\mathcal{E}_{\rho},\Sigma\right)$ commute with $\rho$. Next, we call a set $\Sigma\subseteq\mathcal{L}(\mathbb{H})$ unitarily invariant if for all $\sigma\in\Sigma$ and unitaries $U$, $U\sigma U^{{\dagger}}\in\Sigma$. We have the following proposition that provides conditions on $\Sigma$ under which QMLP reduces to an MLP.

The proof of Proposition 1 given below in Section 5.1 is based on elementary arguments involving data processing inequality for quantum relative entropy, quantum measurements and the unitary invariance of $\Sigma$.

Proposition 1 has an interesting interpretation from the prism of quantum information geometry (see e.g., Amari and Nagaoka 2000; Csiszár and Matus 2003; Csiszár and Shields 2004; Hayashi 2016). To introduce the relevant notions, consider a non-empty compact convex set $\bar{\mathcal{S}}\subseteq\mathcal{S}(\mathbb{H})$. The information projection (or $I-$projection) of $\sigma\in\mathcal{S}(\mathbb{H})$ into $\bar{\mathcal{S}}$ is

if it exists. When $\mathrm{spt}(\bar{\mathcal{S}})\subseteq\mathrm{spt}(\sigma)$, $\mathsf{D}\left(\rho\middle\|\sigma\right)$ is a continuous⁶⁶ 6 In general, $\mathsf{D}\left(\rho\middle\|\sigma\right)$ is a lower semi-continuous convex function of its arguments (see e.g. Polyanskiy and Wu 2025). strictly convex function of $\rho$ over the compact set $\bar{\mathcal{S}}$. Hence, $\rho^{\star}_{\sigma,\bar{\mathcal{S}}}$ exists and is unique. We emphasize that information projection is a minimization of relative entropy in the first argument, while QMLP involves minimization over the second argument (which is thus a quantum reverse $I-$projection, (Csiszár and Shields 2004)).

Although solving QMLP and information-projection are distinct problems, we will see below that an interpretation of Proposition 1 can be obtained by combining them. This connecting link will be provided by the so-called quantum Pythagorean theorem from information geometry (see e.g., Amari and Nagaoka 2000; Hayashi 2016; Hayashi and Ito 2025). However, we will need a generalized version of this theorem, than available in the literature to the best of our knowledge.

To state our version of quantum Pythagorean theorem, we need to introduce a few notions. Call a set $\mathcal{S}\subseteq\mathcal{S}(\mathbb{H})$ of density operators linearly closed if for every $\rho,\tilde{\rho}\in\mathcal{S}$ and $\alpha\in\mathbb{R}$ such that $\alpha\rho+(1-\alpha)\tilde{\rho}\in\mathcal{S}(\mathbb{H})$. Denote the set of bounded linear operators by $\mathcal{B}(\mathbb{H})$, and let $\mathcal{C}=\{L_{i}\}_{i=1}^{k}\subseteq\mathcal{B}(\mathbb{H})$ be a finite set. For a given $\rho_{0}\in\mathcal{S}(\mathbb{H})$ and $\mathcal{C}$, the set of all density operators $\rho\in\mathcal{S}(\mathbb{H})$ such that $\operatorname{Tr}{\left[\rho L_{i}\right]}=\operatorname{Tr}{\left[\rho_{0}L_{i}\right]}$, for $1\leq i\leq k$, is called a mixture or linear family generated by $\mathcal{C}$ (and $\rho_{0}$), and denoted by $\mathcal{M}(\rho_{0},\mathcal{C})$. For a sub-density operator $\sigma_{0}$, i.e., $\sigma_{0}\geq 0$ and $\operatorname{Tr}{\left[\sigma_{0}\right]}\in[0,1]$, and $\mathcal{C}$, the set of all density operators of the form

is called exponential family generated by $\mathcal{C}$ (and $\rho_{0}$), and denoted by $\mathrm{Exp}(\rho_{0},\mathcal{C})$. Here, $c(\rho_{0},\bm{\beta},\mathcal{C})$ is the normalization factor to make $\operatorname{Tr}{\left[\rho\right]}=1$ and $e^{\log\sigma_{0}+\sum_{i=1}^{k}\beta_{i}L_{i}}$ is interpreted to act as the zero operator on the kernel of $\sigma_{0}$. Note that since $\mathcal{C}$ can contain non self-adjoint operators, and so not all $\bm{\beta}$ in the definition of exponential family corresponds to a density operator. Specifically, $\bm{\beta}$ and $\mathcal{C}$ have to be such that $\sum_{i=1}^{k}\beta_{i}L_{i}$ is self-adjoint. Also, observe that a linearly closed set is convex by definition and mixture families are linearly closed. On the other hand, the exponential family is not closed nor linearly closed.

We can now state our version of the quantum Pythagorean theorem (illustrated in Figure 1).

The proof of Theorem 1 is given in Section 5.2 below. It extends the proof of Csiszár and Shields 2004 to the non-commutative setting, carefully accounting for support conditions. By specializing Part $(i)$ to the commutative setting, the classical result with $0<\sigma\in\mathcal{S}(\mathbb{H}_{d})$, as stated in Csiszár and Shields 2004, is recovered. Moreover, Part $(iii)$ considers a more general exponential and mixture families generated by bounded linear operators than we are aware of. This generalization is used in the proof of Part $(iv)$, which involves exponential and mixture families generated by the bounded (non necessarily self-adjoint) class $\mathcal{C}=\{|e_{i}(\rho)\rangle\langle e_{j}(\rho)|\}_{1\leq i\neq j\leq d}$, where $\{e_{i}(\rho)\}_{i=1}^{\infty}$ denotes an orthonormal eigenbasis of $\rho$. It is also interesting that the $I-$projection of $\sigma\in\mathcal{S}(\mathbb{H}_{d})$ to $\bar{\mathcal{S}}(\mathcal{E}_{\rho})$ is simply characterized as the pinched operator $\sum_{i=1}^{d}\mathsf{P}_{i}(\rho)\sigma\mathsf{P}_{i}(\rho)$.

We can now provide an interpretation of Proposition 1 in terms of Theorem 1, when $\mathbb{H}$ is finite dimensional. To this end, consider the interesting case where $\Sigma_{\rho}\coloneqq\{\sigma\in\Sigma:\rho\ll\sigma\}$ is non-empty (otherwise, both sides of (28) are infinite). The idea is to consider any $\sigma\in\Sigma_{\rho}$ and apply the quantum Pythagorean theorem with equality given in (15) with $\bar{\mathcal{S}}=\bar{\mathcal{S}}(\mathcal{E}_{\rho})$, i.e., the set of all density operators with the same eigenbasis as $\rho$, which is a compact convex set. This implies that $\mathsf{D}\left(\rho\middle\|\sigma\right)$ is lower bounded by $\mathsf{D}\big(\rho\|\rho^{\star}_{\sigma,\bar{\mathcal{S}}(\mathcal{E}_{\rho})}\big)=D_{\mathsf{KL}}\big(\lambda_{\rho}\|\lambda_{\rho^{\star}_{\sigma,\bar{\mathcal{S}}(\mathcal{E}_{\rho})}}\big)$ for $\rho^{\star}_{\sigma,\bar{\mathcal{S}}(\mathcal{E}_{\rho})}\in\mathcal{J}\left(\mathcal{E}_{\rho},\{\sigma\}\right)$, where the equality is because $\rho^{\star}_{\sigma,\bar{\mathcal{S}}(\mathcal{E}_{\rho})}$ and $\rho$ commute. Hence $\inf_{\sigma\in\Sigma}\mathsf{D}\left(\rho\middle\|\sigma\right)\geq\inf_{\sigma\in\mathcal{J}\left(\mathcal{E}_{\rho},\Sigma\right)}\mathsf{D}\left(\lambda_{\rho}\middle\|\lambda_{\sigma}\right)$. The opposite inequality is obtained from Theorem 1$(iv)$ and the condition $\mathcal{J}\left(\mathcal{E}_{\rho},\Sigma\right)\subseteq\Sigma$, and both inequalities show (13). Given (13), the last claim in Proposition 1 can be easily argued using the unitary invariance of $\Sigma$. See details in Section 5.3 below.

Having seen some information-geometric aspects of QMLP, we next study its statistical behaviour. The following technical proposition will play a key role towards this purpose.

Proposition 2, proved in Section 5.4, is based on elementary arguments involving quantum Pinsker’s inequality (see, e.g., Hayashi 2016). We will use it to prove Theorem 2 which provides statistical non-asymptotic performance guarantees for QMLP.

To this end, we specialize to embeddings into $\mathbb{H}_{d}$ for some $d\in\mathbb{N}$. Embeddings into finite dimensional Hilbert spaces are practically important, since LLMs and QLLMs are based on such embeddings. For $\rho,\sigma>0$, let the Thompson metric (Thompson 1963) on the space of positive density operators be $T(\rho,\sigma)\coloneqq\log\!\big(\big\|\sigma^{-1/2}\rho\sigma^{-1/2}\big\|\vee\big\|\rho^{-1/2}\sigma\rho^{-1/2}\big\|\big)$. Note that $T(\rho,\sigma)=0$ if and only if $\rho=\sigma$.

For technical reasons, we will consider a slightly perturbed version of $\hat{\rho}_{n}(x^{n})$ obtained using the maximally mixed state $\pi_{d}$ in $\mathbb{H}_{d}$ such that the perturbation vanishes asymptotically with $n$. Accordingly, define

where $\hat{\rho}_{n}(x^{n})$ is as defined in (9), and set

Note that the definition in (19) involves a perturbation of $\hat{\rho}_{n}(x^{n})$ with $\pi_{d}$, which requires knowledge of $\mathbb{H}_{d}$. However, this can be determined given the embedding $\varphi$ and knowledge of the support of $P$, as the span of $\{\varphi(x):p(x)>0,~x\in\mathcal{X}\}$.

The next theorem establishes consistency and convergence properties of QMLP.

The proof of Theorem 2 in Section 5.5 combines the variational form of quantum relative entropy (2) with Proposition 2 to control $\left\|\sigma_{n}^{\star}(X^{n})-\rho_{p}\right\|_{1}$ and $\mathsf{D}\left(\rho_{p}\middle\|\hat{\sigma}_{n}^{\star}(X^{n})\right)$, and then applies matrix Bernstein/Hoeffding bounds (Theorem 3).

A few remarks are in order. In a finite-dimensional Hilbert space $\mathbb{H}_{d}$, a set $\Sigma$ satisfying the conditions in Theorem 2 exists. For instance, $\Sigma=\mathcal{S}(\mathbb{H}_{d})$ is a convex compact set satisfying (24) and (26). More generally, any $\epsilon$-covering of $\mathcal{S}(\mathbb{H}_{d})$ in quantum relative entropy as a measure of separation suffices (see (Tang 2022) for relevant covering number bounds in the classical case). Thus, a smaller $\epsilon$ can be achieved by increasing the expressivity of quantum models. Moreover, the assumption $\rho_{p}>0$ is not very restrictive since $\hat{\rho}_{n}(X^{n})\ll\rho_{p}$ almost surely (the complementary event requires $X_{i}\notin\mathrm{spt}(P)$ for some $1\leq i\leq n$, which has probability zero), and the support of $\Sigma$ can be restricted to support of $\rho_{p}$ which can be determined via knowledge of $\mathrm{spt}(P)$ given an embedding. Next, observe that (22) implies a parametric rate of convergence with respect to $n$ for the QMLP to the target distribution (up to the approximation factor $\epsilon$) when $\epsilon>0$ and $O_{d}(1/n)$ rate of convergence when $\epsilon=0$. Hence, there is no curse of dimensionality in the convergence rate. Also, note that (25) and (27), combined with Pinsker’s inequality, yield the analogues of (22) and (23), respectively, with $\left\|\sigma_{n}^{\star}(X^{n})-\rho_{p}\right\|_{1}$ replaced by $\left\|\hat{\sigma}_{n}^{\star}(X^{n})-\rho_{p}\right\|_{1}$. Finally, by data processing inequality for trace norm and quantum relative entropy, the claims in Theorem 2 also hold for the respective measured quantities; hence also to the output distributions of the LLM. Theorem 2 provides insights on certain properties that a good embedding should satisfy. Specifically, a good embedding $\varphi$ should be such that the minimal eigenvalue of $\rho_{p}$ is as high as possible subject to satisfactory prediction performance. As alluded to in Section 3.1, this occurs when neighboring symbols are embedded into closely aligned elements in the Hilbert space, so that the minimal eigenvalue of $\rho_{p}$ is boosted up. In contrast, classical information-theoretic analyses for prediction obtains performance guarantees by considering only the source probability distribution without taking account of similarity or context, which corresponds to the scenario of orthogonal embedding. This could possibly explain the success of architectures such as the transformer from a high-level perspective, since non-orthogonal embeddings into Hilbert spaces naturally capture the degree of similarity via inner products.