Continual Distillation of Teachers from Different Domains

Abstract

Deep learning models continue to scale, with some requiring more storage than many large-scale datasets. Thus, we introduce a new paradigm: Continual Distillation (CD), where a student learns sequentially from a stream of teacher models without retaining access to earlier teachers. CD faces two challenges: teacher training data is unavailable, and teachers have varying expertise. We show that external unlabeled data enables Unseen Knowledge Transfer (UKT), allowing the student to acquire information from domains not present in the training data, while known to the teacher. We also show that sequential distillation causes Unseen Knowledge Forgetting (UKF) when transferred knowledge is lost after training on later teachers. To better trade off between UKT and UKF, we propose Self External Data Distillation (SE2D), a method that preserves logits on external data to stabilize learning across heterogeneous teachers. Experiments on multiple benchmarks show that SE2D reduces UKF and improves cross-domain generalization. The code and implementation for this work are publicly available at: https://github.com/Nicolas1203/continual_distillation.

Figure 1: Overview of the Continual Distillation problem. A student model $\mathcal{S}$ learns through distillation from a sequence of teachers $\{\mathcal{T}_{1},\mathcal{T}_{2},\mathcal{T}_{3}\}$. During distillation, part of the teachers’ training data is unavailable. The objective is for the student to maintain high performance on all domains known to the teachers, but not necessarily introduced to the student.

1 Introduction

Over the last decade, deep learning models have reached unprecedented scales, creating a growing need for computation-efficient strategies. Consequently, Continual Learning (CL) [8, 24, 35] has become a major branch of contemporary deep learning research. The main idea is straightforward: a model is trained on a sequence of datasets where previous data becomes unavailable over time. The rationale is that re-training on both past and current data as new data arrives can be computationally expensive or require substantial storage. In this context, data access is limited during training.

With the recent adoption of Foundation Models (FMs) [9, 21, 11, 26, 1] as the backbone of modern deep learning, we propose a new paradigm, Continual Distillation (CD), tailored to FMs, illustrated in Figure 1. In CD, instead of learning from a sequence of datasets, we propose to learn from a sequence of models trained on different datasets. Specifically, a single student model learns sequentially from multiple teacher models without retaining access to earlier ones. Similar to training data in CL, new FMs are regularly introduced over time, and storing them is cumbersome since they can rapidly require more storage than large-scale datasets. For instance, storing 10B parameters requires approximately 38GB [7], while FMs often exceed 100B parameters. Even accessing FMs through restricted APIs presents serious limitations, as previous versions may become unavailable after updates. Our CD setup reflects a realistic scenario in which one aims to leverage the ever-evolving stream of FMs to train smaller and more specialized models through distillation. In analogy to CL, where a single model benefits from a continuous stream of data, in CD, a single student model benefits from a continuous stream of teacher models.

However, CD introduces two main challenges. First, the original training data of a foundation model is typically unavailable, undisclosed, or prohibitively large to reuse [2]. Thus, choosing distillation data is critical, and such data can potentially emerge from a domain unknown to the teacher [36]. Second, each teacher generally exhibits distinct abilities and performance; for instance, one may excel at recognizing animals, while another may specialize in distinguishing insects.

In this work, we focus on the case of Continual Distillation, where, analogous to Domain Incremental Learning, teachers have been trained on a domain incremental set of datasets. However, we assume that teachers share partially overlapping domains. We believe this assumption to be realistic, as, for example, it is safe to assume that all FMs have been trained on ImageNet. Therefore, we make the following key assumptions: (1) each teacher is trained on data from different domains while sharing a specific domain; (2) the training data for distillation is fixed; and (3) the training data is unlabeled. The ultimate objective is to obtain a student model that achieves high performance on every domain known by at least one teacher, without having access to all teacher training data or labels. In this context, we find that training data can be decomposed into two categories: External Data (ED), unknown to all teachers, and Internal Data (ID), known to all teachers. Importantly, we discover that ED enables the transfer of knowledge from domains unseen by the student but known by the teacher during distillation, which we refer to as Unseen Knowledge Transfer (UKT). Naturally, we observe that the student inevitably forgets unseen knowledge transferred by previous teachers when learning from future ones. We refer to this phenomenon as Unseen Knowledge Forgetting (UKF). The central problem of CD becomes reaching an optimal UKT-UKF trade-off.

Experimentally, we observe that while the usage of ED enables UKT, mainstream distillation strategies fail to address UKF, as their primary focus relies on maximizing knowledge transfer only. We therefore propose Self External Data Distillation (SE2D), a CL-inspired method tailored for the CD problem, focusing on preserving logits of ED to maintain performance on domains unseen by the student. SE2D allows positive UKT while maintaining performance in older domains. Our contributions are as follows:

• We introduce the paradigm of Continual Distillation, motivated by the practical challenges that arise when previous teacher models are no longer accessible.

• We demonstrate that the choice of distillation data is crucial for transferring knowledge to domains unseen by the student, and choose to take advantage of External Data, unknown to the teachers.

• We identify Unseen Knowledge Forgetting, mitigate it by preserving External Data logits, and validate our approach across various benchmarks.

2 Related Work

Continual Learning. Continual Learning (CL) [5, 24] focuses on enabling models to learn from a sequence of tasks while retaining previously acquired knowledge. Traditionally, each CL task is defined by different non-overlapping datasets with unique properties. In Class Incremental Learning (CIL) [10], each task is composed of a unique set of classes. In Domain Incremental Learning (DIL) [3], classes are shared, but input distributions vary. While access and change in data have been widely studied, we are, to the best of our knowledge, the first to consider an alternative scenario where data is fixed but accessible model change over time. In this case, we focus on a situation where the teachers’ training domain differs, not unlike a DIL setup, where each teacher would be trained on a specific task of a DIL setting. Since foundation models become ubiquitous [26], large, and ever-changing, a paradigm shift from ever-evolving data to ever-evolving teachers appears relevant. While Knowledge Distillation is often employed in CL to mitigate forgetting [22, 28, 32], we instead investigate the phenomenon of forgetting within distillation itself, treating it as the central challenge of our study.

Knowledge Distillation. Knowledge Distillation (KD) [15] is a technique that allows a smaller student model to replicate the behavior of a larger, more capable teacher model. Ideally, KD assumes that both the teacher and student are trained on the same dataset, and has been widely used for model compression [29, 39] and transfer learning [38]. When such data is not available, distillation is considered to be data-free [36]. In that context, the objective is usually to reproduce the teacher’s training domain as closely as possible. Thanks to the capability of knowledge transfer from KD, it has become one of the cornerstones for solving the forgetting problem in CL [28, 12]. Nevertheless, the ideal setting (i.e., the students are distilled with the original teachers’ training dataset) of KD rarely holds in CL due to the unavailability of historical data. Consequently, CL methods conduct distillation with new training samples [20], small memory buffer samples [22], generated samples [37], or adversarial samples [14].

While prior work has explored multi-teacher knowledge distillation and the use of distillation in continual learning, our setting differs in several key aspects. First, existing multi-teacher distillation methods typically assume simultaneous access to all teachers, whereas we consider a sequential setting where only one teacher is accessible at a time. Second, unlike continual learning approaches that focus on data streams, our formulation assumes a fixed dataset and instead models a stream of evolving teacher models. Finally, in contrast to standard distillation-based continual learning methods, we do not assume access to past data or replay buffers, nor to previously seen teachers. This combination of constraints defines a distinct and practically relevant scenario that, to the best of our knowledge, has not been explicitly studied before. We therefore frame this problem as Continual Distillation, emphasizing the shift from data-centric to model-centric CL.

Figure 2: Visualization $\mathcal{D}_{e}$ and $\mathcal{D}_{i}$ and different teacher training domains in our experimental setting. $\mathcal{D}_{i}$ and $\mathcal{D}_{e}$ are publicly available like ImageNet or Wikipedia. However, some private or restricted data might be included in training, such as medical data.

3 Continual Distillation

3.1 Generic Definition

We define Continual Distillation (CD) as the process of distilling the knowledge from a sequence of teacher models continuously into one student model, on a fixed dataset. When distilling from one teacher to the student, other teachers are considered unavailable. The distillation process from a given teacher to the student is analogous to a task in standard CL. Formally, given a sequence of teachers $\{\mathcal{T}_{0},\mathcal{T}_{1},\dots,\mathcal{T}_{N}\}$, each trained on a dataset $\mathcal{D}_{t}^{\mathcal{T}}$, the student $\mathcal{S}$ is optimized to minimize distillation loss $\mathcal{L}_{dist}$, with respect to $\mathcal{T}_{t}$ on a distillation dataset $\mathcal{D}^{\mathcal{S}}$. Importantly, we only consider distillation and no label-dependent loss is considered. In this work, we focus on logits-based distillation as representations are architecture-dependent, computation-intensive, and require access to the entire teacher model. We present the overall procedure in Figure 1. We denote by $\mathbb{D}_{\mathcal{S}}$($\mathcal{T}_{t}$, $\mathcal{D}^{\mathcal{S}})$ the operation of distilling from teacher $\mathcal{T}$ trained with $\mathcal{D}^{\mathcal{T}}_{t}$ to student $\mathcal{S}$ on distillation dataset $\mathcal{D}^{\mathcal{S}}$.

3.2 Specific Problem Scenario

Traditional KD assumes that the teacher training datasets are available for distillation. In CD, not only are such datasets considered unavailable, but dataset domains might differ from one teacher training to another. In other words, $\mathcal{D}_{t}^{\mathcal{T}}$, $\mathcal{D}_{t^{\prime}}^{\mathcal{T}}$ and $\mathcal{D}^{\mathcal{S}}$ may cover partially or totally different domains. Therefore, various scenarios can be defined in CD depending on the domain overlap between teachers training data $\mathcal{D}_{t}^{\mathcal{T}}$ and distillation data $\mathcal{D}^{\mathcal{S}}$. Realistically, when considering FMs, it is safe to assume that part of the training data is shared. Typically, publicly available datasets such as Wikipedia or ImageNet are commonly used for training FMs. However, additional data, exclusive to a specific model, might also be included during training. A visualisation of such a scenario is presented in Figure 2. Formally, we consider the case of partially exclusive teacher domains such that all teachers share a specific domain:

Figure 3: Illustration of Unseen Knowledge Transfer (UKT) and Unseen Knowledge Forgetting (UKF). When distilling only from $\mathcal{T}_{1}$ on Internal Data $\mathcal{D}_{i}$, only Domain $0$ knowledge is transferred. When distilling from the same teacher $\mathcal{T}_{1}$ on $\mathcal{D}_{e}\cup\mathcal{D}_{i}$, Domains $0$ and $1$ knowledge is transferred, although the student has never seen Domain $1$. However, when continuing the sequence and distilling from $\mathcal{T}_{2}$, while the student acquires knowledge from Domain $2$, it forgets part of the knowlege from Domain $1$.

$$\mathcal{D}_{t}^{\mathcal{T}}\cap\mathcal{D}_{t^{\prime}}^{\mathcal{T}}=\mathcal{D}_{i},\ \forall t,t^{\prime},$$ (1)

with $\mathcal{D}_{i}$ the Internal Data (ID). The remaining distillation data is considered unknown to all teachers, hence:

$$\mathcal{D}^{\mathcal{S}}=\mathcal{D}_{e}\cup\mathcal{D}_{i},$$ (2)

where $\mathcal{D}_{e}$ denotes the External Data (ED) such that for any teacher of index $t$, $\mathcal{D}_{e}\cap\mathcal{D}_{t}=\emptyset$.

3.3 Unseen Knowledge Transfer and Forgetting

We define ‘unseen’ as domains not present in the student’s training data but present in the teacher’s knowledge. In CD, $\mathcal{D}_{e}$ is unknown to the teachers. Such an ED can either be introduced or can appear unknowingly when generating data, a standard procedure of data-free distillation. While this could appear to be a limitation, we observe that leveraging ED allows the student to acquire knowledge about domains that have never been explicitly seen during training. We refer to this phenomenon as Unseen Knowledge Transfer (UKT). Intuitively, when integrating ED, generic knowledge is transferred because of the teacher’s uncertainty. Conversely, when the teacher is confident, specific knowledge only is transferred. In CD, we propose to take advantage of UKT by purposefully integrating ED during distillation to extract additional knowledge from the teacher.

However, in CD, the student sequentially learns from multiple teachers, each providing distinct unseen domain knowledge. While UKT enables the student to acquire information about domains not directly represented in the teacher data, this transferred knowledge is often fragile. As the student learns from subsequent teachers, they tend to lose information previously transferred from earlier ones. We refer to this phenomenon as Unseen Knowledge Forgetting (UKF). UKF differs fundamentally from the catastrophic forgetting traditionally studied in Domain Incremental Learning, as the forgotten knowledge does not originate from the student’s own training data but from the teacher’s knowledge. Since the student is never directly exposed to such knowledge, we call it unseen knowledge. An intuitive illustration of UKF and UKT is given in Figure 3.

3.4 Self External Data Distillation (SE2D)

We introduce Self-External Data Distillation (SE2D), a method designed to mitigate UKF in Continual Distillation. In SE2D, the student model is trained not only from the current teacher but also from its own checkpoint saved after the previous task. Such a strategy is quite common in Continual Learning [20, 27]; however, we propose to adapt it to the specific problem at hand. Therefore, the distillation process from the checkpoint is performed exclusively on external data $\mathcal{D}_{e}$, which is unknown to all teachers. An overview of our proposed approach is given in Figure 4.

Figure 4: Overview of Self External Data Distillation (SE2D). While the distillation from the teacher sequence is done on the entire distillation data, distillation from the checkpoint of the student at the end of the previous task is done only on $\mathcal{D}_{e}$.

Prior observations indicate that performance on domains unseen by the student yet known by past teachers depends heavily on these external samples. We restrict self-distillation to external data $\mathcal{D}_{e}$ to specifically preserve knowledge that is not directly supported by the shared internal domain $\mathcal{D}_{i}$. Applying self-distillation on $\mathcal{D}_{i}$ would mainly reinforce already stable knowledge, while our goal is to maintain transferred knowledge from unseen domains, which is primarily captured through $\mathcal{D}_{e}$.

Practically, at each distillation step $t$, the student $\mathcal{S}_{t}$ learns from both the current teacher $\mathcal{T}_{t}$ and the previous student checkpoint $\mathcal{S}_{t-1}$:

$$\mathcal{L}_{\text{SE2D}}=\mathcal{L}_{\text{KD}}(\mathcal{S}_{t},\mathcal{T}_{t};\mathcal{D}^{\mathcal{S}})+\mathcal{L}_{\text{KD}}(\mathcal{S}_{t},\mathcal{S}_{t-1};\mathcal{D}_{e}),$$ (3)

where $\mathcal{L}_{\text{KD}}$ denotes the temperature-scaled Kullback–Leibler divergence between the softmax distributions of the student logits $z_{\mathcal{S}}(x)$ and teacher logits $z_{\mathcal{T}}(x)$:

$$\mathcal{L}_{\text{KD}}(\mathcal{S},\mathcal{T};\mathcal{D}^{\mathcal{S}})=T^{2}\,\mathbb{E}_{x\sim\mathcal{D}^{\mathcal{S}}}\left[\text{KL}\left(\sigma\!\left(\tfrac{z_{\mathcal{T}}(x)}{T}\right)\;\Big\|\;\sigma\!\left(\tfrac{z_{\mathcal{S}}(x)}{T}\right)\right)\right],$$ (4)

where $T$ is the distillation temperature and $\sigma(\cdot)$ denotes the softmax function. This simple yet effective mechanism enables the student to accumulate knowledge over time while retaining transferable information from previous distillation stages. In Section 5, we present experimental results of SE2D across various benchmarks.

4 Experimental Setup

In the following, we describe the experimental setup for Continual Distillation. More details regarding the implementation are given in the appendix.

4.1 Domain Selection

Teachers Domains.

To reproduce a Continual Distillation context, we work on datasets containing data of the same classes but coming from different domains. For example, in Figure 1, each domain contains the class “cat”; however, the color of the cat is different depending on the considered domain. Another example would be the condition in which the picture was taken (inside or outside). Such a domain shift is typically considered in DIL [33, 16].

External Data Selection.

In CD, the choice of ED is crucial. In realistic scenarios, ED could be introduced accidentally, as knowing FMs exact training domain is unlikely. In this work, we deliberately exploit such external data, specifically selected to fall outside all teacher training domains, to study its influence on knowledge transfer and forgetting. We consider two scenarios: (1) related external domains, where the data share the same semantic classes as the teacher domains (e.g., if teachers are trained on sea animals such as dolphins, sharks, and whales, the external data may include jellyfish); and (2) unrelated external domains, where the data are semantically distinct (e.g., using images of trucks or digits instead of marine animals). In both cases, teachers have minimal performance on ED. Additional discussion is included in appendix.

4.2 Teacher Selection

For the experimental setup, we hypothesize that most contemporary FMs share a substantial amount of common knowledge but differ primarily in their performance on specific tasks. Based on this assumption, we train a sequence of teacher models that all share a common domain while each possesses a unique domain not shared by any other teacher. Consequently, all teachers in the sequence provide a shared knowledge base together with a teacher-specific domain that the student must learn and retain through the use of external data. An example is given in Table 1 where each teacher is trained on pairs $(0,1)$, $(0,2)$, and so on.

4.3 Datasets

For simulating CD, we build upon Domain Incremental Learning datasets. We consider each domain separately and pre-train teachers on subsets of such domains. CIFAR20 [30], a variation of CIFAR-100 using the 20 superclasses instead of the 100 fine-grained classes. Since in CIFAR-100, each superclass is composed of 5 sub-classes, using the superclasses allows for defining 5 different domains where each domain is images from different subclasses but identical super-class. CIFAR20 contains $10,000$ train images and $2,000$ test images per domain. All such domains are considered related. Additionally, we experiment with CUB [34] and MNIST as unrelated datasets. Digits, that we define as the combination of various digit datasets, each one representing a different domain. Namely, we mix MNIST [19], MNIST-M [13], USPS [17] and SVHN [23]. As a related domain, we consider KMNIST [6], a dataset composed of Japanese Hiragana. We reckon this dataset as having similar difficulty as MNIST, even though of different classes. DomainNet, an adapted version of the DomainNet dataset [25], containing six visual domains: Real, Clipart, Painting, Infograph, Sketch, and Quickdraw. Each domain shares the same set of 345 classes but differs significantly in style and texture statistics. This dataset contains around 600,000 images, and the domains are unbalanced, increasing the difficulty. All domains are considered related.

4.4 Considered Methods

We selected mainstream and recent state-of-the-art baselines for logits-based distillation methods. We considered the following. KL-divergence. This method consists of a standard distillation loss and serves as a baseline [18]. Logits Standardization (LS) [31] is a method recently proposed that improves upon logits distillation by standardizing student and teacher logits. Medium Difficulty Samples (MDS) [4] is a data-pruning strategy that considers distilling on samples of medium difficulty only. The original method was proposed in a supervised scenario where the teacher’s cross-entropy was considered as the criterion for assessing sample difficulty. In our setup, we adapted this method with the same principle, using teacher entropy as a sample difficulty estimator. Decoupled Knowledge Distillation (DKD) [39], a standard distillation method that decomposes the conventional distillation objective into two components: a target class term and a non-target class term. While most efficient in supervised scenarios, DKD can easily be used in unsupervised scenarios by considering the teacher’s maximum prediction as the target. Self-Distillation. Distillation is regularly used in CL [22, 12, 28]. A common strategy is to save a checkpoint of the current model at the end of the task and use such a checkpoint for distillation when training on the subsequent task. This methods uses both $\mathcal{D}_{i}$ and $\mathcal{D}_{e}$ for distillation.

5 Experimental Results

Table 1: Accuracy (%) of the student model on test sets, domain-wise, on CIFAR20, for various domain overlaps, after distilling for $1$ epoch. The larger the ratio $|\mathcal{D}_{e}|/|\mathcal{D}^{\mathcal{S}}|$, the more the student performs on domains unseen during training (underlined values).

Domain	0	1	2	3	$|\mathcal{D}_{e}|/|\mathcal{D}^{\mathcal{S}}|$
After $\mathbb{D}(\mathcal{T}_{01},\mathcal{D}^{\mathcal{S}}_{0})$	93.25	37.80	45.30	36.90	0%
After $\mathbb{D}(\mathcal{T}_{02},\mathcal{D}^{\mathcal{S}}_{0})$	93.95	31.60	42.20	35.25	0%
After $\mathbb{D}(\mathcal{T}_{03},\mathcal{D}^{\mathcal{S}}_{0})$	92.10	32.00	39.05	33.00	0%
After $\mathbb{D}$($\mathcal{T}_{012}$, $\mathcal{D}^{\mathcal{S}}_{014}$)	92.70	93.15	68.35	53.80	33%
After $\mathbb{D}$($T_{013}$, $\mathcal{D}^{\mathcal{S}}_{014}$)	92.45	92.95	48.95	72.55	33%
After $\mathbb{D}$($\mathcal{T}_{01}$, $\mathcal{D}^{\mathcal{S}}_{04}$)	96.35	77.15	48.95	48.45	50%
After $\mathbb{D}$($\mathcal{T}_{02}$, $\mathcal{D}^{\mathcal{S}}_{04}$)	96.35	43.30	80.10	46.55	50%
After $\mathbb{D}$($\mathcal{T}_{03}$, $\mathcal{D}^{\mathcal{S}}_{04}$)	95.70	49.40	57.60	77.80	50%
After $\mathbb{D}$($\mathcal{T}_{01}$, $\mathcal{D}^{\mathcal{S}}_{034}$)	94.60	85.20	51.85	57.15	66%
After $\mathbb{D}$($\mathcal{T}_{02}$, $\mathcal{D}^{\mathcal{S}}_{034}$)	94.60	44.00	83.55	51.55	66%

5.1 External Data Impact on UKT and UKF

External Data Improves Knowledge Transfer.

A first observation that can be made in the CD setting is the necessity of training with ED to fully distill knowledge from the teacher. To showcase this effect, we train a sequence of teachers on CIFAR20 on domain pairs $\{0,1\},\{0,2\},\{0,3\}$, while distilling to the student model on ID only (domain $0$), using the KL-divergence. The results are displayed in Table 1, where the accuracy of the student after each task is reported. It is important to note that initially, considered teachers achieve above $95\%$ accuracy on their respective domains. Eventually, distilling only on the domain $0$ yields competitive performance on this domain; however, performances on other domains remain extremely limited at any training step. The student performs only on domains that have been encountered during training. In Table 1, we maintain the same teacher sequence but include ED for distillation. Such data are from the domain $4$ of CIFAR20, which is unknown to the teacher. In this case, we can observe that the student maintains performance on domain $0$ while achieving much stronger performances on other domains, despite never being encountered during training. UKT is observed only when distilling with external data.

External Data Accentuates UKF.

While ED typically facilitates knowledge transfer, it can bias the model toward the most recent teacher at the expense of earlier knowledge. Tables 2 and 3 show that ED does not guarantee uniform gains and may even accentuate forgetting. For instance, on Digits, DKD and LS suffer significant drops on SVHN and MNIST-M; notably, DKD’s MNIST-M performance falls from $54.50\%$ to $33.84\%$. However, this effect is not universal, as DKD shows slight gains on DomainNet’s Infograph (Table 4). This suggests that while ED can trigger substantial UKF, the impact depends heavily on the distillation method and domain distribution.

Table 2: Performances (%, higher is better) of the student at the end of training on CIFAR20 for 4 scenarios. Internal Data Only (D0), Related External Data (D4), CUB as ED, and MNIST as ED. The number of runs is set to $3$. The gain columns shows the gain over using Internal Data Only. Grey: Internal Data; Blue: Domain known by the teacher; Red: External Data (ED); White: Ignored.

CIFAR20 - Internal Data Only
Method	InDomColD0	OtherDomD1	OtherDomD2	OtherDomD3	DomainColD4 ✗	AvgColAvg. (0-3)	DomainColGain ($\uparrow$)
DomainCol$\mathcal{T}_{best}$ (upper bound)	InDomCol97.75	OtherDom95.80	OtherDom96.70	OtherDom95.75	DomainCol-	AvgCol96.5	DomainCol0.00
DomainColKL-divergence	InDomCol98.10 $\pm$ 0.05	OtherDom41.40 $\pm$ 0.84	OtherDom53.38 $\pm$ 0.35	OtherDom54.87 $\pm$ 3.70	DomainCol-	AvgCol61.94 $\pm$ 1.24	DomainCol0.00
DomainColDKD [CVPR’22]	InDomCol95.70 $\pm$ 1.55	OtherDom35.53 $\pm$ 1.94	OtherDom45.42 $\pm$ 2.05	OtherDom41.02 $\pm$ 2.55	DomainCol-	AvgCol54.42 $\pm$ 2.02	DomainCol0.00
DomainColLS [CVPR’24]	InDomCol96.50 $\pm$ 1.52	OtherDom39.13 $\pm$ 2.90	OtherDom49.77 $\pm$ 4.40	OtherDom49.58 $\pm$ 5.78	DomainCol-	AvgCol58.75 $\pm$ 3.65	DomainCol0.00
DomainColMDS [ICLR’25]	InDomCol94.20 $\pm$ 0.73	OtherDom34.30 $\pm$ 2.80	OtherDom44.70 $\pm$ 1.40	OtherDom41.00 $\pm$ 1.90	DomainCol-	AvgCol53.55 $\pm$ 1.72	DomainCol0.00
DomainColSelf-Distillation	InDomCol97.45 $\pm$ 0.43	OtherDom37.58 $\pm$ 2.06	OtherDom50.42 $\pm$ 1.41	OtherDom45.83 $\pm$ 2.15	DomainCol-	AvgCol57.82 $\pm$ 1.51	DomainCol0.00

CIFAR20 + Related External Data
Method	InDomColD0	OtherDomD1	OtherDomD2	OtherDomD3	OutDomColD4	AvgColAvg. (0-3)	DomainColGain ($\uparrow$)
DomainCol$\mathcal{T}_{best}$ (upper bound)	InDomCol97.75	OtherDom95.80	OtherDom96.70	OtherDom95.75	OutDomCol-	AvgCol96.5	DomainCol0.00
DomainColKL-divergence	InDomCol97.05 $\pm$ 0.09	OtherDom48.55 $\pm$ 1.15	OtherDom55.08 $\pm$ 0.70	OtherDom84.77 $\pm$ 0.87	OutDomCol-	AvgCol71.36 $\pm$ 0.70	DomainCol9.42
DomainColDKD [CVPR’22]	InDomCol96.05 $\pm$ 0.50	OtherDom44.13 $\pm$ 0.98	OtherDom51.67 $\pm$ 0.92	OtherDom68.55 $\pm$ 2.60	OutDomCol-	AvgCol65.10 $\pm$ 1.25	DomainCol10.68
DomainColLS [CVPR’24]	InDomCol96.85 $\pm$ 0.15	OtherDom47.25 $\pm$ 0.69	OtherDom54.25 $\pm$ 0.46	OtherDom83.20 $\pm$ 1.87	OutDomCol-	AvgCol70.39 $\pm$ 0.79	DomainCol11.64
DomainColMDS [ICLR’25]	InDomCol96.55 $\pm$ 0.07	OtherDom45.26 $\pm$ 2.10	OtherDom54.90 $\pm$ 0.71	OtherDom73.51 $\pm$ 0.56	OutDomCol-	AvgCol67.56 $\pm$ 0.86	DomainCol14.01
DomainColSelf-Distillation	InDomCol97.71 $\pm$ 0.18	OtherDom61.23 $\pm$ 0.83	OtherDom64.21 $\pm$ 0.51	OtherDom76.58 $\pm$ 0.94	OutDomCol-	AvgCol74.93 $\pm$ 0.61	DomainCol17.11
DomainColSE2D (ours)	InDomCol97.46 $\pm$ 0.19	OtherDom70.71 $\pm$ 1.05	OtherDom62.85 $\pm$ 0.50	OtherDom73.65 $\pm$ 1.67	OutDomCol-	AvgCol76.17 $\pm$ 0.85	DomainColn/a

CIFAR20 + CUB
Method	InDomColD0	OtherDomD1	OtherDomD2	OtherDomD3	OutDomColCUB	AvgColAvg. (0-3)	DomainColGain ($\uparrow$)
DomainCol$\mathcal{T}_{best}$ (upper bound)	InDomCol97.75	OtherDom95.80	OtherDom96.70	OtherDom95.75	OutDomCol-	AvgCol96.5	DomainCol0.00
DomainColKL-divergence	InDomCol97.24 $\pm$ 0.37	OtherDom43.89 $\pm$ 1.02	OtherDom55.13 $\pm$ 0.76	OtherDom71.80 $\pm$ 1.73	OutDomCol-	AvgCol67.02 $\pm$ 0.97	DomainCol5.08
DomainColDKD [CVPR’22]	InDomCol93.39 $\pm$ 0.88	OtherDom33.46 $\pm$ 2.51	OtherDom43.10 $\pm$ 2.93	OtherDom40.70 $\pm$ 2.20	OutDomCol-	AvgCol52.66 $\pm$ 2.13	DomainCol-1.76
DomainColLS [CVPR’24]	InDomCol94.79 $\pm$ 3.17	OtherDom38.53 $\pm$ 4.87	OtherDom50.21 $\pm$ 5.74	OtherDom59.32 $\pm$ 11.60	OutDomCol-	AvgCol60.71 $\pm$ 6.34	DomainCol1.96
DomainColMDS [ICLR’25]	InDomCol97.15 $\pm$ 0.50	OtherDom41.12 $\pm$ 0.96	OtherDom52.38 $\pm$ 1.76	OtherDom60.38 $\pm$ 2.48	OutDomCol-	AvgCol62.76 $\pm$ 0.89	DomainCol9.21
DomainColSelf-Distillation	InDomCol97.47 $\pm$ 0.12	OtherDom47.97 $\pm$ 1.07	OtherDom58.40 $\pm$ 1.34	OtherDom61.97 $\pm$ 1.82	OutDomCol-	AvgCol66.45 $\pm$ 1.09	DomainCol8.63
DomainColSE2D (ours)	InDomCol97.74 $\pm$ 0.10	OtherDom53.93 $\pm$ 0.43	OtherDom58.02 $\pm$ 0.45	OtherDom64.54 $\pm$ 1.81	OutDomCol-	AvgCol68.56 $\pm$ 0.70	DomainColn/a
CIFAR20 + MNIST
Method	InDomColD0 (ID)	OtherDomD1	OtherDomD2	OtherDomD3	OutDomColMNIST	AvgColAvg. (0-3)	DomainColGain ($\uparrow$)
DomainCol$\mathcal{T}_{best}$ (upper bound)	InDomCol97.75	OtherDom95.80	OtherDom96.70	OtherDom95.75	OutDomCol-	AvgCol96.5	DomainCol0.00
DomainColKL-divergence	InDomCol94.45 $\pm$ 4.20	OtherDom38.24 $\pm$ 6.06	OtherDom48.44 $\pm$ 8.56	OtherDom57.97 $\pm$ 15.71	OutDomCol-	AvgCol59.78 $\pm$ 8.63	DomainCol-2.16
DomainColDKD [CVPR’22]	InDomCol91.00 $\pm$ 3.35	OtherDom30.71 $\pm$ 3.60	OtherDom41.36 $\pm$ 3.61	OtherDom37.96 $\pm$ 4.56	OutDomCol-	AvgCol50.26 $\pm$ 3.78	DomainCol-4.16
DomainColLS [CVPR’24]	InDomCol92.04 $\pm$ 4.43	OtherDom35.93 $\pm$ 6.37	OtherDom45.79 $\pm$ 7.50	OtherDom52.42 $\pm$ 13.24	OutDomCol-	AvgCol56.54 $\pm$ 7.88	DomainCol-2.21
DomainColMDS [ICLR’25]	InDomCol86.83 $\pm$ 1.04	OtherDom28.53 $\pm$ 2.20	OtherDom37.92 $\pm$ 2.41	OtherDom37.38 $\pm$ 3.24	OutDomCol-	AvgCol47.17 $\pm$ 1.57	DomainCol-6.38
DomainColSelf-Distillation	InDomCol91.90 $\pm$ 4.94	OtherDom36.35 $\pm$ 9.46	OtherDom45.23 $\pm$ 9.18	OtherDom43.58 $\pm$ 12.43	OutDomCol-	AvgCol54.26 $\pm$ 9.00	DomainCol-3.56
DomainColSE2D (ours)	InDomCol92.64 $\pm$ 4.66	OtherDom39.94 $\pm$ 9.73	OtherDom47.55 $\pm$ 9.55	OtherDom48.88 $\pm$ 13.23	OutDomCol-	AvgCol57.25 $\pm$ 9.29	DomainColn/a

Table 3: Performances (%, higher is better) of the student at the end of training on Digits for 2 scenarios. Internal Data Only (MNIST) and Related External Data (KMNIST). The number of runs is set to $3$. Average and standard deviations are reported.

Digits - Internal Data Only
Method	InDomColMNIST	OtherDomSVHN	OtherDomMNIST-M	OtherDomUSPS	DomainColKMNIST ✗	AvgColAvg.	DomainColGain ($\uparrow$)
DomainCol$\mathcal{T}_{best}$ (upper bound)	InDomCol99.2	OtherDom97.84	OtherDom99.25	OtherDom98.8	DomainCol-	AvgCol98.77	DomainCol0.00
DomainColKL-divergence	InDomCol99.17 $\pm$ 0.04	OtherDom35.80 $\pm$ 1.49	OtherDom62.80 $\pm$ 2.69	OtherDom95.81 $\pm$ 0.46	DomainCol-	AvgCol73.40 $\pm$ 1.17	DomainCol0.00
DomainColDKD [CVPR’22]	InDomCol98.70 $\pm$ 0.11	OtherDom33.35 $\pm$ 1.28	OtherDom54.50 $\pm$ 2.45	OtherDom95.07 $\pm$ 0.43	DomainCol-	AvgCol70.40 $\pm$ 1.07	DomainCol0.00
DomainColLS [CVPR’24]	InDomCol99.13 $\pm$ 0.09	OtherDom37.60 $\pm$ 1.40	OtherDom64.10 $\pm$ 1.73	OtherDom96.00 $\pm$ 0.25	DomainCol-	AvgCol74.21 $\pm$ 0.87	DomainCol0.00
DomainColMDS [ICLR’25]	InDomCol98.15 $\pm$ 1.13	OtherDom34.51 $\pm$ 4.80	OtherDom54.97 $\pm$ 9.49	OtherDom93.21 $\pm$ 3.01	DomainCol-	AvgCol70.21 $\pm$ 3.80	DomainCol0.00
DomainColSelf-Distillation	InDomCol99.23 $\pm$ 0.10	OtherDom35.08 $\pm$ 1.03	OtherDom65.87 $\pm$ 1.38	OtherDom95.28 $\pm$ 0.53	DomainCol-	AvgCol73.87 $\pm$ 0.76	DomainCol0.00

Digits + Related External Data
Method	InDomColMNIST	OtherDomSVHN	OtherDomMNIST-M	OtherDomUSPS	OutDomColKMNIST	AvgColAvg.	DomainColGain ($\uparrow$)
DomainColKL-divergence	InDomCol99.13 $\pm$ 0.05	OtherDom31.53 $\pm$ 1.55	OtherDom59.84 $\pm$ 2.57	OtherDom96.51 $\pm$ 0.10	OutDomCol-	AvgCol71.75 $\pm$ 1.07	DomainCol-1.65
DomainColDKD [CVPR’22]	InDomCol98.35 $\pm$ 0.32	OtherDom25.21 $\pm$ 1.62	OtherDom33.84 $\pm$ 4.28	OtherDom92.87 $\pm$ 1.63	OutDomCol-	AvgCol62.57 $\pm$ 1.96	DomainCol-7.83
DomainColLS [CVPR’24]	InDomCol99.13 $\pm$ 0.08	OtherDom32.28 $\pm$ 0.37	OtherDom61.47 $\pm$ 2.15	OtherDom96.33 $\pm$ 0.33	OutDomCol-	AvgCol72.30 $\pm$ 0.73	DomainCol-1.91
DomainColMDS [ICLR’25]	InDomCol99.12 $\pm$ 0.04	OtherDom33.03 $\pm$ 1.79	OtherDom60.74 $\pm$ 0.97	OtherDom96.13 $\pm$ 0.05	OutDomCol-	AvgCol62.50 $\pm$ 0.90	DomainCol-7.71
DomainColSelf-Distillation	InDomCol99.38 $\pm$ 0.01	OtherDom55.86 $\pm$ 1.60	OtherDom90.76 $\pm$ 0.35	OtherDom96.33 $\pm$ 0.15	OutDomCol-	AvgCol85.58 $\pm$ 0.53	DomainCol11.71
DomainColSE2D (ours)	InDomCol99.33 $\pm$ 0.04	OtherDom61.84 $\pm$ 2.05	OtherDom90.44 $\pm$ 0.18	OtherDom96.33 $\pm$ 0.10	OutDomCol-	AvgCol87.00 $\pm$ 0.60	DomainColn/a

Table 4: Performances (%, higher is better) of the student at the end of training on DomainNet for 2 scenarios. Internal Data Only (Clipart) and Related External Data (Sketch). The number of runs is set to $3$. Average and standard deviations are reported.

DomainCol	DomainNet - Internal Data Only
Method	InDomColClipart	OtherDomInfograph	OtherDomPainting	OtherDomQuickdraw	OtherDomReal	DomainColSketch ✗	AvgColAvg.	DomainColGain ($\uparrow$)
DomainCol$\mathcal{T}_{best}$ (upper bound)	InDomCol74.35	OtherDom35.83	OtherDom66.66	OtherDom66.19	OtherDom78.90	DomainCol-	AvgCol64.39	DomainCol0.00
DomainColKL-divergence	InDomCol77.08 $\pm$ 0.45	OtherDom18.16 $\pm$ 0.24	OtherDom44.17 $\pm$ 0.50	OtherDom17.19 $\pm$ 0.73	OtherDom67.29 $\pm$ 0.24	DomainCol-	AvgCol44.78 $\pm$ 0.29	DomainCol0.00
DomainColDKD [CVPR’22]	InDomCol77.32 $\pm$ 0.22	OtherDom18.04 $\pm$ 0.09	OtherDom42.76 $\pm$ 0.40	OtherDom17.43 $\pm$ 0.31	OtherDom63.99 $\pm$ 0.14	DomainCol-	AvgCol43.91 $\pm$ 0.17	DomainCol0.00
DomainColLS [CVPR’24]	InDomCol77.21 $\pm$ 0.37	OtherDom17.21 $\pm$ 0.91	OtherDom40.95 $\pm$ 0.14	OtherDom16.42 $\pm$ 1.01	OtherDom60.39 $\pm$ 0.20	DomainCol-	AvgCol42.43 $\pm$ 0.44	DomainCol0.00
DomainColMDS [ICLR’25]	InDomCol76.92 $\pm$ 0.42	OtherDom18.08 $\pm$ 0.23	OtherDom42.14 $\pm$ 0.38	OtherDom17.44 $\pm$ 1.03	OtherDom63.29 $\pm$ 0.20	DomainCol-	AvgCol43.57 $\pm$ 0.34	DomainCol0.00
DomainColSelf-Distillation	InDomCol80.57 $\pm$ 0.08	OtherDom20.85 $\pm$ 0.33	OtherDom46.23 $\pm$ 0.11	OtherDom23.53 $\pm$ 0.25	OtherDom65.38 $\pm$ 0.04	DomainCol-	AvgCol47.31 $\pm$ 0.11	DomainCol0.00

DomainCol	DomainNet + Related External Data
Method	InDomColClipart	OtherDomInfograph	OtherDomPainting	OtherDomQuickdraw	OtherDomReal	OutDomColSketch	AvgColAvg.	DomainColGain ($\uparrow$)
DomainColKL-divergence	InDomCol76.00 $\pm$ 0.18	OtherDom18.89 $\pm$ 0.09	OtherDom44.77 $\pm$ 0.79	OtherDom15.53 $\pm$ 0.15	OtherDom70.65 $\pm$ 0.46	OutDomCol-	AvgCol45.17 $\pm$ 0.31	DomainCol0.39
DomainColDKD [CVPR’22]	InDomCol76.53 $\pm$ 0.14	OtherDom18.70 $\pm$ 0.44	OtherDom44.24 $\pm$ 0.32	OtherDom16.24 $\pm$ 0.40	OtherDom68.29 $\pm$ 0.34	OutDomCol-	AvgCol44.80 $\pm$ 0.12	DomainCol0.89
DomainColLS [CVPR’24]	InDomCol75.28 $\pm$ 0.39	OtherDom16.88 $\pm$ 0.71	OtherDom40.78 $\pm$ 0.49	OtherDom15.47 $\pm$ 0.43	OtherDom62.76 $\pm$ 0.64	OutDomCol-	AvgCol42.24 $\pm$ 0.31	DomainCol-0.20
DomainColMDS [ICLR’25]	InDomCol75.32 $\pm$ 0.39	OtherDom17.53 $\pm$ 0.07	OtherDom42.28 $\pm$ 0.11	OtherDom16.12 $\pm$ 0.61	OtherDom67.29 $\pm$ 0.11	OutDomCol-	AvgCol43.71 $\pm$ 0.19	DomainCol0.14
DomainColSelf-Distillation	InDomCol80.10 $\pm$ 0.18	OtherDom21.53 $\pm$ 0.09	OtherDom48.15 $\pm$ 0.23	OtherDom25.28 $\pm$ 0.19	OtherDom68.75 $\pm$ 0.01	OutDomCol-	AvgCol48.76 $\pm$ 0.07	DomainCol1.45
DomainColSE2D (ours)	InDomCol78.05 $\pm$ 0.11	OtherDom21.98 $\pm$ 0.29	OtherDom47.76 $\pm$ 0.44	OtherDom23.81 $\pm$ 0.34	OtherDom68.43 $\pm$ 0.14	OutDomCol-	AvgCol48.01 $\pm$ 0.20	DomainColn/a

5.2 UKF Mitigation

Traditional Methods and UKF.

In Tables 2, 3, and 4, we present the results for all methods in CD setups with related ED. Firstly, it can easily be seen that, as expected, performances on earlier domains tend to be the lowest for all methods, which is a direct demonstration of UKF where the student forgot transferred knowledge on unseen domains. In all scenarios, all traditional distillation methods suffer from UKF. Secondly, Self-Distillation can partially mitigate UKF and surpass distillation approaches due to the regularization effect of self-distillation. However, a clear trade-off emerges between UKT and UKF, not unlike the stability-plasticity trade-off in Continual Learning, where a model has to balance learning and remembering capabilities. Therefore, while self-distillation surpasses most baselines on earlier domains, it struggles to achieve high performances on more recent domains.

Performances of SE2D.

SE2D allows for better knowledge retention when compared to most baselines, especially on older tasks where it can surpass baselines by more than $9\%$ on domain $1$ of CIFAR20, as reported in Table 2. However, such results hold only for sufficiently related external data where the student can adequately learn from the teacher from one task to another. In Table 2, when using MNIST in combination, the advantages of SE2D become largely limited. This behavior is even more pronounced when training on DomainNet, where SE2D falls behind Self Distillation. We identify the main reason to be the low teacher quality, giving poor supervision on ED for SE2D. Additionally, results in Table 4 show that even when using domains from DomainNet (supposedly related), performance gain is inconsistent for all methods. We believe the vast discrepancy between domains in this dataset hinders UKT, highlighting the importance of the origin of ED for efficient CD. This is discussed in more detail in appendix.

5.3 Discussion

External Data Ratio Matters.

Another phenomenon that can be observed in Table 1 is that increasing $\frac{|\mathcal{D}_{e}|}{|\mathcal{D}^{\mathcal{S}}|}$, the proportion of ED compared to ID, enhances performances on domains unseen by the student. Therefore, a clear trend emerges: the more data unknown to the teacher is used, the stronger the UKT is observed.

Figure 5: Average Accuracy of the student across all domains known by teachers at the end of training on CIFAR20.

External Data Origin Matters.

Naturally, the origin of the external data has an impact on the intensity of UKT. Intuitively, a large domain difference between external data and teacher domains might make distillation more challenging. To showcase the impact of the domain gap, we experimented with related and unrelated external domains on CIFAR20. Table 2 presents the results with various ED scenarios. Notably, it can be observed that leveraging ED leads to consistent improvement in average on all domains when related enough to ID. For example, when using D4 and CUB as ED, the performances of KL-divergence distillation increase from $61.94\%$ to $71.36\%$ and $67.02\%$, respectively. However, when using MNIST as ED, the performances slightly drop to $59.78\%$. Such a trend is observed for all considered methods, as it can be observed in Figure 5. Semantically, CUB consists of images of birds and is more similar to CIFAR20 than MNIST, even though the number of classes does not align. Interestingly, we observe that leveraging ED is essential to promote UKT. However, as the domain shift between ED and ID increases (e.g., in CUB), performance tends to degrade. When the domain gap becomes too large (as in MNIST), using ED can even result in lower performance than without it.

Limitations of SE2D.

While SE2D reduces UKF, its impact relies on (1) the domain gap between teacher and external data and (2) teacher performance on domains unseen by the student. SE2D also requires data-origin knowledge; the student must distinguish between the teacher’s known and unknown domains. This is particularly complex when data are generated to imitate training sets, making it non-trivial to identify data outside the teacher’s domain.

6 Conclusions and Future Work

In this work, we introduced a new paradigm titled Continual Distillation, where a single model learns on a fixed dataset from a sequence of teachers. Such a new setup is relevant in the context of ever-evolving Foundation Models, which are costly to train, expensive to store, demanding to run inference on, and in many cases only accessible via restricted APIs. In such a context, we observe that the domain of origin of distillation data is crucial for controlling which knowledge is indeed transferred to the student. Notably, we unveil two characteristics: Unknown Knowledge Transfer and Unknown Knowledge Forgetting, which represent the ability of the student to modify its knowledge on domains that they have never encountered. Such knowledge control depends only on the teacher and the data used for distillation. The objective then becomes reaching the best UKT-UKF trade-off. In that sense, we proposed Self External Data Distillation (SE2D), which allows us to reduce UKF and maintain strong average performance on all domains, including domains unseen during training. However, we uncover that the domain gap between external data and the teacher domain must be carefully considered in order to foster UKT. Similarly, performant teachers are required for SE2D performance to be ensured.

Eventually, UKT comprises opportunities and risks, as uncontrolled or undesired knowledge could be involuntarily embedded in a student model through distillation depending on the considered data. Such a vulnerability could be easily exploited and introduce unknown bias to model training. Such an aspect of UKT should be explored in future work. Potential future directions include working with larger models, such as language or multimodal models.

Appendix A Experimental Setup

A.1 Implementation Details

For training, we start from pre-trained weights and use the Adam optimizer with a learning rate of $0.0001$ for 3 epochs. As students start from pre-trained weights, we observe negligible improvement when using more epochs. Images are resized to $224\times 224$ to fit the size used during pre-training. We use random horizontal flips as augmentations and use data normalization. We use a batch size of 64. Regarding the distillation, we use the KL-divergence with a temperature of $10$. Experiments are conducted with ViT architecture, namely the ViT-B/16. We use ViT-base as teachers for CIFAR20 and DomainNet. For Digits, we use a ViT-tiny as a teacher. Every teacher is initialised from pre-trained weights and trained for $50$ epochs with Adam optimizer and a learning rate of $0.0001$. We use the same architectures for students in the main draft and additionally experiment with ViT-tiny. For all results, 3 seeds are used.

For DomainNet, since domains are unbalanced, we oversample or undersample them so that each task has the same number of steps. The base task length is determined by the Internal Data (ID) size, and the External Data (ED) is sampled accordingly. For example, if ID is larger than ED, we oversample from ED.

A.2 Metric

For all experiments, we report the domain-wise Final Accuracy ($\%$) of the student at the end of the training sequence.

Appendix B Additional Discussions

Feasibility of obtaining unknown ED.

In CD, a question naturally arises as to whether obtaining ED unknown to FMs is feasible, since such models are trained on vast and diverse data. We argue that obtaining unknown data is feasible in two highly plausible scenarios: Temporal Gap: Any data uploaded to public repositories after the FM’s training cutoff is guaranteed to be unknown. Private/Synthetic Data: In industrial settings, private proprietary datasets or synthetically generated data serve as excellent ED.

Assessing ED quality.

As presented throughout this paper, ED selection is key in Continual Distillation. While quantifying semantic similarity is hard without teacher data, we propose using the teacher’s own predictive uncertainty, by measuring the entropy, as a proxy. In Figure B.1, we show the entropy distribution of a teacher trained on two domains of CIFAR20, for various domains of CIFAR20 as well as CUB and MNIST. We observe that the more the external data is ”unrelated”, the more ”flat” the entropy distribution becomes. To select adequate ED without knowing the training distribution, a user can 1) filter out samples based on an entropy threshold, 2) use the 4th-order moment (kurtosis) of the entropy distribution to quantify ”flatness”. As shown in Fig. B.1, lower flatness correlates with higher UKT potential. Additionally, in Figure B.2, the entropy distribution varies widely across domains, partially explaining the mitigated results observed on this dataset.

Figure B.1: Entropy distribution of the teacher (ViT-Base) trained on domains 0 and 1 of CIFAR20 for various datasets and domains.

Figure B.2: Entropy distribution of the teacher (ViT-Base) trained on domains Real and Clipart of Domainnet for various DomainNet domains.

Appendix C Additional Experiments

C.1 Additional Metrics

Forgetting

For each method, we measure forgetting as the drop in performance on the learned domains after training from new teachers. Formally, for a student trained to imitate a teacher of index $t$, the forgetting for domain $d$ is computed as:

$$F_{d}=\max_{i<t}A_{d}^{(i)}-A_{d}^{(t)}$$

where $A_{d}^{(i)}$ is the accuracy on domain $d$ after distillation from teacher $t$. The overall forgetting is averaged across all domains:

$$F=\frac{1}{D}\sum_{d=1}^{D}F_{d}$$

This metric captures the extent to which a method forgets previously learned knowledge when adapting to new tasks. The results are presented in Tables C.1, C.2 and C.3. It can be observed that our method leads to competitive forgetting in all scenarios.

Table C.1: Forgetting (%, lower is better) of the student at the end of training on CIFAR20 for 4 scenarios. Internal Data Only (D0), Related External Data (D4), CUB as ED, and MNIST as ED. The number of runs is set to $3$.

CIFAR20 - Internal Data Only
DomainColMethod	InDomColD0	OtherDomD1	OtherDomD2	OtherDomD3	DomainColD4 ✗	AvgColAvg. (0-3)
DomainColKL-divergence	InDomCol0	OtherDom11.95	OtherDom2.98	OtherDom0	DomainCol-	AvgCol3.73
DomainColDKD [CVPR’22]	InDomCol0	OtherDom2.38	OtherDom3.47	OtherDom0	DomainCol-	AvgCol1.46
DomainColLS [CVPR’24]	InDomCol0	OtherDom10.42	OtherDom1.88	OtherDom0	DomainCol-	AvgCol3.08
DomainColMDS [ICLR’25]	InDomCol0	OtherDom11.95	OtherDom2.98	OtherDom0	DomainCol-	AvgCol3.23
DomainColSelf-Distillation	InDomCol0	OtherDom16.02	OtherDom2.47	OtherDom0	DomainCol-	AvgCol4.12

CIFAR20 + Related External Data
DomainColMethod	InDomColD0	OtherDomD1	OtherDomD2	OtherDomD3	OutDomColD4	AvgColAvg. (0-3)
DomainColKL-divergence	InDomCol0.52	OtherDom38.15	OtherDom30.26	OtherDom0	OutDomCol-	AvgCol17.23
DomainColDKD [CVPR’22]	InDomCol0.13	OtherDom25.15	OtherDom16.25	OtherDom0	OutDomCol-	AvgCol10.38
DomainColLS [CVPR’24]	InDomCol0.62	OtherDom39.23	OtherDom32.08	OtherDom0	OutDomCol-	AvgCol17.98
DomainColMDS [ICLR’25]	InDomCol0.29	OtherDom37.74	OtherDom31.00	OtherDom0	OutDomCol-	AvgCol17.26
DomainColSelf-Distillation	InDomCol0	OtherDom25.33	OtherDom7.93	OtherDom0	OutDomCol-	AvgCol8.32
DomainColSE2D (ours)	InDomCol0	OtherDom16.10	OtherDom3.67	OtherDom0	OutDomCol-	AvgCol4.44

CIFAR20 + CUB
DomainColMethod	InDomColD0	OtherDomD1	OtherDomD2	OtherDomD3	OutDomColCUB	AvgColAvg. (0-3)
DomainColKL-divergence	InDomCol0.93	OtherDom30.68	OtherDom17.82	OtherDom0	OutDomCol-	AvgCol12.36
DomainColDKD [CVPR’22]	InDomCol0.89	OtherDom5.37	OtherDom3.86	OtherDom0	OutDomCol-	AvgCol2.03
DomainColLS [CVPR’24]	InDomCol2.54	OtherDom32.28	OtherDom20.66	OtherDom0	OutDomCol-	AvgCol11.87
DomainColMDS [ICLR’25]	InDomCol0.61	OtherDom30.79	OtherDom16.26	OtherDom0	OutDomCol-	AvgCol11.41
DomainColSelf-Distillation	InDomCol0.81	OtherDom27.88	OtherDom2.48	OtherDom0	OutDomCol-	AvgCol7.29
DomainColSE2D (ours)	InDomCol0.40	OtherDom21.61	OtherDom6.81	OtherDom0	OutDomCol-	AvgCol7.21
CIFAR20 + MNIST
DomainColMethod	InDomColD0	OtherDomD1	OtherDomD2	OtherDomD3	OutDomColMNIST	AvgColAvg. (0-3)
DomainColKL-divergence	InDomCol0	OtherDom15.20	OtherDom7.40	OtherDom0	OutDomCol-	AvgCol5.15
DomainColDKD [CVPR’22]	InDomCol0	OtherDom2.74	OtherDom0.57	OtherDom0	OutDomCol-	AvgCol0.83
DomainColLS [CVPR’24]	InDomCol1.57	OtherDom14.58	OtherDom9.50	OtherDom0	OutDomCol-	AvgCol5.41
DomainColMDS [ICLR’25]	InDomCol0.97	OtherDom15.34	OtherDom7.67	OtherDom0	OutDomCol-	AvgCol5.00
DomainColSelf-Distillation	InDomCol0	OtherDom9.69	OtherDom1.02	OtherDom0	OutDomCol-	AvgCol2.18
DomainColSE2D (ours)	InDomCol0.85	OtherDom9.88	OtherDom2.85	OtherDom0	OutDomCol-	AvgCol4.43

Algorithm 1 Continual Distillation with KL divergence and SGD.

0: Sequence of teachers $\{\mathcal{T}_{1},\mathcal{T}_{2},\dots,\mathcal{T}_{T}\}$, student model $\mathcal{S}_{\theta}$, distillation dataset $\mathcal{D}^{\mathcal{S}}$

1: for $t=1$ to $N$ do

2:  for $x\in\mathcal{D}^{\mathcal{S}}$ do

3:   Obtain teacher predictions $p_{t}(x)=\mathcal{T}_{t}(x)$

4:   Student predictions $q_{\theta}(x)=\mathcal{S}_{\theta}(x)$

5:   Distillation loss: $\mathcal{L}_{t}=\mathrm{KL}\big(p_{t}(x)\,\|\,q_{\theta}(x)\big)$

6:   Update student parameters $\theta\leftarrow\theta-\eta\nabla_{\theta}\mathcal{L}_{t}$

7:  end for

8: end for

9: return Trained student model $\mathcal{S}_{\theta}$

Algorithm 2 Overview of SE2D training algorithm with Continual Distillation.

0: Sequence of teachers $\{\mathcal{T}_{1},\mathcal{T}_{2},\dots,\mathcal{T}_{T}\}$, student model $\mathcal{S}_{\theta}$, distillation dataset $\mathcal{D}^{\mathcal{S}}=\mathcal{D}_{i}\cup\mathcal{D}_{e}$

1: for $t=1$ to $N$ do

2:  if $t=1$ then

3:   for $x\in\mathcal{D}^{\mathcal{S}}$ do

4:    Obtain teacher predictions $p_{t}(x)=\mathcal{T}_{t}(x)$

5:    Compute student predictions $q_{\theta}(x)=\mathcal{S}_{\theta}(x)$

6:    Compute distillation loss: $\mathcal{L}_{t}=\mathrm{KL}\big(p_{t}(x)\,\|\,q_{\theta}(x)\big)$

7:    Update student parameters $\theta\leftarrow\theta-\eta\nabla_{\theta}\mathcal{L}_{t}$

8:   end for

9:  else

10:   Load previous student checkpoint $\mathcal{S}_{\theta}^{t-1}$

11:   for $(x_{e},x_{i})\in(\mathcal{D}_{e},\mathcal{D}_{i})$ do

12:    Compute student predictions on internal data: $q_{\theta}(x_{i})=\mathcal{S}_{\theta}(x_{i})$

13:    Compute student predictions on external data: $q_{\theta}(x_{e})=\mathcal{S}_{\theta}(x_{e})$

14:    Obtain previous student predictions on external data: $p_{t-1}(x_{e})=\mathcal{S}_{\theta}^{t-1}(x_{e})$

15:    Obtain current teacher predictions on all data: $p_{t}^{\text{teacher}}((x_{e},x_{i}))=\mathcal{T}_{t}((x_{e},x_{i}))$

16:    Compute distillation loss on external data from previous student: $\mathcal{L}_{\text{student}}=\mathrm{KL}\big(p_{t-1}(x_{e})\,\|\,q_{\theta}(x_{e})\big)$

17:    Compute distillation loss on all data from teacher: $\mathcal{L}_{\text{teacher}}=\mathrm{KL}\big(p_{t}^{\text{teacher}}((x_{e},x_{i}))\,\|\,q_{\theta}((x_{e},x_{i}))\big)$

18:    Compute total loss: $\mathcal{L}_{t}=\mathcal{L}_{\text{student}}+\mathcal{L}_{\text{teacher}}$

19:    Update student parameters $\theta\leftarrow\theta-\eta\nabla_{\theta}\mathcal{L}_{t}$

20:   end for

21:  end if

22: end for

23: return Trained student model $\mathcal{S}_{\theta}$

Table C.2: Forgetting (%, lower is better) of the student at the end of training on Digits for 2 scenarios. Internal Data Only (D0), Related External Data (D4). The number of runs is set to 3.

Digits - Internal Data Only
DomainColMethod	InDomColMNIST	OtherDomSVHN	OtherDomMNIST-M	OtherDomUSPS	DomainColKMNIST ✗	AvgColAvg.
DomainColKL-divergence	InDomCol0.23	OtherDom3.34	OtherDom16.20	OtherDom0	DomainCol-	AvgCol4.94
DomainColDKD [CVPR’22]	InDomCol0.47	OtherDom0.53	OtherDom11.73	OtherDom0	DomainCol-	AvgCol3.12
DomainColLS [CVPR’24]	InDomCol0.10	OtherDom4.09	OtherDom12.88	OtherDom0	DomainCol-	AvgCol4.40
DomainColMDS [ICLR’25]	InDomCol0.23	OtherDom3.34	OtherDom16.20	OtherDom0	DomainCol-	AvgCol4.94
DomainColSelf-Distillation	InDomCol0.13	OtherDom4.06	OtherDom7.46	OtherDom0	DomainCol-	AvgCol3.54

Digits + Related External Data
DomainColMethod	InDomColMNIST	OtherDomSVHN	OtherDomMNIST-M	OtherDomUSPS	OutDomColKMNIST	AvgColAvg.
DomainColKL-divergence	InDomCol0.24	OtherDom40.68	OtherDom38.41	OtherDom0	OutDomCol-	AvgCol19.17
DomainColDKD [CVPR’22]	InDomCol0.84	OtherDom12.00	OtherDom50.77	OtherDom0	OutDomCol-	AvgCol15.87
DomainColLS [CVPR’24]	InDomCol0.27	OtherDom40.50	OtherDom36.59	OtherDom0	OutDomCol-	AvgCol19.22
DomainColMDS [ICLR’25]	InDomCol0.25	OtherDom40.69	OtherDom38.35	OtherDom0	OutDomCol-	AvgCol19.16
DomainColSelf-Distillation	InDomCol0.10	OtherDom16.33	OtherDom6.35	OtherDom0	OutDomCol-	AvgCol5.58
DomainColSE2D (ours)	InDomCol0.13	OtherDom10.37	OtherDom4.90	OtherDom0	OutDomCol-	AvgCol3.73

Table C.3: Forgetting (%, lower is better) on DomainNet for 2 scenarios: Internal Data Only (Seq -1) and Related External Data (Seq 0). Averaged across 3 runs.

DomainCol	DomainNet - Internal Data Only
DomainColMethod	InDomColClipart	OtherDomInfograph	OtherDomPainting	OtherDomQuickdraw	OtherDomReal	DomainColSketch ✗	AvgColAvg.
DomainColKL-divergence	InDomCol4.63	OtherDom9.91	OtherDom13.70	OtherDom20.31	OtherDom0	DomainCol-	AvgCol12.14
DomainColDKD [CVPR’22]	InDomCol3.85	OtherDom10.34	OtherDom13.89	OtherDom22.38	OtherDom0	DomainCol-	AvgCol12.61
DomainColLS [CVPR’24]	InDomCol3.50	OtherDom11.48	OtherDom15.39	OtherDom23.73	OtherDom0	DomainCol-	AvgCol13.52
DomainColMDS [ICLR’25]	InDomCol5.20	OtherDom10.70	OtherDom15.69	OtherDom21.46	OtherDom0	DomainCol-	AvgCol13.26
DomainColCheckpoint	InDomCol1.27	OtherDom5.86	OtherDom11.30	OtherDom20.68	OtherDom0	DomainCol-	AvgCol9.78
DomainColSE2D (ours)	InDomCol4.60	OtherDom9.58	OtherDom13.49	OtherDom20.27	OtherDom0	DomainCol-	AvgCol11.98

DomainCol	DomainNet + Related External Data
DomainColMethod	InDomColClipart	OtherDomInfograph	OtherDomPainting	OtherDomQuickdraw	OtherDomReal	OutDomColSketch ✗	AvgColAvg.
DomainColKL-divergence	InDomCol5.56	OtherDom11.89	OtherDom18.07	OtherDom29.12	OtherDom0	OutDomCol-	AvgCol16.16
DomainColDKD [CVPR’22]	InDomCol4.64	OtherDom12.15	OtherDom18.31	OtherDom29.69	OtherDom0	OutDomCol-	AvgCol16.20
DomainColLS [CVPR’24]	InDomCol5.40	OtherDom13.70	OtherDom20.98	OtherDom32.44	OtherDom0	OutDomCol-	AvgCol18.13
DomainColMDS [ICLR’25]	InDomCol6.25	OtherDom13.03	OtherDom19.68	OtherDom29.62	OtherDom0	OutDomCol-	AvgCol17.15
DomainColCheckpoint	InDomCol1.70	OtherDom6.63	OtherDom14.54	OtherDom29.43	OtherDom0	OutDomCol-	AvgCol13.08
DomainColSE2D (ours)	InDomCol3.42	OtherDom6.91	OtherDom15.54	OtherDom30.86	OtherDom0	OutDomCol-	AvgCol14.18

Accuracy Curves

We report per-domain accuracy curves during training with related external data in Figures D.1 to Figure D.4.

C.2 Additional Architecures

We report results with additional architectures. Namely, we experimented with a ViT-tiny as a student instead of the ViT-base version in the main manuscript. Similarly, we experimented with larger models as teacher using CLIP-base teachers (ViT-L/14). Results are presented in Table C.5.

Table C.4: Performances (%, higher is better) of the student at the end of training on CIFAR20 for 2 scenarios with a ViT-tiny. Internal Data Only (D0), Related External Data (D4). The number of runs is set to 3. Average and standard deviations are reported.

CIFAR20 - Internal Data Only
DomainColMethod	InDomColD0	OtherDomD1	OtherDomD2	OtherDomD3	DomainColD4 ✗	AvgColAvg. (0-3)
DomainCol$\mathcal{T}_{best}$ (upper bound)	InDomCol97.75	OtherDom95.80	OtherDom96.70	OtherDom95.75	DomainCol-	AvgCol96.5
DomainColKL-divergence	InDomCol96.27 $\pm$ 0.21	OtherDom37.85 $\pm$ 0.78	OtherDom52.28 $\pm$ 0.42	OtherDom49.48 $\pm$ 1.28	DomainCol-	AvgCol58.97 $\pm$ 0.67
DomainColDKD [CVPR’22]	InDomCol95.40 $\pm$ 1.05	OtherDom33.27 $\pm$ 1.60	OtherDom45.38 $\pm$ 0.90	OtherDom40.45 $\pm$ 1.34	DomainCol-	AvgCol53.62 $\pm$ 1.22
DomainColLS [CVPR’24]	InDomCol96.25 $\pm$ 0.85	OtherDom38.95 $\pm$ 2.17	OtherDom51.58 $\pm$ 2.47	OtherDom50.15 $\pm$ 4.40	DomainCol-	AvgCol59.23 $\pm$ 2.47
DomainColMDS [ICLR’25]	InDomCol94.45 $\pm$ 0.44	OtherDom35.08 $\pm$ 0.88	OtherDom45.35 $\pm$ 0.41	OtherDom41.98 $\pm$ 3.10	DomainCol-	AvgCol54.22 $\pm$ 1.21
DomainColSelf-Distillation	InDomCol96.27 $\pm$ 0.35	OtherDom37.93 $\pm$ 1.22	OtherDom51.82 $\pm$ 0.53	OtherDom46.43 $\pm$ 0.97	DomainCol-	AvgCol58.11 $\pm$ 0.77

CIFAR20 + Related External Data
DomainColMethod	InDomColD0	OtherDomD1	OtherDomD2	OtherDomD3	OutDomColD4	AvgColAvg. (0-3)
DomainCol$\mathcal{T}_{best}$ (upper bound)	InDomCol97.75	OtherDom95.80	OtherDom96.70	OtherDom95.75	OutDomCol-	AvgCol96.5
DomainColKL-divergence	InDomCol96.36 $\pm$ 0.33	OtherDom46.54 $\pm$ 1.17	OtherDom55.86 $\pm$ 1.10	OtherDom75.96 $\pm$ 1.76	OutDomCol-	AvgCol68.68 $\pm$ 1.09
DomainColDKD [CVPR’22]	InDomCol95.18 $\pm$ 0.46	OtherDom40.77 $\pm$ 0.95	OtherDom50.92 $\pm$ 1.07	OtherDom60.69 $\pm$ 1.32	OutDomCol-	AvgCol61.89 $\pm$ 0.95
DomainColLS [CVPR’24]	InDomCol96.45 $\pm$ 0.26	OtherDom46.80 $\pm$ 0.74	OtherDom55.08 $\pm$ 0.61	OtherDom76.49 $\pm$ 1.50	OutDomCol-	AvgCol68.7 $\pm$ 0.78
DomainColSelf-Distillation	InDomCol97.26 $\pm$ 0.25	OtherDom55.42 $\pm$ 0.31	OtherDom61.62 $\pm$ 0.70	OtherDom68.74 $\pm$ 1.46	OutDomCol-	AvgCol70.76 $\pm$ 0.68
DomainColSE2D (ours)	InDomCol96.77 $\pm$ 0.25	OtherDom62.33 $\pm$ 1.19	OtherDom59.45 $\pm$ 1.36	OtherDom65.82 $\pm$ 2.03	OutDomCol-	AvgCol71.09 $\pm$ 1.21

Table C.5: Domain Accuracy (%, higher is better) on DomainNet with CLIP-based teachers. Mean and standard deviation are reported.

DomainCol	DomainNet - Internal Data Only
DomainColMethod	InDomColClipart	OtherDomInfograph	OtherDomPainting	OtherDomQuickdraw	OtherDomReal	DomainColSketch ✗	AvgColAvg.
DomainCol$\mathcal{T}_{best}$ (upper bound)	InDomCol86.33	OtherDom53.89	OtherDom79.74	OtherDom69.62	OtherDom89.86	DomainCol-	AvgCol76.23
DomainColKL-divergence	InDomCol78.33 $\pm$ 0.23	OtherDom19.00 $\pm$ 0.30	OtherDom39.25 $\pm$ 0.81	OtherDom35.36 $\pm$ 0.49	OtherDom51.51 $\pm$ 0.74	DomainCol-	AvgCol44.69 $\pm$ 0.32
DomainColDKD [CVPR’22]	InDomCol78.55 $\pm$ 0.03	OtherDom18.89 $\pm$ 0.41	OtherDom39.69 $\pm$ 0.19	OtherDom34.77 $\pm$ 0.38	OtherDom52.45 $\pm$ 0.49	DomainCol-	AvgCol44.87 $\pm$ 0.06
DomainColLS [CVPR’24]	InDomCol74.39 $\pm$ 0.39	OtherDom15.54 $\pm$ 0.66	OtherDom33.92 $\pm$ 0.56	OtherDom20.09 $\pm$ 1.11	OtherDom46.62 $\pm$ 0.76	DomainCol-	AvgCol38.11 $\pm$ 0.64
DomainColMDS [ICLR’25]	InDomCol75.51 $\pm$ 0.10	OtherDom17.52 $\pm$ 0.20	OtherDom35.67 $\pm$ 1.39	OtherDom26.22 $\pm$ 0.70	OtherDom47.71 $\pm$ 0.69	DomainCol-	AvgCol40.53 $\pm$ 0.27
DomainColSelf-Distillation	InDomCol79.99 $\pm$ 0.55	OtherDom21.72 $\pm$ 0.80	OtherDom43.65 $\pm$ 0.57	OtherDom26.52 $\pm$ 1.42	OtherDom57.21 $\pm$ 0.13	DomainCol-	AvgCol45.82 $\pm$ 0.61

DomainCol	DomainNet + Related External Data
DomainColMethod	InDomColClipart	OtherDomInfograph	OtherDomPainting	OtherDomQuickdraw	OtherDomReal	OutDomColSketch ✗	AvgColAvg.
DomainCol$\mathcal{T}_{best}$ (upper bound)	InDomCol86.33	OtherDom53.89	OtherDom79.74	OtherDom69.62	OtherDom89.86	OutDomCol-	AvgCol76.23
DomainColKL-divergence	InDomCol78.22 $\pm$ 0.39	OtherDom19.98 $\pm$ 0.17	OtherDom41.59 $\pm$ 0.55	OtherDom42.83 $\pm$ 0.37	OtherDom52.62 $\pm$ 0.40	OutDomCol-	AvgCol47.05 $\pm$ 0.22
DomainColDKD [CVPR’22]	InDomCol78.17 $\pm$ 0.11	OtherDom20.01 $\pm$ 0.19	OtherDom41.78 $\pm$ 0.39	OtherDom42.69 $\pm$ 0.84	OtherDom52.37 $\pm$ 0.24	OutDomCol-	AvgCol47.00 $\pm$ 0.01
DomainColLS [CVPR’24]	InDomCol74.64 $\pm$ 0.13	OtherDom16.63 $\pm$ 0.31	OtherDom37.09 $\pm$ 0.09	OtherDom26.78 $\pm$ 0.35	OtherDom47.45 $\pm$ 0.35	OutDomCol-	AvgCol40.52 $\pm$ 0.17
DomainColMDS [ICLR’25]	InDomCol75.21 $\pm$ 0.59	OtherDom18.55 $\pm$ 0.45	OtherDom39.32 $\pm$ 0.13	OtherDom31.84 $\pm$ 0.73	OtherDom49.32 $\pm$ 0.39	OutDomCol-	AvgCol42.85 $\pm$ 0.39
DomainColSelf-Distillation	InDomCol79.47 $\pm$ 0.50	OtherDom22.84 $\pm$ 0.60	OtherDom47.51 $\pm$ 1.33	OtherDom30.15 $\pm$ 0.77	OtherDom58.51 $\pm$ 1.37	OutDomCol-	AvgCol47.69 $\pm$ 0.84
DomainColSE2D (ours)	InDomCol78.52 $\pm$ 0.10	OtherDom21.34 $\pm$ 0.45	OtherDom47.12 $\pm$ 0.35	OtherDom29.18 $\pm$ 0.30	OtherDom58.01 $\pm$ 0.61	OutDomCol-	AvgCol46.83 $\pm$ 0.31

C.3 Additional Sequences

As presented in the main paper, DomainNet is particularly challenging. Therefore, we conducted additional experiments where challenging domains have either been removed or used as external data. Such results are presented in Table C.6. Despite SE2D’s lower overall performance compared to Self-Distillation, the disparity between the two methods diminishes in less complex scenarios, where SE2D simultaneously expands its lead over the baseline.

Table C.6: Accuracy per domain (%). Grey: Internal Data; Blue: Domain known by the teacher (active); Red: External Data (ED); White: Ignored. Avg computed on Internal Data + Active domains.

	DomainNet - Sequence 1 (Quickdraw is used as ED and Infograph is ignored)
Method	ClipartGreyClipart	IgnoredWhiteInfograph	UsedBluePainting	EDRedQuickdraw	UsedBlueReal	UsedBlueSketch	AvgColAvg.
KL-divergence	ClipartGrey74.66 $\pm$ 0.14	IgnoredWhite-	UsedBlue33.01 $\pm$ 0.21	EDRed-	UsedBlue46.17 $\pm$ 0.62	UsedBlue48.68 $\pm$ 0.15	AvgCol50.63
DKD [CVPR’22]	ClipartGrey76.18 $\pm$ 0.20	IgnoredWhite-	UsedBlue36.20 $\pm$ 0.49	EDRed-	UsedBlue49.70 $\pm$ 0.62	UsedBlue54.85 $\pm$ 0.32	AvgCol54.23
MDS [ICLR’25]	ClipartGrey70.76 $\pm$ 0.40	IgnoredWhite-	UsedBlue30.01 $\pm$ 0.78	EDRed-	UsedBlue43.52 $\pm$ 1.09	UsedBlue51.56 $\pm$ 0.68	AvgCol48.96
Self-Distillation	ClipartGrey80.43 $\pm$ 0.10	IgnoredWhite-	UsedBlue47.47 $\pm$ 0.71	EDRed-	UsedBlue61.10 $\pm$ 0.59	UsedBlue58.15 $\pm$ 0.31	AvgCol61.79
SE2D (ours)	ClipartGrey76.89 $\pm$ 0.25	IgnoredWhite-	UsedBlue38.72 $\pm$ 0.49	EDRed-	UsedBlue52.83 $\pm$ 0.20	UsedBlue58.84 $\pm$ 0.25	AvgCol56.82

	DomainNet - Sequence 2 (Infograph is used as ED)
Method	ClipartGreyClipart	EDRedInfograph	UsedBluePainting	UsedBlueQuickdraw	UsedBlueReal	UsedBlueSketch	AvgColAvg.
KL-divergence	ClipartGrey75.78 $\pm$ 0.14	EDRed-	UsedBlue34.61 $\pm$ 0.52	UsedBlue16.59 $\pm$ 0.22	UsedBlue49.27 $\pm$ 0.43	UsedBlue52.43 $\pm$ 0.44	AvgCol45.73
DKD [CVPR’22]	ClipartGrey76.73 $\pm$ 0.33	EDRed-	UsedBlue36.48 $\pm$ 0.68	UsedBlue17.53 $\pm$ 0.44	UsedBlue51.84 $\pm$ 0.55	UsedBlue56.83 $\pm$ 0.34	AvgCol47.88
MDS [ICLR’25]	ClipartGrey75.28 $\pm$ 0.17	EDRed-	UsedBlue35.43 $\pm$ 0.92	UsedBlue16.47 $\pm$ 0.48	UsedBlue50.74 $\pm$ 0.88	UsedBlue56.65 $\pm$ 0.37	AvgCol46.91
Self-Distillation	ClipartGrey80.45 $\pm$ 0.06	EDRed-	UsedBlue48.48 $\pm$ 0.41	UsedBlue20.84 $\pm$ 0.89	UsedBlue64.79 $\pm$ 0.40	UsedBlue59.04 $\pm$ 0.24	AvgCol54.72
SE2D (ours)	ClipartGrey78.08 $\pm$ 0.20	EDRed-	UsedBlue49.02 $\pm$ 0.36	UsedBlue18.50 $\pm$ 0.40	UsedBlue63.29 $\pm$ 0.32	UsedBlue58.99 $\pm$ 0.22	AvgCol53.58

	DomainNet - Sequence 3 (Sketch is used as ED, Infograph and Quickdraw are ignored)
Method	ClipartGreyClipart	IgnoredWhiteInfograph	UsedBluePainting	IgnoredWhiteQuickdraw	UsedBlueReal	EDRedSketch	AvgColAvg.
KL-divergence	ClipartGrey75.75 $\pm$ 0.35	IgnoredWhite-	UsedBlue42.44 $\pm$ 0.67	IgnoredWhite-	UsedBlue65.15 $\pm$ 0.53	EDRed-	AvgCol61.11
DKD [CVPR’22]	ClipartGrey76.25 $\pm$ 0.23	IgnoredWhite-	UsedBlue45.52 $\pm$ 0.72	IgnoredWhite-	UsedBlue70.50 $\pm$ 0.12	EDRed-	AvgCol64.09
MDS [ICLR’25]	ClipartGrey75.25 $\pm$ 0.07	IgnoredWhite-	UsedBlue44.74 $\pm$ 0.94	IgnoredWhite-	UsedBlue70.11 $\pm$ 0.35	EDRed-	AvgCol63.37
Self-Distillation	ClipartGrey79.54 $\pm$ 0.26	IgnoredWhite-	UsedBlue59.61 $\pm$ 0.30	IgnoredWhite-	UsedBlue71.48 $\pm$ 0.26	EDRed-	AvgCol70.21
SE2D (ours)	ClipartGrey78.04 $\pm$ 0.15	IgnoredWhite-	UsedBlue59.36 $\pm$ 0.68	IgnoredWhite-	UsedBlue70.61 $\pm$ 0.40	EDRed-	AvgCol69.34

Appendix D Algorithms

To provide a clear overview of our training methodology, we present the Continual Distillation procedure in Algorithm 1. Furthermore, a detailed description of our proposed SE2D approach is provided in Algorithm 2.

Figure D.1: Accuracy of the student on all domains at all steps for the considered methods on CIFAR20, training with Internal Data only.

Figure D.2: Accuracy of the student on all domains at all steps for the considered methods on CIFAR20, training with External Data.

Figure D.3: Accuracy of the student on all domains at all steps for the considered methods on Digits, training with Internal Data only.

Figure D.4: Accuracy of the student on all domains at all steps for the considered methods on Digits, training with External Data.