Oblivionis

A Lightweight Learning and Unlearning Framework for Federated Large Language Models

Fuyao Zhang*1, Xinyu Yan*1, Tiantong Wu1, Wenjie Li1, 2, Tianxiang Chen3, Yang Cao4, Ran Yan1, Longtao Huang5, Wei Yang Bryan Lim✦1, Qiang Yang6
1Nanyang Technological University    2Hebei Normal University    3Beihang University    4Institute of Science Tokyo    5Alibaba Group    6Hong Kong Polytechnic University
Paper Code

🚀 What is Oblivionis?

We introduce Oblivionis, a lightweight framework that integrates federated learning and targeted unlearning for LLMs, formulating them as a joint multi-objective optimization task to enable privacy-preserving training and compliance with GDPR's right to be forgotten.

🎯 Three Key Objectives

🎖️ Effectiveness
Selectively removing all influences of a client's local private data from the global model
✨ Robustness
Ensuring the model maintains high utility on retained data
💡 Lightweight Design
Enabling unlearning with minimal computational resources and model parameters

🎯 We support

6

Federated Learning Methods

FedAv, FedProx, FedAdagrad, FedAdam, FedYogi

5

Unlearning Methods

GradAscent, GradDiff, NPO, SimNPO, Retrain

10+

Metrics

Including Probability, ROUGE-L, Truth Ratio, and other 10+ metrics

6

Datasets

Including TOFU: Forget, Retain, Real Author, Word Fact; MUSE: Books, News

🔧 Framework Overview

Framework Architecture
Framework Architecture

✅ Federated Fine-tuning

✅ Federated Targeted Unlearning

📊 Main Results

Performance Comparison Chart
Evaluated on metrics MU (Model Utility) and FTR (Forget Truth Ratio)

🔬 Further Analysis

Further Analysis Results
Further Analysis Results
Further Analysis Results
For more experimental details, please refer to our technical report

✨ Key Features

Dual-objective Optimization

Built a complete optimization process

Comprehensive Benchmark Integration

Facilitating standardized and reproducible research

Superior Performance Trade-off

Robust balance between forgetting and model utility

📄 Citation

@misc{zhang2025oblivionislightweightlearningunlearning,
      title={Oblivionis: A Lightweight Learning and Unlearning Framework for Federated Large Language Models}, 
      author={Fuyao Zhang and Xinyu Yan and Tiantong Wu and Wenjie Li and Tianxiang Chen and Yang Cao and Ran Yan and Longtao Huang and Wei Yang Bryan Lim and Qiang Yang},
      year={2025},
      eprint={2508.08875},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2508.08875}, 
}
Get Started

Disclaimer

We are aware of an existing work whose title bears some similarity to ours. The term Oblivionis in our title originates from the Latin root oblivio, meaning “forgetting” or “oblivion.” We would like to clarify that our study is independent and distinct in scope, methodology, and contributions.