cispa_all_3938.pdf (444.55 kB)

Preserving privacy with PATE for heterogeneous data

Download (444.55 kB)
Version 2 2023-12-14, 12:27
Version 1 2023-11-29, 18:25
conference contribution
posted on 2023-12-14, 12:27 authored by Akshay Dodwadmath, Sebastian StichSebastian Stich
Differential privacy has become the standard system to provide privacy guarantees for user data in machine learning models. One of the popular techniques to ensure privacy is the Private Aggregation of Teacher Ensembles (PATE) framework. PATE trains an ensemble of teacher models on private data and transfers the knowledge to a student model, with rigorous privacy guarantees derived using differential privacy. So far, PATE has been shown to work assuming the public and private data are distributed homogeneously. We show that in the case of high mismatch (non iid-ness) in these distributions, the teachers suffer from high variance in their individual training updates, causing them to converge to vastly different optimum states. This leads to lower consensus and accuracy for data labelling. To address this, we propose a modification to the teacher training process in PATE, that incorporates teacher averaging and update correction which reduces the variance in teacher updates. Our technique leads to improved prediction accuracy of the teacher aggregation mechanism, especially for highly heterogeneous data. Furthermore, our evaluation shows our technique is necessary to sustain the student model performance, and allows it to achieve considerable gains over the original PATE in the utility-privacy metric.


Preferred Citation

Akshay Dodwadmath and Sebastian Stich. Preserving privacy with PATE for heterogeneous data. In: Neural Information Processing Systems Workshop (NeurIPS-W). 2022.

Primary Research Area

  • Reliable Security Guarantees

Name of Conference

Neural Information Processing Systems Workshop (NeurIPS-W)

Legacy Posted Date


Open Access Type

  • Green


@inproceedings{cispa_all_3938, title = "Preserving privacy with PATE for heterogeneous data", author = "Dodwadmath, Akshay and Stich, Sebastian U.", booktitle="{Neural Information Processing Systems Workshop (NeurIPS-W)}", year="2022", }

Usage metrics


    No categories selected


    Ref. manager