XLDA: Linear Discriminant Analysis for Scaling Continual Learning to Extreme Classification at the Edge
Published in International Conference on Machine Learning (ICML), PAC-Bayes Workshop, 2023
FLOPs requirement for extreme classification grows with the number of classes. We propose a LDA to be viewed as an O(1) classifier training algorithm that not only amortizes memory access and matrix multiply costs for training a given class, but at the same time provides continual learning capability.
In particular, we present:
- XLDA: a framework for on-device Class-incremental learning where an LDA classifier is shown to be equivalent to a fully-connected layer in extreme classification scenarios.
- Optimizations to enable XLDA-based training and inference on-device, under compute and storage constraints. We show upto 42x speed up using a batched training approach and upto 5x inference speedup with nearest neighbor search on extreme datasets like AliProducts (50k classes) and Google Landmarks V2 (81k classes).
Recommended citation: Shah, K., Veerendranath, V., Hebbar, A. and Bhat, R., 2023. XLDA: Linear Discriminant Analysis for Scaling Continual Learning to Extreme Classification at the Edge. arXiv preprint arXiv:2307.11317.
Download Paper