CL2R: Compatible Lifelong Learning Representations
Abstract
1 Introduction
2 Related Work
3 Main Contributions
4 CL2R Problem Formulation
5 Compatibility Evaluation
5.1 Proposed CL2R Metrics
6 Proposed CL2R Training
7 Experimental Results
7.1 Datasets and Verification Protocol
7.2 Implementation Details
training-set | test-set | ||||
---|---|---|---|---|---|
network | input size | dataset | # classes | dataset | # pairs |
ResNet-32 | \(32 \times 32\) | CIFAR100 | 100 | CIFAR10 | 6k |
ResNet-18 | \(64 \times 64\) | Tiny-ImageNet200 | 200 | ImageNet20 | 6k |
ResNet-18 | \(224 \times 224\) | ImageNet900 | 900 | ImageNet100 | 6k |
ResNet-50 | \(112 \times 112\) | CASIA-WebFace | 10,575 | LFW | 6k |
ResNet-50 | \(112 \times 112\) | CASIA-WebFace | 10,575 | IJB-C | 15M |
7.3 Baselines and Compared Methods
7.4 Evaluation on CIFAR10
method | self- test | cross- test | ECC | BC | FC |
---|---|---|---|---|---|
Initial Task | 0.65 | – | – | – | – |
ER | 0.64 | 0.62 | \(\times\) | –0.034 | –0.210 |
LwF | 0.64 | 0.64 | \(\times\) | –0.009 | –0.002 |
BiC | 0.66 | 0.63 | \(\times\) | –0.015 | –0.028 |
LUCIR | 0.70 | 0.66 | \(\surd\) | –0.012 | –0.038 |
FAN | 0.66 | 0.63 | \(\times\) | –0.023 | –0.035 |
FOSTER | 0.66 | 0.57 | \(\times\) | –0.080 | –0.090 |
\(\ell\) -BCT | 0.65 | 0.60 | \(\times\) | –0.047 | –0.044 |
PODNet | 0.67 | 0.66 | \(\surd\) | –0.014 | –0.013 |
Ours | 0.66 | 0.67 | \(\surd\) | –0.017 | –0.006 |
BCT* | 0.72 | 0.65 | \(\surd\) | –0.003 | –0.071 |
Ours (UB)* | 0.73 | 0.69 | \(\surd\) | –0.039 | –0.040 |
7.5 Evaluation on ImageNet
method | self- test | cross- test | ECC | BC | FC |
---|---|---|---|---|---|
Initial Task | 0.61 | – | – | – | – |
ER | 0.62 | 0.59 | \(\times\) | \(-\) 0.012 | \(-\) 0.028 |
LwF | 0.63 | 0.60 | \(\times\) | \(-\) 0.007 | \(-\) 0.032 |
BiC | 0.60 | 0.61 | \(\times\) | \(-\) 0.001 | \(\hphantom{-}\) 0.005 |
LUCIR | 0.60 | 0.62 | \(\surd\) | \(\hphantom{-}\) 0.012 | \(\hphantom{-}\) 0.015 |
FAN | 0.61 | 0.62 | \(\surd\) | \(\hphantom{-}\) 0.008 | \(\hphantom{-}\) 0.009 |
\(\ell\) -BCT | 0.61 | 0.57 | \(\times\) | \(-\) 0.042 | \(-\) 0.038 |
Ours | 0.61 | 0.63 | \(\surd\) | \(\hphantom{-}\) 0.017 | \(\hphantom{-}\) 0.015 |
BCT* | 0.65 | 0.64 | \(\surd\) | \(\hphantom{-}\) 0.026 | \(-0.05\hphantom{0}\) |
Ours (UB)* | 0.66 | 0.64 | \(\surd\) | \(\hphantom{-}\) 0.031 | \(-\) 0.018 |
method | two tasks | three tasks | ||||
---|---|---|---|---|---|---|
AC | BC | FC | AC | BC | FC | |
\(\ell\) -BCT | 0 | \(-\) 0.127 | \(-\) 0.101 | 0.00 | \(-\) 0.073 | \(\hphantom{-}\) 0.006 |
Ours | 1.00 | \(\hphantom{-}\) 0.005 | \(-\) 0.009 | 0.67 | \(\hphantom{-}\) 0.019 | \(-\) 0.011 |
7.6 Face Verification
Method | two tasks | three tasks | five tasks | ten tasks | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
AC | BC | FC | AC | BC | FC | AC | BC | FC | AC | BC | FC | |
\(\ell\) -BCT | 1.00 | 0.005 | \(-\) 0.010 | 0.67 | \(-\) 0.007 | \(-\) 0.005 | 0.40 | \(-\) 0.002 | \(-\) 0.015 | 0.31 | \(-\) 0.002 | \(-\) 0.010 |
Ours | 1.00 | 0.003 | –0.001 | 1.00 | \(\hphantom{-}\) 0.004 | \(-\) 0.005 | 1.00 | \(\hphantom{-}\) 0.006 | –0.005 | 0.82 | \(\hphantom{-}\) 0.006 | –0.005 |
FAR | Method | two tasks | three tasks | five tasks | ten tasks | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
AC | BC | FC | AC | BC | FC | AC | BC | FC | AC | BC | FC | ||
10 \(^{-1}\) | \(\ell\) -BCT | 1.00 | \(\hphantom{-}\) 0.002 | \(-\) 0.010 | 0.33 | \(-\) 0.007 | \(-\) 0.017 | 0.20 | \(-\) 0.031 | \(-\) 0.028 | 0.22 | \(-\) 0.029 | \(-\) 0.029 |
Ours | 1.00 | \(\hphantom{-}\) 0.005 | –0.009 | 1.00 | \(\hphantom{-}\) 0.004 | –0.006 | 0.80 | \(\hphantom{-}\) 0.001 | –0.008 | 0.76 | \(\hphantom{-}\) 0.011 | –0.002 | |
10 \(^{-2}\) | \(\ell\) -BCT | 0 | \(-\) 0.026 | \(-\) 0.015 | 0.33 | \(-\) 0.011 | \(-\) 0.010 | 0.10 | \(-\) 0.038 | \(-\) 0.025 | 0.09 | \(-\) 0.020 | \(-\) 0.034 |
Ours | 1.00 | \(\hphantom{-}\) 0.005 | –0.017 | 1.00 | \(\hphantom{-}\) 0.010 | \(\hphantom{-}\) 0.009 | 0.80 | \(\hphantom{-}\) 0.008 | –0.014 | 0.73 | \(\hphantom{-}\) 0.010 | –0.003 | |
10 \(^{-4}\) | \(\ell\) -BCT | 0 | \(-\) 0.012 | \(-\) 0.004 | 0.33 | \(-\) 0.010 | \(-\) 0.012 | 0 | \(-\) 0.041 | \(-\) 0.028 | 0.09 | \(-\) 0.016 | \(-\) 0.009 |
Ours | 1.00 | \(\hphantom{-}\) 0.023 | \(\hphantom{-}\) 0.005 | 0.67 | \(\hphantom{-}\) 0.002 | \(\hphantom{-}\) 0.005 | 0.80 | \(\hphantom{-}\) 0.001 | –0.003 | 0.73 | \(\hphantom{-}\) 0.012 | \(\hphantom{-}\) 0.007 |
7.7 Compatibility and Catastrophic Forgetting
8 Ablation Studies
classifier | distillation | ten tasks | ||||
---|---|---|---|---|---|---|
Trainable | Fixed | \(\mathcal {L}_{\scriptscriptstyle \textrm {FD}}\) | \(\mathcal {L}_{\scriptscriptstyle \textrm {FD}}^{\scriptscriptstyle \mathcal {M}}\) | AC | BC | FC |
\(\surd\) | 0.04 | \(-\) 0.130 | \(-\) 0.083 | |||
\(\surd\) | \(\surd\) | 0.13 | –0.050 | –0.049 | ||
\(\surd\) | \(\surd\) | 0.22 | \(-\) 0.043 | \(-\) 0.013 | ||
\(\surd\) | 0.27 | \(-\) 0.078 | \(\hphantom{-}\) 0.003 | |||
\(\surd\) | \(\surd\) | 0.40 | \(-\) 0.019 | \(-\) 0.011 | ||
\(\surd\) | \(\surd\) | 0.44 | –0.003 | \(\hphantom{-}\) 0.005 |
9 Conclusion
Footnotes
References
Index Terms
- CL2R: Compatible Lifelong Learning Representations
Recommendations
Lifelong learning in costly feature spaces
AbstractAn important long-term goal in machine learning systems is to build learning agents that, like humans, can learn many tasks over their lifetime, and moreover use information from these tasks to improve their ability to do so ...
Hybrid learning in lifelong learning implementation
ICHL'11: Proceedings of the 4th international conference on Hybrid learningThis paper, starting out from the concept and practice of lifelong learning, explores hybrid learning model as an effective and efficient way to meet individual's lifelong learning needs. The study focuses on the concept and framework of lifelong hybrid ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
- Editor:
- Abdulmotaleb El Saddik
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Check for updates
Author Tags
Qualifiers
- Research-article
- Refereed
Funding Sources
- European Horizon 2020 Programme
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 654Total Downloads
- Downloads (Last 12 months)388
- Downloads (Last 6 weeks)67
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in