Nothing Special   »   [go: up one dir, main page]
Article Dans Une Revue IEEE Transactions on Pattern Analysis and Machine Intelligence Année : 2021
Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019
Zhengying Liu (1) , Adrien Pavao (1) , Zhen Xu (2) , Sergio Escalera (3, 4) , Fabio Ferreira (5) , Isabelle Guyon (6, 1) , Sirui Hong (7) , Frank Hutter (5) , Rongrong Ji (8) , Julio C S Jacques Junior (4) , Ge Li (9) , Marius Lindauer (5) , Zhipeng Luo (9) , Meysam Madadi (4) , Thomas Nierhoff (5) , Kangning Niu (9) , Chunguang Pan (9) , Danny Stoll (5) , Sebastien Treguer (1) , Jin Wang (9) , Peng Wang (10) , Chenglin Wu (7) , Youcheng Xiong (7) , Arbër Zela (5) , Yang Zhang (7)
1 TAU - TAckling the Underspecified (Université Paris Saclay Bâtiment 660 - Shannon ; 91405 Orsay Cedex - France)
"> TAU - TAckling the Underspecified
2 4Paradigm (Beijing, China - Chine)
"> 4Paradigm
3 UB - Universitat de Barcelona (Gran Via de les Corts Catalanes, 585, 08007 Barcelona - Espagne)
"> UB - Universitat de Barcelona
4 CVC - Computer Vision Center (Centre de visio per computador) (Edifici O - Campus UAB - 08193 Bellaterra (Cerdanyola) - Barcelona, Spain - Espagne)
"> CVC - Computer Vision Center (Centre de visio per computador)
5 Albert-Ludwigs-Universität Freiburg = University of Freiburg (Georges-Köhler Allee 51 79110 Freiburg im Breisgau (Germany) - Allemagne) "> Albert-Ludwigs-Universität Freiburg = University of Freiburg
6 Chalearn (États-Unis) "> Chalearn
7 DeepWisdom [Guangdong] (Unit 13-15 5/F Oriental Technology Building No. 16 Keyuan Road Nanshan District Shenzhen Guangdong China - Chine) "> DeepWisdom [Guangdong]
8 Xiamen University (Chine) "> Xiamen University
9 DeepBlueAI [Shanghai] (No.369,Weining Road 200336 Changning District, Shanghai - Chine) "> DeepBlueAI [Shanghai]
10 Lenovo [Beijing] (6 Shangdi West Road, Haidian District Beijing China - Chine) "> Lenovo [Beijing]
Zhengying Liu
  • Fonction : Auteur
  • PersonId : 1038060
TAckling the Underspecified
Adrien Pavao
  • Fonction : Auteur
TAckling the Underspecified
Zhen Xu
  • Fonction : Auteur
4Paradigm
Sergio Escalera
  • Fonction : Auteur
Universitat de Barcelona
Computer Vision Center (Centre de visio per computador)
Fabio Ferreira
  • Fonction : Auteur
Albert-Ludwigs-Universität Freiburg = University of Freiburg
Isabelle Guyon
  • Fonction : Auteur
Chalearn
TAckling the Underspecified
Sirui Hong
  • Fonction : Auteur
DeepWisdom [Guangdong]
Frank Hutter
  • Fonction : Auteur
Albert-Ludwigs-Universität Freiburg = University of Freiburg
Rongrong Ji
  • Fonction : Auteur
Xiamen University
Julio C S Jacques Junior
  • Fonction : Auteur
Computer Vision Center (Centre de visio per computador)
Ge Li
  • Fonction : Auteur
DeepBlueAI [Shanghai]
Marius Lindauer
  • Fonction : Auteur
Albert-Ludwigs-Universität Freiburg = University of Freiburg
Zhipeng Luo
  • Fonction : Auteur
DeepBlueAI [Shanghai]
Meysam Madadi
  • Fonction : Auteur
Computer Vision Center (Centre de visio per computador)
Thomas Nierhoff
  • Fonction : Auteur
Albert-Ludwigs-Universität Freiburg = University of Freiburg
Kangning Niu
  • Fonction : Auteur
DeepBlueAI [Shanghai]
Chunguang Pan
  • Fonction : Auteur
DeepBlueAI [Shanghai]
Danny Stoll
  • Fonction : Auteur
Albert-Ludwigs-Universität Freiburg = University of Freiburg
Sebastien Treguer
  • Fonction : Auteur
TAckling the Underspecified
Jin Wang
  • Fonction : Auteur
DeepBlueAI [Shanghai]
Peng Wang
Lenovo [Beijing]
Chenglin Wu
  • Fonction : Auteur
DeepWisdom [Guangdong]
Youcheng Xiong
  • Fonction : Auteur
DeepWisdom [Guangdong]
Arbër Zela
  • Fonction : Auteur
Albert-Ludwigs-Universität Freiburg = University of Freiburg
Yang Zhang
DeepWisdom [Guangdong]

Résumé

This paper reports the results and post-challenge analyses of ChaLearn’s AutoDL challenge series, which helped sorting out a profusion of AutoML solutions for Deep Learning (DL) that had been introduced in a variety of settings, but lacked fair comparisons. All input data modalities (time series, images, videos, text, tabular) were formatted as tensors and all tasks were multi-label classification problems. Code submissions were executed on hidden tasks, with limited time and computational resources, pushing solutions that get results quickly. In this setting, DL methods dominated, though popular Neural Architecture Search (NAS) was impractical. Solutions relied on fine-tuned pre-trained networks, with architectures matching data modality. Post-challenge tests did not reveal improvements beyond the imposed time limit. While no component is particularly original or novel, a high level modular organization emerged featuring a “meta-learner”, “data ingestor”, “model selector”, “model/learner”, and “evaluator”. This modularity enabled ablation studies, which revealed the importance of (off-platform) meta-learning, ensembling, and efficient data management. Experiments on heterogeneous module combinations further confirm the (local) optimality of the winning solutions. Ourchallenge legacy includes an ever-lasting benchmark (http://autodl.chalearn.org), the open-sourced code of the winners, and a free “AutoDL self-service”.

Domaines

Intelligence artificielle [cs.AI] Machine Learning [stat.ML]
Fichier principal
Vignette du fichier
TPAMI_AutoDL_paper__draft_version_0022.pdf (1) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Zhengying Liu  : Connectez-vous pour contacter le contributeur

https://hal.science/hal-02957135

Soumis le : samedi 8 janvier 2022-13:21:20

Dernière modification le : vendredi 7 février 2025-18:35:56

Télécharger pour visualiser

Dates et versions

hal-02957135 , version 1 (04-10-2020)
hal-02957135 , version 2 (30-11-2020)
hal-02957135 , version 3 (06-05-2021)
hal-02957135 , version 4 (01-12-2021)
hal-02957135 , version 5 (08-01-2022)
Identifiants

Citer

Zhengying Liu, Adrien Pavao, Zhen Xu, Sergio Escalera, Fabio Ferreira, et al.. Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43 (9), pp.3108 - 3125. ⟨10.1109/TPAMI.2021.3075372⟩. ⟨hal-02957135v5⟩

Exporter

BibTeX XML-TEI Dublin Core DC Terms EndNote DataCite

Collections

CNRS INRIA CENTRALESUPELEC INRIA2 UNIV-PARIS-SACLAY ANR LISN GS-ENGINEERING GS-COMPUTER-SCIENCE LISN-AO ANR-IA
479 Consultations
694 Téléchargements

Altmetric

Partager

More