Many optimization algorithm benchmarking platforms allow users to share their experimental data to promote reproducible and reusable research. However, different platforms use different data models and formats, which drastically complicates the identification of relevant datasets, their interpretation, and their interoperability. Therefore, a semantically rich, ontology-based, machine-readable data model that can be used by different platforms is highly desirable. In this paper, we report on the development of such an ontology, which we call OPTION (OPTImization algorithm benchmarking ONtology). Our ontology provides the vocabulary needed for semantic annotation of the core entities involved in the benchmarking process, such as algorithms, problems, and evaluation measures. It also provides means for automatic data integration, improved interoperability, and powerful querying capabilities, thereby increasing the value of the benchmarking data. We demonstrate the utility of OPTION, by annotating and querying a corpus of benchmark performance data from the BBOB collection of the COCO framework and from the Yet Another Black-Box Optimization Benchmark (YABBOB) family of the Nevergrad environment. In addition, we integrate features of the BBOB functional performance landscape into the OPTION knowledge base using publicly available datasets with exploratory landscape analysis. Finally, we integrate the OPTION knowledge base into the IOHprofiler environment and provide users with the ability to perform meta-analysis of performance data.
Authors
Ana Kostovska, Diederick Vermetten, Carola Doerr, Saso Dzeroski, Tome Eftimov
Organizations mentioned
- Bibliographic Reference
- Ana Kostovska, Diederick Vermetten, Carola Doerr, Saso Dzeroski, Tome Eftimov. OPTION: OPTImization Algorithm Benchmarking ONtology. IEEE Transactions on Evolutionary Computation, 2023, 27 (6), pp.1618-1632. ⟨10.1109/TEVC.2022.3232844⟩. ⟨hal-04180573⟩
- DOI
- https://doi.org/10.1109/TEVC.2022.3232844
- HAL Collection
- ['CNRS - Centre national de la recherche scientifique', "Laboratoire d'Informatique de Paris 6", 'Sorbonne Université', 'Sorbonne Université 01/01/2018', 'Faculté des Sciences de Sorbonne Université', 'Sorbonne Université - Texte Intégral', 'Alliance Sorbonne Université']
- HAL Identifier
- 4180573
- Institution
- Jozef Stefan Institute
- Laboratory
- ['Leiden Institute of Advanced Computer Science [Leiden]', 'LIP6']
- Published in
- France
Table of Contents
- Introduction 2
- Background & Related work 3
- Ontologies as representational artefacts 3
- Semantic web technologies 3
- Ontologies for optimization and evolutionary computing 4
- Towards Integrating Performance & Problem Landscape Data 4
- Performance Data 4
- Problem Landscape Data 5
- Domain challenges for data integration 5
- Addressing data integration challenges with ontologies 5
- The OPTION ontology 6
- Ontology design and implementation 6
- Ontology layers 6
- Core entities 7
- Use cases 8
- BBOB 8
- Example annotations of COCO-BBOB performance data 9
- Nevergrad 9
- Landscape data 11
- The OPTION system for annotation, storage and querying 11
- The OPTION KB: annotation and storage 11
- The OPTION KB: querying semantic annotations 11
- Integration of the OPTION Knowledge Base in the IOHprofiler Environment 12
- Extending the OPTION ontology and knowledge base 13
- Conclusions & Future work 13
- References 14
- Biographies 16
- Ana Kostovska 16
- Diederick Vermetten 16
- Carola Doerr 16
- Sašo Džeroski 16
- Panče Panov 16
- Tome Eftimov 16