A major strength of computational cognitive models is their capacity to accurately predict empirical data. However, challenges in understanding how complex models work and the risk of overfitting have often been addressed by trading off predictive accuracy with model simplification. Here, we introduce state-of-the-art model analysis techniques to show how a large number of parameters in a cognitive model can be reduced into a smaller set that is simpler to understand and can be used to make more constrained predictions with. As a test case, we created different versions of the Connectionist Dual-Process model (CDP) of reading aloud whose parameters were optimized on seven different databases. The results showed that CDP was not overfit and could predict a large amount of variance across those databases. Indeed, the quantitative performance of CDP was higher than that of previous models in this area. Moreover, sloppy parameter analysis, a mathematical technique used to quantify the effects of different parameters on model performance, revealed that many of the parameters in CDP have very little effect on its performance. This shows that the dynamics of CDP are much simpler than its relatively large number of parameters might suggest. Overall, our study shows that cognitive models with large numbers of parameters do not necessarily overfit the empirical data and that understanding the behavior of complex models is more tractable using appropriate mathematical tools. The same techniques could be applied to many different complex cognitive models whenever appropriate datasets for model optimization exist.
Authors
- Bibliographic Reference
- Conrad Perry, Rick Evertz, Marco Zorzi, Johannes C. Ziegler. Understanding the complexity of computational models through optimization and sloppy parameter analyses: The case of the Connectionist Dual-Process Model. Journal of Memory and Language, 2024, 134, pp.104468. ⟨10.1016/j.jml.2023.104468⟩. ⟨hal-04226995⟩
- DOI
- https://doi.org/10.1016/j.jml.2023.104468
- HAL Collection
- ['CNRS - Centre national de la recherche scientifique', 'Aix Marseille Université', 'Réseau de recherche en Théorie des Systèmes Distribués, Modélisation, Analyse et Contrôle des Systèmes', 'Laboratoire de psychologie cognitive', 'Institut Langage, Communication et Cerveau', 'ANR', 'NeuroMarseille', 'Institut Créativité et Innovations Aix-Marseille']
- HAL Identifier
- 4226995
- Institution
- Aix Marseille Université
- Laboratory
- Laboratoire de psychologie cognitive
- Published in
- France
Table of Contents
- Understanding the complexity of computational models through optimization and sloppy parameter analyses: The case of the Co ... 2
- Modeling reading aloud and the CDP model 3
- Simple is better 4
- Sloppy parameters 5
- SPA steps 6
- Optimization 6
- Optimization specifics 7
- Results and discussion 8
- Optimization performance 8
- Distribution of parameters 9
- Sloppy parameter analysis 12
- Methods 12
- Hessian computation - finite difference methods 12
- Hessian eigenvalues and eigenvectors 14
- Sloppy parameter analyses results 14
- Parameters that did not affect CDP’s performance 15
- Other parameters 17
- Effect of model fit and types of scaling 17
- Differences between the sloppy parameter analyses and optimization 17
- Conclusion 17
- CRediT authorship contribution statement 18
- Declaration of Competing Interest 18
- Data availability 18
- Acknowledgements 18
- Appendix A Supplementary material 18
- References 18
