Do deep convolutional neural networks accurately model representations beyond the ventral stream?
Abstract
While primate visual cortex has typically been divided into two processing streams, recent research suggests that there may be at least three functionally distinct streams, extending along the ventral, lateral, and parietal surfaces of the brain. Here, we leveraged the Natural Scenes Dataset (Allen et al., 2022) to compare and model responses across these proposed streams. We show that cortical responses cluster by stream and reflect the hierarchical organization of cortex. We then tested how accurately deep convolutional neural networks (DCNNs) trained on supervised object categorization and action recognition objectives could predict responses in each stream. Given the differences in responses across streams and the prevailing view that only the ventral stream serves object categorization, we were surprised to find that these models fit ventral and lateral responses equally well, though they were slightly worse at predicting parietal responses. These findings suggest that additional constraints are required for model predictivity to match the functional organization of visual cortex.
