Exploring the shared representations between PlaceNet365 and ERP responses to natural scenes

Abstract

Understanding how the human brain processes natural scenes is a fundamental question within cognitive neuroscience. Numerous studies have used Representational Similarity Analysis (RSA) to compare deep neural network activations with fMRI and MEG signals, showing hierarchical correspondences in visual processing. However, while relatively few studies have applied RSA between convolutional neural network (CNN) activations and EEG data, to our knowledge, our work is the first to apply this methodology to examine attentional state and stimulus representativeness influence this alignment during natural scene perception. In this thesis, a fine-tuned ResNet18-Places365 model achieving 90.97% scene classification accuracy served as the computational basis for extracting image similarity structures. EEG data were preprocessed and averaged into event-related potentials (ERPs), which were used to construct Representational Similarity Matrices (RSMs) for comparison with model layers. Additionally, an analysis using a support vector machine (SVM) identified significant ERP clusters differentiating good and bad stimuli. RSA results revealed that mid-to-late CNN layers correlated significantly with EEG responses, particularly around 256–312 ms post-stimulus, under both attention conditions. Notably, good exemplars showed a consistent alignment with deeper CNN layers, while bad exemplars under the attention condition elicited stronger neural-model correspondence in mid-level layers. These findings support the interpretation of the N300 as reflecting mid-to-high-level perceptual processes and demonstrate that meaningful brain-CNN alignment persists even during distraction.