# Tracking Semantic Representations Across the Visual Hierarchy (Kay et al., 2008 Dataset) Does the early and intermediate visual areas (V1–V4, V3A/B, Lateral Occipital) reflect not just physical features, but also semantic structure? This project investigates whether neural responses in these areas encode semantic similarity between natural images. In other words, when two images are conceptually similar (e.g., both are dogs), do their voxel activation patterns also become more correlated? The analysis is based on Representational Similarity Analysis (RSA), comparing the structure of semantic relationships between image labels and voxel-wise neural activity patterns. # Dataset Description - Source: Kay et al., *Identifying Natural Images from Human Brain Activity* (Nature, 2008) - Participants: TWO human subjects performing passive viewing of 1,750 natural images / each image repeated 13× - Data: - responses: voxel-wise fMRI responses (1750 × 8428) - stimuli: grayscale visual stimuli (1750 × 128 × 128) - roi: ROI labels for each voxel (V1–V4, LatOcc, etc.) - labels: hierarchical semantic labels (4 levels) # Analysis Pipeline 1. Data loading & preprocessing - Load fMRI responses, ROIs, and hierarchical semantic labels - Standardize voxel responses (Z-scoring) 2. Semantic Similarity Matrix - Compute a 1750×1750 similarity matrix based on hierarchical labels - Weighted scheme: Level 1–4 weights = 4, 3, 2, 1 (sum = 10 for full match) 3. Neural Similarity Matrix - Compute voxel-wise Pearson correlation between fMRI patterns per ROI 4. Representational Similarity Analysis (RSA) - Compute correlation between semantic and neural similarity matrices - Implemented for each ROI (V1, V2, V3, V3A, V3B, V4, LatOcc) 5. Visualization - Scatterplots of semantic vs neural similarity - Bar plot of effect sizes across ROIs - # Results Summary | ROI | Pearson r | |--------|------------| | V1 | 0.002 | | V2 | 0.004 | | V3 | 0.006 | | V3A | 0.012 | | V3B | 0.023 | | V4 | 0.017 | | LatOcc | 0.039 | These results show a gradual increase in semantic–neural correspondence from early (V1/V2) to higher-level visual areas (V4/LatOcc). # Tools and Environment - Python 3.13.2 - Libraries: numpy, matplotlib, seaborn, pandas, scipy - Jupyter Notebook for analysis - Data source: [Colab link](https://colab.research.google.com/github/NeuromatchAcademy/course-content/blob/main/projects/fMRI/load_kay_images.ipynb) # Project Structures Kay2008-Semantic-RSA/ ├── .gitignore ├── README.md ├── requirements.txt │ ├── data/ │ ├── .gitkeep │ ├── kay_images.npz │ ├── kay_labels.npy │ └── semantic_similarity.npy │ ├── notebooks/ │ ├── load-dataset.ipynb │ ├── semantic-similarity.ipynb │ ├── neural-similarity-and-RSA.ipynb │ ├── scripts/ │ ├── README.md │ └── rsa_visualization.ipynb │ ├── results/ │ ├── rsa_basic.csv │ ├── rsa_perm_bootstrap.csv │ └── rsa_weighted.csv │ └── figures/ ├── V1_scatter.png ├── V2_scatter.png ├── V3_scatter.png ├── V3A_scatter.png ├── V3B_scatter.png ├── V4_scatter.png ├── LatOcc_scatter.png └── rsa_barplot.png # Author Fereshteh Fathiziarani Undergraduate Researcher, Psychology student This project was developed as part of the Neuromatch Academy Computational Neuroscience course (2025). # References - Kay, K. N., Naselaris, T., Prenger, R. J., and Gallant, J. L. (2008). *Identifying natural images from human brain activity*. Nature, 452(7185): 352–355. [doi:10.1038/nature06713](https://doi.org/10.1038/nature06713) - Naselaris, T., Prenger, R. J., Kay, K. N., Oliver, M., and Gallant, J. L. (2009). *Bayesian reconstruction of natural images from human brain activity*. Neuron, 63(6): 902–915. [doi:10.1016/j.neuron.2009.09.006](https://doi.org/10.1016/j.neuron.2009.09.006) # Kay2008-Semantic-RSA