Brain Reconstruction Resources
07/22/2023
The field of mind reconstruction from fMRIs, EEGs, and other noninvasive brain imaging techniques is going through a renaissance of sorts. It's the perfect time to work in this space: thanks to developments in deep learning, compute, and dataset gathering, our progress in understanding what's going on in our minds is mind-blowing (heh).
Here are some resources to get you started. This is an evolving list, so if you have any suggestions, please reach out.
Datasets
There are quite a few large open-source datasets, a few are listed below. For fMRIs it’s common practice to pretrain on the Human Connectome Project resting state dataset because it’s so large and can help your encoder learn the latent space well. Then you can fine tune on task specific datasets.
| Name | Type | Size |
|---|---|---|
| OpenNeuro | MRI, MEG, EEG, iEEG, and ECoG datasets | Free platform for sharing 800+ datasets with 34000+ participants |
| Natural Scenes Dataset | fMRI - image pairs | 8 subjects, each observing 9000-10000 images, ~40,000 voxels per scan |
| BOLD5000 | fMRI - image pairs | 4 subjects, each observing 5254 images, ~1,000 - 2,000 voxels per scan |
| Generic Object Decoding | fMRI - image paris | 5 subjects, each observing 1200 images, ~4,000 - 5,000 voxels per scan |
| Human Connectome Project | Resting state fMRIs | 1206 subjects. 4,000 voxels in visual cortex per scan |
| UK Biobank | MRI, EEG, MEG, multimodal | In depth genetic and health information from half a million UK participants |
| Prof. Zhongming Liu’s dataset | fMRI - movie pairs | 3 subjects, ~3,000 - 6,000 voxels after preprocessing |
The MRC Cognition and Brain Sciences Unit has a great list of datasets as well.
Competitions
These challenges are great to test your skills in a fixed environment. Many of them offer cash prizes, and the opportunity to have your work featured within the competition’s published paper.
| Name | Description |
|---|---|
| Algonauts Challenge | Predict fMRI responses given an image. First, second, third place prize is 1,500 Euros |
| Grand Challenge | A whole directory of ML competitions in biomedical imaging |
| Kaggle | All sorts of deep learning competitions |
Papers
One of the best ways to learn quickly is to read up on foundational papers and then read the papers that cite them. Below are some papers to get you started. I also did a high level walkthrough of some legendary ones here.
| Paper | Contribution | Topic |
|---|---|---|
| Decoding the visual and subjective contents of the human brain | Linear regression to predict which out of N images the subject saw | Image reconstruction |
| Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision | Used CNNs to predict classes from images. One of the first to directly reconstruct an image from fMRI data. | Image reconstruction |
| From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI | Introduced self-supervised learning to fMRI data. Training on unlabelled fmri-image pairs. Use encoder (image -> fmri) and decoder (fmri -> image) components. | Image reconstruction |
| Mind-VIS: Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding | Used attention autoencoder to learn latent space for fMRIs. Used conditioned stable diffusion model for the first time | Image reconstruction |
| High-resolution image reconstruction with latent diffusion models from human brain activity | Directly used MLPs on NSD dataset, with no pretraining. Use early visual cortext to get spacial representation, noise added to get gaussian noise starting image. Use higher (ventral) visual cortext to get semantic information. | Image reconstruction |
| MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion | Used contrastive loss with CLIP to make fMRI embedding map better to the cross attention heads of stable diffusion UNet. | Image reconstruction |
| Sharing deep generative representation for perceived image reconstruction from human brain activity | Using covariance matrix to learn how voxels are connected together. Used low rank assumption to reduce noise and complexity | Image reconstruction |
| Reconstructing the Mind’s Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors | Using low-level + high-level generation with img to img diffusion. Fancier contrastive loss function, using hard+soft contrastive loss, switching 30% of the way through training. | Image reconstruction |
| MindReader: Reconstructing complex images from brain activities | Increasing the signals in "bodies" and "words" ROIs in NSD dataset elevates these concepts in images. FMRI signals contain almost as much high level image info (90%) as CLIP embeddings. | Image reconstruction |
| DreamDiffusion: Generating High-Quality Images from Brain EEG Signals | 2 stage encoder - generator shows that images can be reconstructed from EEG signals. | Reconstruction in other modalities |
Research
If you’d like to do research, most professors are quite open to getting help! Just email them describing what about their work excites you, why you want to help, and how you can help. Most will reply.