Exploring Human-Like Object Representations in LLM

Exploring Human-Like Object Representations in Large Language Models: In recent years, artificial intelligence systems—especially large language models (LLMs) and their multimodal counterparts (MLLMs)—have shown remarkable abilities to process language and even “understand” images. However, one pressing question remains: do these systems internally represent natural objects in a manner akin to human cognition? A recent study titled “Human-like object concept representations emerge naturally in multimodal large language models” provides compelling evidence that they do. In this post, we break down the study’s methodology, key results, and broader implications for both AI research and cognitive science.

The Motivation: Bridging AI Representations and Human Cognition

Humans are experts at categorizing and conceptualizing a wide array of objects. Whether it is distinguishing between animate and inanimate items or identifying the subtle features of a fruit, our mental representations capture rich semantic and perceptual information. The study in question asks: > Can LLMs develop similar low-dimensional, interpretable representations simply by processing massive amounts of language—and, in the case of MLLMs, image—data?

By drawing parallels between human behavior, language-based judgments, and neural activity, the researchers set out to test the alignment between AI representations and human object perception.

Methodology: Uncovering Hidden Dimensions Through Behavioral Tasks

The Odd-One-Out Paradigm

At the heart of the study is the classic triplet odd-one-out task—a cognitive psychology paradigm wherein participants (or models) are presented with three objects and asked to choose the one that is least similar to the other two. In this work, the researchers harnessed an enormous behavioral dataset:

4.7 million triplets were collected, with analogous experiments run for human participants (via Amazon Mechanical Turk) and for AI systems like ChatGPT-3.5 (an LLM) and Gemini Pro Vision (an MLLM).

This approach allowed the team to tap into the underlying “mental” representation the participants used. By asking, “Which of these three objects does not belong?” the study could derive similarity structures that reflect conceptual understanding.

Learning Interpretable Embeddings via SPoSE

To transform the triplet judgment data into a form that could be compared across humans and models, the researchers employed the Sparse Positive Similarity Embedding (SPoSE) method. This algorithm infers low-dimensional representations (in this case, 66 dimensions) that capture similarity judgments:

Low-Dimensional Embeddings: These 66 dimensions were shown to be both stable and predictive, meaning they capture the essence of object similarities in a way that mirrors human conceptual clusters (e.g., animals, food, vehicles).
Interpretable Dimensions: Each latent dimension could be associated with an intuitive label (e.g., “food-related,” “weapon-related,” “animal-related”). Such interpretability is crucial in understanding what features the system uses when it “judges” objects.

For example, if two objects score similarly on dimensions related to texture or color, they are more likely to be grouped together. The SPoSE approach has been validated previously in human behavior studies and now proves its versatility in evaluating both AI and human data.

Comparing Against Neural Data

To further solidify the study’s claims of “human-like” representation, the low-dimensional embeddings were compared against neuroimaging data:

fMRI Data and NSD: The study leveraged the Natural Scenes Dataset (NSD), one of the largest fMRI datasets available (NSD fMRI dataset). By applying Representational Similarity Analysis (RSA), the researchers compared the representational geometry of AI embeddings to neural activity within key brain regions, such as the extrastriate body area (EBA), parahippocampal place area (PPA), and fusiform face area (FFA).

The high correspondence between the embeddings and the neural response patterns suggests that these AI models, particularly when trained on both images and language, develop conceptual representations that echo the organization of human brain responses.

Key Findings: Emergence of Human-Like Conceptual Structures

1. Stability and Predictiveness

Robust Embeddings: Across multiple experimental runs—with different random seeds—the 66-dimensional embeddings remained stable. Such reproducibility indicates that the dimensions are not artifacts of the training process but genuinely capture organizing principles of the object space.
Behavioral Prediction: When tested on held-out triplet data, the SPoSE embeddings predicted odd-one-out choices with remarkable accuracy. For example, while humans showed around 64% accuracy (relative to chance at 33%), the models not only approached but in some cases nearly matched human-level predictability.

2. Emergent Category Information

Natural Clustering: By applying visualization techniques (like t-SNE), the study revealed that objects naturally form clusters in the embedding space. These clusters largely mirror known human categories (e.g., animate vs. inanimate, food items, vehicles).
Dimensional Overlap: A significant number of dimensions in the LLM and MLLM embeddings corresponded closely with those derived from human similarity judgments. This overlap implies that, even without explicit training on human behavioral tasks, AI systems learn representations that align with human conceptual knowledge.

3. Alignment with Brain Representations

Neural Correspondence: Using searchlight RSA and voxel-wise encoding techniques, the researchers showed that MLLM embeddings, in particular, strongly predicted neural activity in category-selective brain regions. In several instances, MLLM performance reached up to 85–90% of human performance in terms of predictability.
Modality Matters: While pure language models (LLMs) do develop interpretable dimensions, the integration of visual input in MLLMs enhances the representation of perceptual features such as shape and color—features that are critical in human object recognition.

4. Guided Attention and Prompt Engineering

The study also experimented with guiding model behavior through tailored prompts. For example:

Dimensional Guidance: Adding phrases like “consider the aspect of ‘red’ or ‘color’” led the models to prioritize dimensions more in line with human judgment. Similarly, specifying “human-made” or “artificial” as evaluation criteria shifted the model’s odd-one-out choices accordingly.
Dimension Masking: Attempts to mask specific dimensions revealed that while some shifts did occur, directly suppressing key dimensions may not reliably reorient model behavior. These findings underscore both the flexibility and the challenge of interpreting and controlling AI representations.

Implications and Future Directions

Enhancing Human–Machine Interfaces

The fact that LLMs and MLLMs develop low-dimensional, interpretable representations opens exciting possibilities:

Better Alignment with Human Reasoning: By tuning prompts to guide model attention, we may further align AI decision-making with human preferences, improving overall usability.
Collaborative AI Systems: Understanding the common conceptual basis between human and machine representations can pave the way for improved interfaces and collaborative systems where human and AI “speak the same language.”

Bridging Cognitive Science and AI

The study reinforces an emerging narrative: the representations gleaned by large-scale AI systems are not arbitrary constructs, but share fundamental properties with human cognition. This convergence offers researchers a unique window into both advanced AI interpretability techniques and the neural underpinnings of human object perception.

Future Research

While the study primarily focused on ChatGPT-3.5 and Gemini Pro Vision, future work could:

Extend these analyses to other state-of-the-art models like GPT-4V.
Explore the impact of different types of object-level versus image-level annotations.
Use instruction fine-tuning on large-scale odd-one-out tasks to further refine model-human alignment.

For those interested in digging deeper into the methods or accessing the datasets used, note that the study provides several resources:

THINGS Database: https://osf.io/jum2f/
Preprocessed Behavioral Data: Human and model datasets
Source Code on GitHub: https://github.com/ChangdeDu/LLMs_core_dimensions115

The Final Nut

This study represents an important step in understanding how advanced language and vision–language models can develop conceptual representations that mirror human cognition. By leveraging a robust behavioral paradigm, innovative dimensionality reduction techniques, and direct comparisons with neural data, the researchers demonstrate that the latent spaces of modern AI models are not merely statistical constructs—they echo the fundamental ways in which humans organize and interpret the world. As research in this field continues, the insights gained could lead to more human-like, interpretable, and flexible AI systems that seamlessly integrate with human thought and behavior.

Whether you are an AI researcher, a cognitive scientist, or just curious about how machines might “think” like us, this study offers a fascinating glimpse into the convergence of artificial and biological intelligence.

any questions feel free to contact us or comment below

Just A Squirrel Who Loves Art & AI Tech

Chip Dee

Stay tuned for more insights into the world of AI and cognitive science as these domains continue to intertwine.

Deez Nuts

Exploring Human-Like Object Representations in Large Language Models