ShapeLLM: Universal 3D Object Understanding for Embodied Interaction
ShapeLLM: Universal 3D Object Understanding for Embodied Interaction
Zekun Qi Runpei Dong Shaochen Zhang Haoran Geng Chunrui Han Zheng Ge Li Yi Kaisheng Ma

Abstract
This paper presents ShapeLLM, the first 3D Multimodal Large Language Model(LLM) designed for embodied interaction, exploring a universal 3D objectunderstanding with 3D point clouds and languages. ShapeLLM is built upon animproved 3D encoder by extending ReCon to ReCon++ that benefits from multi-viewimage distillation for enhanced geometry understanding. By utilizing ReCon++ asthe 3D point cloud input encoder for LLMs, ShapeLLM is trained on constructedinstruction-following data and tested on our newly human-curated benchmark, 3DMM-Vet. ReCon++ and ShapeLLM achieve state-of-the-art performance in 3Dgeometry understanding and language-unified 3D interaction tasks, such asembodied visual grounding. Project page: https://qizekun.github.io/shapellm/
Code Repositories
Benchmarks
| Benchmark | Methodology | Metrics |
|---|---|---|
| 3d-object-captioning-on-objaverse-1 | ShapeLLM-13B | Sentence-BERT: 48.52 GPT-4: 48.94 SimCSE: 49.98 |
| 3d-object-captioning-on-objaverse-1 | ShapeLLM-7B | Sentence-BERT: 48.20 GPT-4: 46.92 SimCSE: 49.23 |
| 3d-point-cloud-classification-on-modelnet40 | ReCon++ | Overall Accuracy: 95.0 |
| 3d-point-cloud-classification-on-scanobjectnn | ReCon++ | OBJ-BG (OA): 98.80 OBJ-ONLY (OA): 97.59 Overall Accuracy: 95.25 |
| 3d-point-cloud-linear-classification-on | ReCon++ | Overall Accuracy: 93.6 |
| 3d-question-answering-3d-qa-on-3d-mm-vet | ShapeLLM-13B | Overall Accuracy: 53.1 |
| 3d-question-answering-3d-qa-on-3d-mm-vet | ShapeLLM-7B | Overall Accuracy: 47.4 |
| few-shot-3d-point-cloud-classification-on-1 | ReCon++ | Overall Accuracy: 98.0 Standard Deviation: 2.3 |
| few-shot-3d-point-cloud-classification-on-2 | ReCon++ | Overall Accuracy: 99.5 Standard Deviation: 0.8 |
| few-shot-3d-point-cloud-classification-on-3 | ReCon++ | Overall Accuracy: 94.5 Standard Deviation: 4.1 |
| few-shot-3d-point-cloud-classification-on-4 | ReCon++ | Overall Accuracy: 96.5 Standard Deviation: 3.0 |
| generative-3d-object-classification-on-1 | ShapeLLM-13B | Objaverse (Average): 54.00 |
| generative-3d-object-classification-on-1 | ShapeLLM-7B | Objaverse (Average): 54.50 |
| generative-3d-object-classification-on-2 | ShapeLLM-13B | ModelNet40 (Average): 52.96 |
| generative-3d-object-classification-on-2 | ShapeLLM-7B | ModelNet40 (Average): 53.08 |
| zero-shot-transfer-3d-point-cloud | ReCon++ | Accuracy (%): 87.3 |
| zero-shot-transfer-3d-point-cloud-2 | ReCon++ | OBJ_ONLY Accuracy(%): 65.4 |
Build AI with AI
From idea to launch — accelerate your AI development with free AI co-coding, out-of-the-box environment and best price of GPUs.