AgentThink · Tool‑Augmented CoT for Autonomous Driving

Contents

🎬 Demo Showcase

Experience AgentThink's real‑world performance through demonstration materials that illustrate its capabilities in autonomous driving scenarios.

✨ Highlights

• 🔧 Tool‑Augmented Reasoning: Multi‑modal perception via integrated vision, prediction, occupancy, and mapping tools
• 🧠 Reasoning Chain + Tool Calls: Task decomposition with explicit tool invocation
• 🎯 GRPO Training: Triple reward signals (final answer, step‑wise, tool usage)
• 🚀 Performance Boost: 53.91% accuracy improvement over traditional VLM models

📰 Project Updates

• 🎉 [2025‑08‑20] Our paper was accepted as EMNLP 2025 Findings
• 🚀 [2025‑07‑02] v1.1 released with demo and sample data
• 📄 [2025‑05‑22] Paper published on arXiv
• 🎥 Web Demo and Swift full training pipeline coming soon

🚀 Quick Navigation

⚙️ Getting Started · Environment Setup

🛠️ Basic

Basic Setup

# Create virtual environment
conda create -n agentthink python=3.10
conda activate agentthink

# Install dependencies
pip install -r requirements.txt

# Install ms-swift
bash scripts/env.sh

# Install drivemllm dependency
bash scripts/env_drivemllm.sh

Clone ms‑swift

cd third_party
git clone https://github.com/modelscope/ms-swift.git

Model Inference

🎬 Use your trained checkpoint AgentThink to run inference on val samples AgentThink‑CoT‑val:

# Inference script
bash scripts/inference_scripts/inference.sh [your_CKPT_PATH] [your_OUTPUT_DIR]

# Inference with tool script
bash scripts/inference_scripts/inference_withtool.sh [your_CKPT_PATH] [your_OUTPUT_DIR]

# Inference using multi-node GPUs
bash scripts/inference_scripts/inference_multigpu.sh [your_CKPT_PATH] [your_OUTPUT_DIR]

# Inference AgentThink
bash scripts/inference_agentthink.sh [your_CKPT_PATH] [your_OUTPUT_DIR]

Evaluation Metrics

📊 Use LLM‑as‑Judge to calculate performance metrics:

# Evaluate reasoning ability and MCQ accuracy
python evaluation/evaluation_script.py

🚀 Quick Start

Download the model

Our AgentThink model based on the Qwen2.5‑VL‑7B.

Download the tool model

Clone the Depth Anything v2 DAM and YOLO‑World:

git clone https://github.com/DepthAnything/Depth-Anything-V2

git clone https://github.com/AILab-CVC/YOLO-World

Then download the pretrain models in the YOLO‑World and DepthAnything.

Download the basic tool results

Download the val.pkl file from USC‑GVL / Agent‑Driver.

Folder structure

AgentThink/
├── 📂 data/                    # Dataset and processed data
│   ├── DriveLMMo1_TEST_tool_results.jsonl
│   ├── DriveLMMo1_TEST.jsonl
│   ├── 📂 image2concat/        # Concatenated image files
│   └── 📂 tool_results/        # Results from tool processing
│
├── 📂 demo_image/              # Demonstration images
│   ├── nuscenes_CAM_FRONT_3590.webp
│   ├── nuscenes_CAM_FRONT_3757.webp
│   └── nuscenes_CAM_FRONT_3896.webp
│
├── 📂 pretrained_model/        # Pre-trained model files
│   ├── 📂 AgentThink/
│   │   └── checkpoint-700-merged
│   ├── depth_anything_v2_vitb.pth
│   └── yolov8x-world2.pt
│
├── 📂 assets/                  # Visual assets and resources
├── 📂 evaluation/              # Evaluation scripts and benchmarks
├── 📂 Inference/               # Inference-related scripts and data
├── 📂 results/                 # Output and result files
├── 📂 scripts/                 # Various utility scripts
├── 📂 third_party/             # Third-party libraries and resources
├── README.cn.md                # Chinese documentation
├── README.md                   # Project documentation
├── requirements.txt            # Python dependencies
└── ...                         # Other project files

Demo Inference

# drivemllm
python Inference/inference_demo_drivemllm.py

# drivelmm-o1
python Inference/inference_demo_drivelmm.py

📋 TODO List · Development Roadmap

AgentThink Results

📊 DriveLMM‑o1 Performance

📊 DriveMLLM Comparison

📁 Repository Structure

AgentThink/
├── assets/                 # Visual assets and resources
├── data/                   # Data files and datasets
├── evaluation/             # Evaluation scripts and benchmarks
│   ├── evaluation_script.py
│   └── inference_agentthink.py
├── Inference/              # Inference-related scripts and data
│   ├── inference_demo_data_drivemllm.json
│   ├── inference_demo_data_drivelmm.json
│   └── inference_demo_drivemllm.py
├── results/                # Output and result files
│   └── agentthink/
├── scripts/                # Various utility scripts
│   ├── debug_scripts/
│   ├── inference_scripts/
│   └── tools/              # Tool library implementations
├── third_party/            # Third-party libraries and resources
│   ├── 🐍 inference.py         # Main inference script
│   ├── 🐍 prepare_data.py      # Data preparation script
│   ├── 🐍 utlis.py             # Utility functions
│   ├── 🐚 env.sh               # Environment setup script
│   ├── 🐚 env_drivemllm.sh     # DriveMLLM environment script
│   └── 🐚 prepare_json_data.sh # Long JSON data preparation script
├── 📄 README.md            # Project documentation
├── 📄 README_CN.md         # 中文文档
├── 📄 requirements.txt     # Python dependencies

🔗 Related Works

🪪 License & Citation

License

This project is licensed under Apache License 2.0. See LICENSE file for details.

Citation

@misc{qian2025agentthinkunifiedframeworktoolaugmented,
  title={AgentThink: A Unified Framework for Tool-Augmented Chain-of-Thought Reasoning in Vision-Language Models for Autonomous Driving},
  author={Kangan Qian et al.},
  year={2025},
  eprint={2505.15298},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2505.15298},
}