Labsco
M1ndSmith logo

Pybullet MCP Server

from M1ndSmith

An mcp server for the pybullet library, it supports 20 tools (ex: simulation creation, steps, loading robots urdf, etc)

🔥🔥🔥✓ VerifiedFreeQuick setup
#automation#browser#devops#git#project-mgmt

PyBullet MCP Server

A Model Context Protocol (MCP) server that enables AI assistants to interact with PyBullet physics simulations. Build physics-based projects through natural language interactions with AI agents.

Features

  • 37 MCP Tools: Comprehensive API for physics simulation control including robot joint control
  • Simulation Management: Create and manage multiple independent physics simulations with configurable gravity
  • Object Manipulation: Add primitive shapes (box, sphere, cylinder, capsule) and URDF models with full property control
  • Robot Control: Query joint information, control motors (position/velocity/torque), and calculate inverse kinematics
  • Physics Control: Apply forces, torques, and step through simulations with configurable timesteps
  • State Persistence: Save and load complete simulation states to/from JSON files
  • Constraints: Create joints between objects (fixed, prismatic, spherical)
  • Collision Detection: Query contact points with detailed collision information
  • Visualization: Optional GUI mode with debug visualization and camera control
  • Error Handling: Comprehensive validation with descriptive error messages

Important Notes

  • Coordinate Requirements: All vectors must be provided as complete 3D coordinates [x, y, z]
    • Gravity: [0, 0, -9.81] (not [-9.81])
    • Positions: [x, y, z] (not [x, y] or [x])
    • Forces/Torques: [fx, fy, fz] (not [fx])
    • Orientations: [x, y, z, w] quaternion (not [w])
  • Mass Constraint: Object mass must be positive (mass > 0). Use large mass (e.g., 1000) for static objects
  • GUI Limitation: Only one GUI simulation can be active at a time (PyBullet limitation)
  • URDF Paths: Use absolute paths or paths relative to the server's working directory
  • Revolute Joints: create_constraint does not support "revolute" — use generate_revolute_joint + load_urdf instead. Always pass an explicit output_path inside the workspace to generate_revolute_joint, otherwise the generated URDF lands in /tmp/ and load_urdf will reject it.

Available Tools

The server exposes 37 tools through the MCP protocol:

Simulation Management (5 tools)

  • create_simulation: Initialize a new physics simulation with configurable gravity and optional GUI

    • Parameters: gravity (list[float], default: [0, 0, -9.81]), gui (bool, default: false)
    • Returns: simulation_id, gravity, gui_enabled

  • list_simulations: Get all active simulation IDs

    • Returns: list of simulation IDs

  • destroy_simulation: Clean up and remove a simulation

    • Parameters: sim_id (str)
    • Returns: confirmation message

  • step_simulation: Advance simulation by one or more timesteps

    • Parameters: sim_id (str), steps (int, default: 1)
    • Returns: simulation_id, steps_taken, current_time

  • set_timestep: Configure the timestep duration for a simulation

    • Parameters: sim_id (str), timestep (float)
    • Returns: confirmation message

Object Operations (9 tools)

  • add_box: Add a box shape to the simulation

    • Parameters: sim_id, dimensions (list[float]), position (list[float]), mass (float, default: 1.0), color (list[float], optional)
    • Returns: object_id, shape, position

  • add_sphere: Add a sphere shape to the simulation

    • Parameters: sim_id, radius (float), position (list[float]), mass (float, default: 1.0), color (list[float], optional)
    • Returns: object_id, shape, position

  • add_cylinder: Add a cylinder shape to the simulation

    • Parameters: sim_id, radius (float), height (float), position (list[float]), mass (float, default: 1.0), color (list[float], optional)
    • Returns: object_id, shape, position

  • add_capsule: Add a capsule shape to the simulation

    • Parameters: sim_id, radius (float), height (float), position (list[float]), mass (float, default: 1.0), color (list[float], optional)
    • Returns: object_id, shape, position

  • load_urdf: Load a robot model from a URDF file

    • Parameters: sim_id, file_path (str), position (list[float]), orientation (list[float], optional)
    • Returns: object_id, file_path, position

  • set_object_pose: Update object position and orientation

    • Parameters: sim_id, object_id (int), position (list[float]), orientation (list[float])
    • Returns: confirmation message

  • get_object_state: Query complete object state

    • Parameters: sim_id, object_id (int)
    • Returns: position, orientation, linear_velocity, angular_velocity

  • apply_force: Apply a force vector to an object

    • Parameters: sim_id, object_id (int), force (list[float]), position (list[float], optional)
    • Returns: confirmation message

  • apply_torque: Apply rotational force to an object

    • Parameters: sim_id, object_id (int), torque (list[float])
    • Returns: confirmation message

  • set_object_velocity: Set an object's linear and/or angular velocity directly

    • Parameters: sim_id, object_id (int), linear_velocity (list[float], optional), angular_velocity (list[float], optional)
    • Returns: confirmation message

  • change_dynamics: Modify object physics properties at runtime

    • Parameters: sim_id, object_id (int), link_index (int, default: -1), mass (float, optional), lateral_friction (float, optional), spinning_friction (float, optional), rolling_friction (float, optional), restitution (float, optional), linear_damping (float, optional), angular_damping (float, optional), contact_stiffness (float, optional), contact_damping (float, optional)
    • Returns: confirmation message

  • get_dynamics_info: Query current dynamic properties of an object

    • Parameters: sim_id, object_id (int), link_index (int, default: -1)
    • Returns: mass, lateral_friction, local_inertia_diagonal, restitution, rolling_friction, spinning_friction, contact_damping, contact_stiffness, body_type, collision_margin

Ray Casting (2 tools)

  • ray_test: Cast a single ray to detect obstacles and measure distances

    • Parameters: sim_id, ray_from (list[float]), ray_to (list[float])
    • Returns: hit (bool), object_id, link_index, hit_fraction, hit_position, hit_normal

  • ray_test_batch: Cast multiple rays efficiently for lidar/sensor simulation

    • Parameters: sim_id, rays_from (list[list[float]]), rays_to (list[list[float]])
    • Returns: list of hit results (same fields as ray_test per ray)

Camera Rendering (4 tools)

  • compute_view_matrix: Compute view matrix from camera eye/target/up vectors

    • Parameters: camera_eye_position (list[float]), camera_target_position (list[float]), camera_up_vector (list[float])
    • Returns: view matrix as list of 16 floats

  • compute_view_matrix_from_yaw_pitch: Compute view matrix from spherical coordinates (orbit camera)

    • Parameters: distance (float), yaw (float), pitch (float), target_position (list[float]), up_axis_index (int, default: 2)
    • Returns: view matrix as list of 16 floats

  • compute_projection_matrix: Compute projection matrix from camera parameters

    • Parameters: fov (float), aspect (float), near_plane (float), far_plane (float)
    • Returns: projection matrix as list of 16 floats

  • get_camera_image: Render RGB, depth, and segmentation images from a virtual camera

    • Parameters: sim_id, width (int), height (int), view_matrix (list[float]), projection_matrix (list[float]), renderer (str, default: "ER_BULLET_HARDWARE_OPENGL")
    • Returns: width, height, rgb (base64 PNG), depth (list[float]), segmentation (list[int])

Constraint Management (2 tools)

  • create_constraint: Create a joint between two objects

    • Parameters: sim_id, parent_id (int), child_id (int), joint_type (str), joint_axis (list[float], optional), parent_frame_position (list[float], optional), child_frame_position (list[float], optional)
    • Joint types: "fixed", "prismatic", "spherical"
    • Returns: constraint_id, joint_type
    • Note: "revolute" is NOT supported here — use generate_revolute_joint + load_urdf instead.

  • remove_constraint: Remove a constraint from the simulation

    • Parameters: sim_id, constraint_id (int)
    • Returns: confirmation message

Revolute (Hinge) Joints — Workaround

PyBullet's runtime constraint API does not support revolute joints. The workaround is a two-step process using these two tools:

Step 1 — Generate a URDF with the revolute joint:

  • generate_revolute_joint: Generates a URDF file containing two shapes connected by a revolute joint
    • Parameters: parent_shape (str: "box"/"sphere"/"cylinder"), child_shape (str), parent_dimensions (list[float]), child_dimensions (list[float]), parent_mass (float), child_mass (float), joint_axis (list[float]), joint_origin (list[float], optional), joint_lower_limit (float, default: -π), joint_upper_limit (float, default: π), max_effort (float), max_velocity (float), output_path (str)
    • Returns: urdf_path, parent_shape, child_shape, joint_type, joint_axis, joint_limits
    • output_path must be set to a path inside the server's working directory. If omitted, the file is written to the system temp directory (/tmp/) which is outside the allowed path and will cause load_urdf to fail with an access denied error.

Step 2 — Load the generated URDF:

  • Call load_urdf with the urdf_path returned from generate_revolute_joint
  • Then use set_joint_motor_control to drive the joint

Collision Detection (2 tools)

  • get_all_collisions: Query all contact points in the simulation

    • Parameters: sim_id
    • Returns: list of contact points with positions, normals, forces

  • get_collisions_for_pair: Query contact points between specific objects

    • Parameters: sim_id, obj_a (int), obj_b (int)
    • Returns: list of contact points for the pair

Visualization (2 tools)

  • enable_debug_visualization: Enable debug rendering of contact points and frames

    • Parameters: sim_id, show_contacts (bool, default: true), show_frames (bool, default: false)
    • Returns: confirmation message

  • set_camera: Configure camera position and target for GUI mode

    • Parameters: sim_id, distance (float), yaw (float), pitch (float), target (list[float])
    • Returns: confirmation message

Persistence (2 tools)

  • save_simulation: Save simulation state to a JSON file

    • Parameters: sim_id, file_path (str)
    • Returns: confirmation with file path

  • load_simulation: Load simulation state from a JSON file

    • Parameters: file_path (str), gui (bool, default: false)
    • Returns: new simulation_id, file_path

Robot Control (5 tools)

  • get_num_joints: Query number of joints in a URDF model

    • Parameters: sim_id (str), object_id (int)
    • Returns: number of joints (int)

  • get_joint_info: Get detailed joint properties

    • Parameters: sim_id (str), object_id (int), joint_index (int)
    • Returns: joint_name, joint_type, lower_limit, upper_limit, max_force, max_velocity, joint_axis

  • get_joint_state: Get current joint state

    • Parameters: sim_id (str), object_id (int), joint_index (int)
    • Returns: joint_position, joint_velocity, reaction_forces, motor_torque

  • set_joint_motor_control: Control robot joints

    • Parameters: sim_id (str), object_id (int), joint_index (int), control_mode (str), target_position (float, optional), target_velocity (float, optional), force (float, optional), position_gain (float, optional), velocity_gain (float, optional)
    • Control modes: "POSITION_CONTROL", "VELOCITY_CONTROL", "TORQUE_CONTROL"
    • Returns: confirmation message

  • calculate_inverse_kinematics: Calculate joint angles for target end-effector pose

    • Parameters: sim_id (str), object_id (int), end_effector_link_index (int), target_position (list[float]), target_orientation (list[float], optional), lower_limits (list[float], optional), upper_limits (list[float], optional), joint_ranges (list[float], optional), rest_poses (list[float], optional)
    • Returns: list of joint positions

Example Workflows

Basic Falling Box

Create a simple simulation with a box falling under gravity:

# Through MCP client (natural language):
"Create a simulation with Earth gravity"
"Add a box at position (0, 0, 5) with dimensions 1x1x1"
"Step the simulation 200 times"
"What is the position of object 0?"

Stacked Objects

Create a stack of objects:

"Create a simulation"
"Add a box at (0, 0, 0.5) with dimensions 10x10x1 and mass 1000"  # Ground
"Add a box at (0, 0, 1.5) with dimensions 1x1x1"
"Add a sphere at (0, 0, 3) with radius 0.5"
"Step the simulation 300 times"
"Get all collisions"

Constrained Objects

Create objects connected by a joint:

"Create a simulation"
"Add a box at (0, 0, 2) with dimensions 1x1x1"  # Object 0
"Add a sphere at (2, 0, 2) with radius 0.5"     # Object 1
"Create a fixed constraint between object 0 and object 1"
"Apply a torque of [0, 0, 10] to object 1"
"Step the simulation 200 times"

Save and Load

Persist a simulation:

"Create a simulation"
"Add a box at (0, 0, 1)"
"Add a sphere at (1, 0, 1)"
"Step the simulation 50 times"
"Save the simulation to my_sim.json"

# Later...
"Load the simulation from my_sim.json"
"Step the simulation 50 more times"

Robot Simulation

Load and control a URDF robot model:

"Create a simulation with GUI enabled"
"Load URDF from /path/to/robot.urdf at position (0, 0, 1)"
"How many joints does object 0 have?"
"Get information about joint 0 of object 0"
"Get the current state of joint 0"
"Set joint 0 to position 1.57 with position control and force 100"
"Step the simulation 100 times"
"Get the state of joint 0 again to see it moved"
"Calculate inverse kinematics for object 0 end-effector link 6 at position [0.5, 0, 0.5]"

Robot Control Features:

  • Query number of joints and joint properties (type, limits, axis)
  • Read joint states (position, velocity, forces, torque)
  • Control joints with three modes:
    • Position control: Move joint to target angle/position
    • Velocity control: Spin joint at target speed
    • Torque control: Apply direct torque to joint
  • Calculate inverse kinematics for end-effector positioning

Persistence File Format

Simulation states are saved as JSON files with the following structure:

{
  "gravity": [0.0, 0.0, -9.81],
  "timestep": 0.01,
  "objects": [
    {
      "object_id": 0,
      "type": "primitive",
      "shape": "box",
      "dimensions": [0.5, 0.5, 0.5],
      "position": [0.0, 0.0, 1.0],
      "orientation": [0.0, 0.0, 0.0, 1.0],
      "linear_velocity": [0.0, 0.0, -0.98],
      "angular_velocity": [0.0, 0.0, 0.0],
      "mass": 1.0,
      "color": [1.0, 0.0, 0.0, 1.0]
    },
    {
      "object_id": 1,
      "type": "urdf",
      "urdf_path": "/path/to/model.urdf",
      "position": [2.0, 0.0, 0.5],
      "orientation": [0.0, 0.0, 0.0, 1.0],
      "linear_velocity": [0.0, 0.0, 0.0],
      "angular_velocity": [0.0, 0.0, 0.0]
    }
  ],
  "constraints": [
    {
      "constraint_id": 0,
      "parent_id": 0,
      "child_id": 1,
      "joint_type": "fixed"
    }
  ]
}

Compatibility Notes

  • Object IDs: Object IDs are reassigned when loading (may differ from saved IDs)
  • URDF Files: URDF file paths must be valid when loading. Use absolute paths for reliability
  • Simulation ID: A new simulation ID is generated when loading
  • Format Version: Current format is compatible with PyBullet 3.2.5+
  • Constraints: Constraints are fully serialized and restored

Architecture

The server follows a layered architecture:

MCP Client (Claude Desktop)
         ↓
   MCP Protocol
         ↓
FastMCP Server (37 tools)
         ↓
Manager Classes (helpers)
         ↓
PyBullet Physics Engine

Key Components:

  • FastMCP Server (src/server.py): Exposes 37 MCP tools using @mcp.tool decorators
  • SimulationManager: Manages PyBullet physics clients and simulation lifecycle
  • ObjectManager: Handles object creation, manipulation, and state queries
  • ConstraintManager: Creates and manages joints between objects
  • PersistenceHandler: Serializes/deserializes simulation state to JSON
  • CollisionQueryHandler: Queries contact points and collision information

Each MCP tool validates inputs, calls the appropriate manager, and returns Python objects (dict/list/str/int) directly - FastMCP handles the MCP protocol conversion automatically.

Support

For issues, questions, or contributions:

  • Review this README and troubleshooting section
  • Review PyBullet documentation for physics-specific questions