Labsco
shuji-bonji logo

xcomet-mcp-server

โ˜… 1

from shuji-bonji

Translation quality evaluation using xCOMET models. Provides quality scoring (0-1), error detection with severity levels, and optimized batch processing with 25x speedup.

๐Ÿ”ฅ๐Ÿ”ฅ๐Ÿ”ฅโœ“ VerifiedFreeAdvanced setup
#automation#documents#git#translation

xCOMET MCP Server

npm version CI MCP License: MIT

ๆ—ฅๆœฌ่ชž็‰ˆ README ใฏใ“ใกใ‚‰

โš ๏ธ This is an unofficial community project, not affiliated with Unbabel.

Translation quality evaluation MCP Server powered by xCOMET (eXplainable COMET).

๐ŸŽฏ Overview

xCOMET MCP Server provides AI agents with the ability to evaluate machine translation quality. It integrates with the xCOMET model from Unbabel to provide:

  • Quality Scoring: Scores between 0-1 indicating translation quality
  • Error Detection: Identifies error spans with severity levels (minor/major/critical)
  • Batch Processing: Evaluate multiple translation pairs efficiently (optimized single model load)
  • GPU Support: Optional GPU acceleration for faster inference
Copy & paste โ€” that's it
graph LR
    A[AI Agent] --> B[Node.js MCP Server]
    B -- stdio JSON-RPC --> C[Python Worker]
    C --> D[xCOMET Model<br/>Persistent in Memory]
    D --> C
    C --> B
    B --> A

    style D fill:#9f9

๐Ÿ› ๏ธ Available Tools

xcomet_evaluate

Evaluate translation quality for a single source-translation pair.

Parameters:

NameTypeRequiredDescription
sourcestringโœ…Original source text
translationstringโœ…Translated text to evaluate
referencestringโŒReference translation
source_langstringโŒSource language code (ISO 639-1)
target_langstringโŒTarget language code (ISO 639-1)
response_format"json" | "markdown"โŒOutput format (default: "json")
use_gpubooleanโŒUse GPU for inference (default: false)

Example:

Copy & paste โ€” that's it
{
  "source": "The quick brown fox jumps over the lazy dog.",
  "translation": "็ด ๆ—ฉใ„่Œถ่‰ฒใฎใ‚ญใƒ„ใƒใŒๆ€ ๆƒฐใช็Šฌใ‚’้ฃ›ใณ่ถŠใˆใ‚‹ใ€‚",
  "source_lang": "en",
  "target_lang": "ja",
  "use_gpu": true
}

Response:

Copy & paste โ€” that's it
{
  "score": 0.847,
  "errors": [],
  "summary": "Good quality (score: 0.847) with 0 error(s) detected."
}

xcomet_detect_errors

Focus on detecting and categorizing translation errors.

Parameters:

NameTypeRequiredDescription
sourcestringโœ…Original source text
translationstringโœ…Translated text to analyze
referencestringโŒReference translation
min_severity"minor" | "major" | "critical"โŒMinimum severity (default: "minor")
response_format"json" | "markdown"โŒOutput format
use_gpubooleanโŒUse GPU for inference (default: false)

xcomet_batch_evaluate

Evaluate multiple translation pairs in a single request.

Performance Note: With the persistent server architecture (v0.3.0+), the model stays loaded in memory. Batch evaluation processes all pairs efficiently without reloading the model.

Parameters:

NameTypeRequiredDescription
pairsarrayโœ…Array of {source, translation, reference?} (max 500)
source_langstringโŒSource language code
target_langstringโŒTarget language code
response_format"json" | "markdown"โŒOutput format
use_gpubooleanโŒUse GPU for inference (default: false)
batch_sizenumberโŒBatch size 1-64 (default: 8). Larger = faster but uses more memory

Example:

Copy & paste โ€” that's it
{
  "pairs": [
    {"source": "Hello", "translation": "ใ“ใ‚“ใซใกใฏ"},
    {"source": "Goodbye", "translation": "ใ•ใ‚ˆใ†ใชใ‚‰"}
  ],
  "use_gpu": true,
  "batch_size": 16
}

๐Ÿ”— Integration with Other MCP Servers

xCOMET MCP Server is designed to work alongside other MCP servers for complete translation workflows:

Copy & paste โ€” that's it
sequenceDiagram
    participant Agent as AI Agent
    participant DeepL as DeepL MCP Server
    participant xCOMET as xCOMET MCP Server
    
    Agent->>DeepL: Translate text
    DeepL-->>Agent: Translation result
    Agent->>xCOMET: Evaluate quality
    xCOMET-->>Agent: Score + Errors
    Agent->>Agent: Decide: Accept or retry?

Recommended Workflow

  1. Translate using DeepL MCP Server (official)
  2. Evaluate using xCOMET MCP Server
  3. Iterate if quality is below threshold

Example: DeepL + xCOMET Integration

Configure both servers in Claude Desktop:

Copy & paste โ€” that's it
{
  "mcpServers": {
    "deepl": {
      "command": "npx",
      "args": ["-y", "@anthropic/deepl-mcp-server"],
      "env": {
        "DEEPL_API_KEY": "your-api-key"
      }
    },
    "xcomet": {
      "command": "npx",
      "args": ["-y", "xcomet-mcp-server"],
      "env": {
        "XCOMET_PYTHON_PATH": "~/.xcomet-venv/bin/python3"
      }
    }
  }
}

Then ask Claude:

"Translate this text to Japanese using DeepL, then evaluate the translation quality with xCOMET. If the score is below 0.8, suggest improvements."

โšก Performance

Persistent Worker Architecture (v0.3.0+, stdio since v0.5.0)

The server uses a persistent Python worker process that keeps the xCOMET model loaded in memory. The Node.js MCP server talks to the worker over stdin/stdout using a line-delimited JSON-RPC protocol โ€” no local HTTP listener, no port binding, no FastAPI.

RequestTimeNotes
First request~25-90sModel loading (varies by model size)
Subsequent requests~500msModel already loaded

This provides a 177x speedup for consecutive evaluations compared to reloading the model each time.

Eager Loading (v0.3.1+)

Enable XCOMET_PRELOAD=true to pre-load the model at server startup:

Copy & paste โ€” that's it
{
  "mcpServers": {
    "xcomet": {
      "command": "npx",
      "args": ["-y", "xcomet-mcp-server"],
      "env": {
        "XCOMET_PRELOAD": "true"
      }
    }
  }
}

With preload enabled, all requests are fast (~500ms), including the first one.

Copy & paste โ€” that's it
graph LR
    A[MCP Request] --> B[Node.js Server]
    B -- stdio JSON-RPC --> C[Python Worker]
    C --> D[xCOMET Model<br/>in Memory]
    D --> C
    C --> B
    B --> A

    style D fill:#9f9

Batch Processing Optimization

The xcomet_batch_evaluate tool processes all pairs with a single model load:

PairsEstimated Time
10~30-40 sec
50~1-1.5 min
100~2 min

GPU vs CPU Performance

Mode100 Pairs (Estimated)
CPU (batch_size=8)~2 min
GPU (batch_size=16)~20-30 sec

Note: GPU requires CUDA-compatible hardware and PyTorch with CUDA support. If GPU is not available, set use_gpu: false (default).

Best Practices

1. Let the persistent server do its job

With v0.3.0+, the model stays in memory. Multiple xcomet_evaluate calls are now efficient:

Copy & paste โ€” that's it
โœ… Fast: First call loads model, subsequent calls reuse it
   xcomet_evaluate(pair1)  # ~90s (model loads)
   xcomet_evaluate(pair2)  # ~500ms (model cached)
   xcomet_evaluate(pair3)  # ~500ms (model cached)

2. For many pairs, use batch evaluation

Copy & paste โ€” that's it
โœ… Even faster: Batch all pairs in one call
   xcomet_batch_evaluate(allPairs)  # Optimal throughput

3. Memory considerations

  • XCOMET-XL requires ~8-10GB RAM
  • For large batches (500 pairs), ensure sufficient memory
  • If memory is limited, split into smaller batches (100-200 pairs)

Auto-Restart (v0.3.1+)

The server automatically recovers from failures:

  • Monitors health every 30 seconds
  • Restarts after 3 consecutive health check failures
  • Up to 3 restart attempts before giving up

๐Ÿ“Š Quality Score Interpretation

Score RangeQualityRecommendation
0.9 - 1.0ExcellentReady for use
0.7 - 0.9GoodMinor review recommended
0.5 - 0.7FairPost-editing needed
0.0 - 0.5PoorRe-translation recommended

๐Ÿงช Development

Copy & paste โ€” that's it
# Install dependencies
npm install

# Build TypeScript
npm run build

# Watch mode
npm run dev

# Run tests
npm test

# Test with MCP Inspector
npm run inspect

๐Ÿ“‹ Changelog

See CHANGELOG.md for version history and updates.

๐Ÿ“ License

MIT License - see LICENSE for details.

๐Ÿ™ Acknowledgments

๐Ÿ“š References