Labsco
cougz logo

GitPrism

★ 1

from cougz

GitPrism is a fast, token-efficient, stateless pipeline that converts public GitHub repositories into LLM-ready Markdown.

🔥🔥🔥🔥✓ VerifiedFreeAdvanced setup
#automation#database#documents#cloud#devops#git

GitPrism

Dashboard

A fast, token-efficient, stateless pipeline that converts public GitHub repositories into LLM-ready Markdown. Deployed as a single Cloudflare Worker serving humans, AI agents, and MCP clients from one shared core engine.

                    ┌─────────────────────────────────────────────┐
                    │          Single Cloudflare Worker            │
                    │               (gitprism)                    │
                    │                                             │
   Humans ────────► │  /              → Astro Static UI           │
                    │                   (Workers Static Assets)   │
                    │                                             │
   AI Agents ─────► │  /ingest?...    → REST API                  │
                    │  /<github-url>  → URL Proxy (shorthand)     │
                    │                                             │
   MCP Clients ───► │  /mcp           → Stateless MCP Server      │
                    │                   (createMcpHandler)        │
                    │                                             │
                    │         ┌───────────────────┐               │
                    │         │   Core Engine      │               │
                    │         │  URL Parser        │               │
                    │         │  Zipball Fetch     │               │
                    │         │  fflate Decomp     │               │
                    │         │  Filter/Ignore     │               │
                    │         │  MD Formatter      │               │
                    │         └───────────────────┘               │
                    └─────────────────────────────────────────────┘
                                       │
                                       ▼
                              GitHub Zipball API
                          (authenticated via secret)

Authentication

GitPrism supports optional GitHub personal access tokens to bypass shared rate limits and use your personal GitHub API quota.

Default Rate Limits

  • Without authentication: 30 requests per minute per IP address
  • With server token: Shared across all users (5,000 req/hr for the Worker's IP)
  • With user token: Your personal GitHub quota (5,000 requests per hour)

Using Your Own Token

REST API:

curl -H "X-GitHub-Token: ghp_xxxx" https://gitprism.cloudemo.org/ingest?repo=owner/repo

Web UI: Click the ⚙️ settings icon in the search bar to open the token settings panel. Your token is stored in localStorage and never sent to any server other than GitHub via this Worker.

MCP Tool:

{
  "url": "https://github.com/owner/repo",
  "detail": "full",
  "github_token": "ghp_xxxx"
}

Token Requirements

Response Headers

The X-Token-Source header indicates which token was used:

  • user — Your personal token (via X-GitHub-Token header)
  • server — The Worker's shared token
  • none — No token available

Development

# Build the Astro UI (required before deploying or running wrangler dev)
npm run build

# Run tests (169 tests)
npm test

# Watch mode
npm run test:watch

# Type-check
npm run typecheck

# Local dev server (requires ui/dist/ to exist — run npm run build first)
npm run dev

Architecture

Project Structure

gitprism/
├── src/
│   ├── index.ts              # Worker entry point, routing
│   ├── types.ts              # Shared interfaces and error classes
│   ├── engine/
│   │   ├── parser.ts         # URL parsing and validation
│   │   ├── fetcher.ts        # GitHub zipball download + size check
│   │   ├── decompressor.ts   # fflate decompression + processing
│   │   ├── filter.ts         # Ignore lists, .gitignore, binary detection
│   │   ├── formatter.ts      # Markdown output generators (4 levels)
│   │   └── ingest.ts         # Shared pipeline (used by API + MCP)
│   ├── mcp/
│   │   └── server.ts         # createMcpHandler setup
│   ├── api/
│   │   ├── handler.ts        # REST API handler, streaming, caching
│   │   └── llmstxt.ts        # /llms.txt endpoint
│   └── utils/
│       ├── cache.ts          # Workers Cache API helpers
│       ├── ratelimit.ts      # Rate limiting helper
│       └── headers.ts        # Response header builder
├── test/                     # Vitest test files (169 tests)
├── ui/
│   ├── src/                  # Astro source
│   ├── dist/                 # Build output (gitignored)
│   └── astro.config.mjs
├── PLAN.md                   # Detailed implementation plan
└── wrangler.jsonc

Key Decisions

DecisionRationale
Single Worker (no Pages)Workers Static Assets is the recommended approach. No CORS, simpler deployment.
createMcpHandler() (no Durable Objects)Tool is stateless. No per-session state needed.
fflate over jszipStreaming decompression, smaller bundle, lower peak memory in V8 isolates.
Server-side GITHUB_TOKENRaises rate limit from 60 to 5,000 req/hr without user auth.
Pre-flight size checkPrevents OOM crashes from large repos.
Cache API from day oneIdentical repo+ref+detail produces identical output. Caching cuts latency and GitHub API usage.
Streaming TransformStream for fullReduces peak memory, improves time-to-first-byte.

File Filtering

Hardcoded Ignore List

The following are always excluded regardless of .gitignore:

Directories: node_modules/, vendor/, .git/, __pycache__/, .venv/, venv/, dist/, build/, .next/, .nuxt/, .svelte-kit/, .output/, .cache/, .parcel-cache/, coverage/, .tox/, .mypy_cache/

Files: package-lock.json, yarn.lock, pnpm-lock.yaml, bun.lockb, Cargo.lock, composer.lock, Gemfile.lock, go.sum, poetry.lock, *.min.js, *.min.css, *.map, *.wasm, *.pb.go, *.pyc, *.pyo

Binary extensions: .png, .jpg, .jpeg, .gif, .ico, .webp, .bmp, .tiff, .svg, .woff, .woff2, .ttf, .eot, .otf, .pdf, .zip, .tar, .gz, .bz2, .7z, .rar, .exe, .dll, .so, .dylib, .bin, .o, .a, .mp3, .mp4, .avi, .mov, .mkv, .flac, .wav, .ogg, .sqlite, .db, .DS_Store

Binary content detection: Files containing null bytes in their first 8 KB are skipped regardless of extension.

.gitignore Support

The root .gitignore of the repository is parsed and applied. Supports:

  • Wildcard patterns (*.log, **/*.tmp)
  • Directory patterns with trailing slash (logs/)
  • Rooted patterns (/build)
  • Negation patterns (!important.log)
  • Comments (# this line is ignored)

Limitation: Only the root .gitignore is evaluated. Nested .gitignore files (e.g., src/.gitignore) are not supported in v1.

Code Mode Compatibility

The ingest_repo MCP tool is compatible with Code Mode agents by design:

  • Clear, descriptive tool name (ingest_repo)
  • Multi-sentence description explaining all four detail levels
  • Strongly-typed Zod schemas with .describe() on every parameter
  • No server-side changes needed — standard MCP tools with typed schemas are inherently Code Mode compatible

Limits

LimitValueConfigurable
Max zip archive size50 MBMAX_ZIP_BYTES env var
Max output size10 MBMAX_OUTPUT_BYTES env var
Max file count5,000MAX_FILE_COUNT env var
Rate limit (no token)30 req/min per IPwrangler.jsonc ratelimits binding
Rate limit (user token)5,000 req/hrGitHub's per-user quota
Cache TTL24 hoursCACHE_TTL_SECONDS env var

Rate limit behavior:

  • Without X-GitHub-Token: Cloudflare rate limiter enforces 30 requests per minute per IP
  • With X-GitHub-Token: Cloudflare rate limiter is bypassed; your personal GitHub quota applies (5,000 req/hr)
  • The X-Token-Source response header indicates which token was used (user, server, or none)

Caching behavior:

Cache keys use resolved commit SHAs for automatic invalidation when repos update. Old cache entries expire naturally after TTL. If SHA resolution fails, caching is skipped and fresh data is always fetched.