Labsco
mirkosertic logo

MCP Lucene Server

★ 4

from mirkosertic

MCP Lucene Server is a Model Context Protocol (MCP) server that exposes Apache Lucene's full-text search capabilities through a conversational interface. It allows AI assistants (like Claude) to help users search, index, and manage document collections without requiring technical knowledge of Lucene or search engines.

🔥🔥🔥🔥✓ VerifiedFreeAdvanced setup
#automation#scraping#database#documents#devops#git#maps#security

MCP Lucene Server

Build and Release

A Model Context Protocol (MCP) server that exposes Apache Lucene fulltext search capabilities with automatic document crawling and indexing. This server supports both STDIO transport (for Claude Desktop integration) and HTTP transport (for web-based clients and remote access).

Features

Automatic Document Crawling

  • Automatically indexes PDFs, Microsoft Office, and OpenOffice documents
  • Multi-threaded crawling for fast indexing
  • Real-time directory monitoring for automatic updates
  • Incremental indexing with full reconciliation (skips unchanged files, removes orphans)

Powerful Search

  • Simple keyword search (no Lucene syntax needed) and full Lucene query syntax search
  • Field-specific filtering (by author, language, file type, etc.)
  • Structured passages with quality metadata for LLM consumption
  • Paginated results with filter suggestions

Semantic Search

  • Optional pure KNN embedding-based semantic search using multilingual-e5 embeddings with Late Chunking
  • Finds semantically related documents even without exact keyword matches
  • Requires VECTOR_MODEL to be configured. See SEMANTICSEARCH.md for details.

Query Profiling & Debugging

  • Deep query analysis and profiling (profileQuery tool)
  • Understand why queries return certain results and how scoring works
  • Filter impact analysis showing document reduction per filter
  • Document scoring explanations with BM25 breakdown
  • Term statistics (IDF, rarity, document frequency)
  • Actionable optimization recommendations
  • LLM-optimized structured output for easy interpretation

Rich Metadata Extraction

  • Automatic language detection
  • Author, title, creation date extraction
  • File type and size information
  • SHA-256 content hashing for change detection

JDBC Metadata Enrichment

  • Load additional metadata from PostgreSQL, MySQL, or any JDBC-compatible database at index time
  • JSON-based metadata with explicit field types (keyword, text, int, long, date)
  • Multi-value field support, automatic facet registration
  • Background sync job for incremental metadata updates (configurable interval)
  • All DB-sourced fields prefixed with dbmeta_ to avoid schema collisions

Text Normalization

  • Automatic removal of broken/invalid characters (replacement chars, control chars, zero-width chars)
  • Whitespace normalization (multiple spaces collapsed to single space)
  • Ensures clean, readable search results and passages

Performance Optimized

  • Batch processing for efficient indexing
  • NRT (Near Real-Time) search with dynamic optimization
  • Configurable thread pools for parallel processing
  • Progress notifications during bulk operations

Easy Integration

  • Dual transport support: STDIO (default) and HTTP
  • STDIO transport for seamless Claude Desktop integration
  • HTTP transport for web-based clients and remote access
  • Comprehensive MCP tools for search and crawler control
  • Flexible configuration via YAML and system properties
  • Cross-platform notifications (macOS Notification Center, Windows Toast, Linux notify-send)

Table of Contents

Documentation

Additional technical documentation:

  • PIPELINE.md — Analyzer chains, query pipeline, and tokenization details
  • SEMANTICSEARCH.md — Semantic search architecture: Late Chunking, Block Join indexing, KNN scoring, and configuration
  • ONNX.md — ONNX model export, optimization and INT8 quantization guide for e5-base and e5-large

MCP Tools

Tools are organized into groups. Use LUCENE_TOOLS_INCLUDE and LUCENE_TOOLS_EXCLUDE to control which tools are exposed (see Tool Exposure Configuration).

Search Tools (group: search)

simpleSearch

Search the Lucene fulltext index using plain text keyword search. Special characters are treated as literals — no Lucene syntax knowledge required. Uses BM25 with German and English stemming.

Parameters:

  • query (optional): Plain text search query. Can be null or "*" to match all documents (useful with filters).
  • filters (optional): Array of structured filters for precise field-level filtering (see Structured Filters below)
  • page (optional): Page number, 0-based (default: 0)
  • pageSize (optional): Results per page (default: 10, max: 100)
  • sortBy (optional): Sort field - _score (default), modified_date, created_date, file_size, or any dbmeta_* metadata field (INT/LONG/DATE/KEYWORD) registered from JDBC enrichment
  • sortOrder (optional): Sort order - asc or desc (default: desc)

extendedSearch

Search the Lucene fulltext index using full Lucene query syntax. Supports Boolean operators, wildcards, proximity queries, and field-specific queries. Uses BM25 with German and English stemming.

Parameters:

  • query (optional): The search query using Lucene query syntax. Can be null or "*" to match all documents (useful with filters).
  • filters (optional): Array of structured filters for precise field-level filtering (see Structured Filters below)
  • page (optional): Page number, 0-based (default: 0)
  • pageSize (optional): Results per page (default: 10, max: 100)
  • sortBy (optional): Sort field - _score (default), modified_date, created_date, file_size, or any dbmeta_* metadata field (INT/LONG/DATE/KEYWORD) registered from JDBC enrichment
  • sortOrder (optional): Sort order - asc or desc (default: desc)

Sorting Results:

By default, results are sorted by relevance score (most relevant first). You can sort by metadata fields:

Sort FieldDescriptionDefault Order
_scoreRelevance score (default)Descending (best match first)
modified_dateLast modified dateDescending (most recent first)
created_dateCreation dateDescending (most recent first)
file_sizeFile size in bytesDescending (largest first)
dbmeta_*Any single-valued JDBC metadata field with type INT, LONG, DATE, or KEYWORDAscending or descending

Sort Examples:

// Most recently modified documents
{ "query": "contract", "sortBy": "modified_date", "sortOrder": "desc" }

// Oldest documents first
{ "query": "contract", "sortBy": "created_date", "sortOrder": "asc" }

// Smallest files (for quick review)
{ "query": "summary", "sortBy": "file_size", "sortOrder": "asc" }

// Combine sorting with filters
{
  "query": "*",
  "sortBy": "modified_date",
  "sortOrder": "desc",
  "filters": [
    { "field": "file_extension", "value": "pdf" },
    { "field": "modified_date", "operator": "range", "from": "2024-01-01" }
  ]
}

Note: When sorting by metadata fields, relevance scores are still computed and used as a secondary sort criterion for tie-breaking.

Structured Filters:

The filters array accepts objects with these fields:

FieldRequiredDescription
fieldyesField name to filter on
operatornoeq (default), in, not, not_in, range
valuefor eq/notSingle value for exact match or exclusion
valuesfor in/not_inArray of values (OR semantics within the field)
fromfor rangeRange start (inclusive)
tofor rangeRange end (inclusive)
addedAtnoClient timestamp — round-tripped in activeFilters response

Operator reference:

OperatorDescriptionExample
eqExact match (default){field: "language", value: "en"}
inMatch any of values{field: "file_extension", operator: "in", values: ["pdf", "docx"]}
notExclude value{field: "language", operator: "not", value: "unknown"}
not_inExclude multiple values{field: "language", operator: "not_in", values: ["unknown", ""]}
rangeNumeric/date range{field: "modified_date", operator: "range", from: "2024-01-01", to: "2025-12-31"}

Filterable fields:

  • Faceted (DrillSideways): language, file_extension, file_type, author
  • String (exact match): file_path, content_hash
  • Numeric/date (range): file_size, created_date, modified_date, indexed_date

Date format: ISO-8601 — "2024-01-15", "2024-01-15T10:30:00", or "2024-01-15T10:30:00Z"

Filter combination rules:

  • Filters on different fields use AND logic
  • Multiple eq filters or in values on the same faceted field use OR logic (DrillSideways)
  • not/not_in filters are applied as MUST_NOT clauses

AI-Powered Synonym Expansion:

This server is designed to work with AI assistants like Claude. Instead of using traditional Lucene synonym files, the AI generates context-appropriate synonyms automatically by constructing OR queries.

Why this is better than traditional synonyms:

  • Context-aware: The AI understands your intent and picks relevant synonyms (e.g., "contract" in legal context vs. "contract" in construction)
  • No maintenance: No need to maintain static synonym configuration files
  • Domain-adaptive: Works across legal, technical, medical, or casual language automatically
  • Multilingual: Generates synonyms in any language without configuration

When you ask Claude to "find documents about cars", it automatically searches for (car OR automobile OR vehicle) - giving you better results than a static synonym list.

Technical Details (Lexical Matching):

The server uses a multi-analyzer indexing pipeline and multi-field weighted query pipeline for comprehensive search:

  • Unicode normalization — NFKC normalization, diacritic folding, ligature expansion via ICUFoldingFilter
  • Leading wildcard optimization — content_reversed field stores reversed tokens for efficient *vertrag-style queries
  • Case-insensitive wildcards — wildcard/prefix terms are automatically lowercased
  • OpenNLP lemmatization — dictionary-based lemmatization for German and English, including irregular forms (ran→run, ging→gehen, paid→pay, analyses→analysis)
  • Dual-language indexing — both German and English lemma fields indexed for all documents, enabling mixed-language matching
  • German umlaut transliteration — content_translit_de shadow field maps digraphs (Mueller→Müller)
  • Automatic phrase expansion — exact phrases auto-expand to include proximity matches (see below)
  • Adaptive prefix scoring — BM25 scoring for specific prefixes (>= 4 chars)

See PIPELINE.md for complete analyzer chain documentation, concrete examples, and query pipeline details.

The AI assistant compensates for remaining limitations (no synonym expansion, no phonetic matching) by expanding queries intelligently.

Best Practices for Better Results:

  1. Generate Synonyms Yourself: Use OR to combine related terms:

    • Instead of: contract
    • Use: (contract OR agreement OR deal)

  2. Use Wildcards for Variations: Handle different word forms:

    • Instead of: contract
    • Use: contract* (matches contracts, contracting, contracted)

  3. Leverage Facets: Use the returned facet values to discover exact terms in the index:

    • Check facets.author to find exact author names
    • Check facets.language to see available languages
    • Use these exact values for filtering

  4. Combine Techniques:

    (contract* OR agreement*) AND (sign* OR execut*) AND author:"John Doe"

Supported Query Syntax (extendedSearch):

  • Simple terms: hello world (implicit AND between terms)
  • Phrase queries: "exact phrase" (preserves word order)
  • Boolean operators: term1 AND term2, term1 OR term2, NOT term
  • Trailing wildcard: contract* matches contracts, contracting, contracted
  • Leading wildcard: *vertrag efficiently finds Arbeitsvertrag, Kaufvertrag (optimised via reverse token field)
  • Infix wildcard: *vertrag* finds both Vertragsbedingungen and Arbeitsvertrag
  • Single char wildcard: te?t matches test, text
  • Fuzzy search: term~2 finds terms within Levenshtein edit distance 2 (default: 2)
  • Proximity search: "term1 term2"~5 finds terms within 5 words of each other
  • Field-specific search: title:hello content:world
  • Grouping: (contract OR agreement) AND signed
  • Range queries: modified_date:[1609459200000 TO 1640995200000] (timestamps in milliseconds)

Automatic Phrase Proximity Expansion:

Multi-word phrase queries are automatically expanded: "Domain Design" becomes ("Domain Design")^2.0 OR ("Domain Design"~3). Exact matches rank highest (2.0x boost), while near-matches (within 3 words) also surface at lower scores. Single-word phrases and user-specified slop are not expanded.

See PIPELINE.md for detailed examples and configuration.

Adaptive Prefix Query Scoring:

Prefix queries with >= 4 characters (vertrag*, design*) use real BM25 scoring, ranking shorter/more frequent terms higher than long compounds. Shorter prefixes (ver*) use constant scoring for performance. This balances ranking quality with speed automatically.

See PIPELINE.md for scoring examples and technical details.

German Compound Word Search:

Use wildcards for German compounds: *vertrag finds Arbeitsvertrag, vertrag* finds Vertragsbedingungen. Leading wildcards are optimized via the content_reversed field.

Automatic Lemmatization:

OpenNLP lemmatization handles morphological variants automatically. German: "Haus" finds "Häuser", "gehen" finds "ging". English: "run" finds "ran", "pay" finds "paid". Exact matches always rank highest.

Dual-Language Support:

All documents indexed with both German and English lemma fields, enabling mixed-language matching. German docs with English technical terms ("Recommendation Engines") match singular queries ("Recommendation Engine") via the English lemmatizer, and vice versa.

German Umlaut Transliteration:

The content_translit_de field maps ASCII digraphs to umlauts: "Mueller" matches "Müller", "Kaese" matches "Käse".

See PIPELINE.md for complete analyzer chains, concrete token examples, and query pipeline details.

Returns:

  • Paginated document results, each containing a passages array with highlighted text and quality metadata
  • Document-level relevance scores
  • facets: Facet values and counts from the result set (uses DrillSideways when facet filters are active, showing alternative values)
  • activeFilters: Mirrors the input filters with a matchCount for each filter (count from facets, or -1 for range/non-faceted filters)
  • Search execution time in milliseconds (searchTimeMs)

Filter examples:

// Browse all English PDFs
{ "query": null, "filters": [
    { "field": "language", "value": "en" },
    { "field": "file_extension", "value": "pdf" }
]}

// Date range filter
{ "query": "contract*", "filters": [
    { "field": "modified_date", "operator": "range", "from": "2024-01-01", "to": "2025-12-31" }
]}

// Multiple values with exclusion
{ "query": "report", "filters": [
    { "field": "file_extension", "operator": "in", "values": ["pdf", "docx"] },
    { "field": "language", "operator": "not", "value": "unknown" }
]}

Semantic Search Tools (group: semantic)

Semantic search tools require VECTOR_MODEL to be configured (e.g., VECTOR_MODEL=e5-base).

semanticSearch

Pure KNN embedding-based semantic search. Finds semantically related documents even without exact keyword matches. Results are ordered by cosine similarity. Requires VECTOR_MODEL to be configured.

Parameters:

  • query (required): Natural language query — the server computes an embedding and finds the nearest document chunks.
  • filters (optional): Array of structured filters (same format as simpleSearch/extendedSearch)
  • page (optional): Page number, 0-based (default: 0)
  • pageSize (optional): Results per page (default: 10, max: 100)
  • similarityThreshold (optional): Minimum cosine similarity score to include a result (0.0–1.0, default: 0.70). Lower = more results (broader match); higher = fewer results (closer match).

Use profileSemanticSearch to tune similarityThreshold for your corpus.

profileSemanticSearch

Debug tool for semantic search. Shows embedding time, cosine scores, matched chunks, and how many candidates passed the similarity threshold. Use this to tune similarityThreshold for your data.

Parameters:

  • query (required): Natural language query to profile
  • filters (optional): Array of structured filters
  • similarityThreshold (optional): Threshold to test (0.0–1.0, default: 0.70)

Debug Tools (group: debug)

profileQuery

Analyze and debug simpleSearch / extendedSearch queries. Provides detailed insights into how Lucene processes your query, which terms contribute to scoring, how filters affect results, and where optimization opportunities exist.

Parameters:

  • query (optional): The search query (same as simpleSearch/extendedSearch)
  • filters (optional): Array of structured filters (same as search tools)
  • page (optional): Page number, 0-based (default: 0)
  • pageSize (optional): Results per page (default: 10, max: 100)
  • sortBy (optional): Sort field (same as search tools)
  • sortOrder (optional): Sort order (same as search tools)
  • queryMode (optional): SIMPLE (default) or EXTENDED — selects the query parser mode to match the search tool you are profiling
  • analyzeFilterImpact (optional): If true, analyzes how each filter reduces result count. WARNING: Expensive operation requiring multiple queries. Default: false
  • analyzeDocumentScoring (optional): If true, provides detailed scoring explanations for top documents using Lucene's Explanation API. WARNING: Expensive operation. Default: false
  • analyzeFacetCost (optional): If true, measures faceting computation overhead. WARNING: Expensive operation. Default: false
  • maxDocExplanations (optional): Maximum number of documents to explain when analyzeDocumentScoring=true (default: 5, max: 10)

Analysis Levels:

Level 1: Fast Analysis (Always Included)

  • Query structure and component breakdown
  • Query type identification (BooleanQuery, TermQuery, WildcardQuery, etc.)
  • Estimated cost per query component
  • Term statistics (document frequency, IDF, rarity classification)
  • Search metrics (total hits, filter reduction percentage)

Level 2: Filter Impact Analysis (Opt-in, Expensive)

  • Shows how each filter affects result count
  • Calculates selectivity (low/medium/high/very high)
  • Measures execution time per filter
  • Helps identify redundant or ineffective filters

Level 3: Document Scoring Explanations (Opt-in, Expensive)

  • Detailed score breakdown for top-ranked documents
  • Shows which terms contribute most to each document's score
  • Provides human-readable scoring summaries
  • Uses Lucene's Explanation API but parsed into LLM-friendly format

Level 4: Facet Cost Analysis (Opt-in, Expensive)

  • Measures faceting computation overhead
  • Shows cost per facet dimension
  • Helps decide if faceting should be disabled for performance

Returns:

A structured analysis object containing:

{
  success: boolean,
  queryAnalysis: {
    originalQuery: string,
    parsedQueryType: string,
    components: [{
      type: string,              // "TermQuery", "WildcardQuery", etc.
      field: string,
      value: string,
      occur: string,             // "MUST", "SHOULD", "FILTER", "MUST_NOT"
      estimatedCost: number,
      costDescription: string    // "~450 documents (moderate)"
    }],
    rewrites: [{                 // Query optimizations performed by Lucene
      original: string,
      rewritten: string,
      reason: string
    }],
    warnings: string[]
  },
  searchMetrics: {
    totalIndexedDocuments: number,
    documentsMatchingQuery: number,
    documentsAfterFilters: number,
    filterReductionPercent: number,
    termStatistics: {
      [term: string]: {
        term: string,
        documentFrequency: number,
        totalTermFrequency: number,
        idf: number,
        rarity: string           // "very common", "common", "uncommon", "rare"
      }
    }
  },
  filterImpact?: {               // Only if analyzeFilterImpact=true
    baselineHits: number,
    finalHits: number,
    filterImpacts: [{
      filter: {...},
      hitsBeforeFilter: number,
      hitsAfterFilter: number,
      documentsRemoved: number,
      reductionPercent: number,
      selectivity: string,       // "low", "medium", "high", "very high"
      executionTimeMs: number
    }],
    totalExecutionTimeMs: number
  },
  documentExplanations?: [{      // Only if analyzeDocumentScoring=true
    filePath: string,
    rank: number,
    score: number,
    scoringBreakdown: {
      totalScore: number,
      components: [{
        term: string,
        field: string,
        contribution: number,
        contributionPercent: number,
        details: {
          idf: number,
          tf: number,
          termFrequency: number,
          documentLength: number,
          averageDocumentLength: number,
          explanation: string
        }
      }],
      summary: string            // "Score dominated by term 'contract' (60.8%)"
    },
    matchedTerms: string[]
  }],
  facetCost?: {                  // Only if analyzeFacetCost=true
    facetingOverheadMs: number,
    facetingOverheadPercent: number,
    dimensions: {
      [dimension: string]: {
        dimension: string,
        uniqueValues: number,
        totalCount: number,
        computationTimeMs: number
      }
    }
  },
  recommendations: string[]      // Actionable optimization suggestions
}

Example: Basic Query Analysis

Ask Claude: "Profile my search for 'contract AND signed' to understand its performance"

This performs fast analysis showing:

  • Query structure (Boolean AND query with two terms)
  • Term statistics (how common "contract" and "signed" are)
  • Cost estimates (how many documents will be examined)
  • Optimization recommendations

Example: Deep Analysis with Scoring

{
  "query": "(contract OR agreement) AND signed",
  "filters": [
    { "field": "language", "value": "en" },
    { "field": "modified_date", "operator": "range", "from": "2024-01-01" }
  ],
  "analyzeDocumentScoring": true,
  "maxDocExplanations": 3
}

This provides detailed scoring explanations for the top 3 documents, showing:

  • Which terms matched in each document
  • How much each term contributed to the final score
  • Why document A ranked higher than document B

Example: Filter Optimization

{
  "query": "*",
  "filters": [
    { "field": "file_extension", "value": "pdf" },
    { "field": "language", "value": "en" },
    { "field": "file_type", "value": "application/pdf" }
  ],
  "analyzeFilterImpact": true
}

This analyzes filter effectiveness, potentially revealing:

  • file_extension=pdf reduces results by 75% (high selectivity)
  • file_type=application/pdf reduces results by 0% (redundant with file_extension)
  • Recommendation: Remove redundant file_type filter

Example: Understanding Automatic Phrase Expansion

When you search for an exact phrase like "Domain Design", the query is automatically expanded to improve recall while maintaining precision:

{
  "query": "\"Domain Design\"",
  "analyzeDocumentScoring": true,
  "maxDocExplanations": 3
}

The profiler reveals how the query was expanded:

Query Analysis:

  • Original Query: "Domain Design"
  • Parsed Type: BooleanQuery
  • Rewrite: Automatic phrase proximity expansion (exact match boosted + proximity variants)
  • Query Components:
    • PhraseQuery (boost=4.0) - "domain design" (exact match, highest boost)
    • PhraseQuery (boost=2.0) - "domain design"~3 (proximity match, slop=3)
    • Additional stemmed variants with lower boosts

Document Scoring:

  • Exact match ("Domain Design"): Score 0.81 - matches both clauses, exact clause dominates
  • Proximity match ("Domain-driven Design"): Score 0.15 - matches only proximity clause
  • Proximity match ("Domain Effective Design"): Score 0.15 - matches only proximity clause

This shows:

  1. Exact matches rank highest due to the 4.0x accumulated boost (2.0 from stemming x 2.0 from phrase expansion)
  2. Proximity matches still found with slop=3 (allowing up to 3 words between terms)
  3. Clear score separation between exact and proximity matches ensures precision

Performance Notes:

  • Basic analysis (default): Very fast, negligible overhead (~5-10ms)
  • Filter impact analysis: Requires N+1 queries where N is the number of filters. Can take seconds for complex filter sets.
  • Document scoring analysis: Requires Lucene to compute full Explanation objects. Cost grows with maxDocExplanations.
  • Facet cost analysis: Requires facet computation. Cost depends on number of unique facet values.

Best Practices:

  1. Start with basic analysis (no optional flags) to get quick insights
  2. Enable expensive analysis only when debugging specific performance issues
  3. Use analyzeDocumentScoring to understand why certain documents rank highly
  4. Use analyzeFilterImpact to optimize filter order and remove redundant filters
  5. Pay attention to the recommendations array for actionable optimization tips

Crawler Tools (group: crawler)

startCrawl

Start crawling configured directories to index documents.

Parameters:

  • fullReindex (optional): If true, clears the index before crawling (default: false). When false and reconciliation-enabled is true, an incremental crawl is performed instead.

Features:

  • Automatically extracts content from PDFs, Office documents, and OpenOffice files
  • Detects document language
  • Extracts metadata (author, title, creation date, etc.)
  • Multi-threaded processing for fast indexing
  • Progress notifications during crawling
  • Incremental mode (default): Only new or modified files are indexed; deleted files are removed from the index automatically. Falls back to a full crawl if reconciliation encounters an error.

getCrawlerStats

Get real-time statistics about the crawler progress.

Returns:

  • filesFound: Total files discovered
  • filesProcessed: Files processed so far
  • filesIndexed: Files successfully indexed
  • filesFailed: Files that failed to process
  • bytesProcessed: Total bytes processed
  • filesPerSecond: Processing throughput
  • megabytesPerSecond: Data throughput
  • elapsedTimeMs: Time elapsed since crawl started
  • perDirectoryStats: Statistics breakdown per directory
  • orphansDeleted: Number of index entries removed because the file no longer exists on disk (incremental mode)
  • filesSkippedUnchanged: Number of files skipped because they were not modified since the last crawl (incremental mode)
  • reconciliationTimeMs: Time spent comparing the index against the filesystem (incremental mode)
  • crawlMode: Either "full" or "incremental"
  • currentlyProcessing: Array of files currently being processed (extracted/indexed). Each entry contains:
    • filePath: Full path to the file being processed
    • processingDurationMs: How long the file has been processing (in milliseconds)
  • lastCrawlCompletionTimeMs: Unix timestamp (ms) of the last successful crawl completion (null if no previous crawl)
  • lastCrawlDocumentCount: Number of documents in the index after the last successful crawl (null if no previous crawl)
  • lastCrawlMode: Mode of the last crawl - "full" or "incremental" (null if no previous crawl)

getCrawlerStatus

Get the current state of the crawler.

Returns:

  • state: One of IDLE, CRAWLING, PAUSED, or WATCHING

pauseCrawler

Pause an ongoing crawl operation. The crawler can be resumed later with resumeCrawler.

resumeCrawler

Resume a paused crawl operation.

listCrawlableDirectories

List all configured crawlable directories.

Returns:

  • success: Boolean indicating operation success
  • directories: List of absolute directory paths currently configured
  • totalDirectories: Count of configured directories
  • configPath: Path to the configuration file (~/.mcplucene/config.yaml)
  • environmentOverride: Boolean indicating if LUCENE_CRAWLER_DIRECTORIES env var is set

Example response:

{
  "success": true,
  "directories": [
    "/Users/yourname/Documents",
    "/Users/yourname/Downloads"
  ],
  "totalDirectories": 2,
  "configPath": "/Users/yourname/.mcplucene/config.yaml",
  "environmentOverride": false
}

addCrawlableDirectory

Add a directory to the crawler configuration.

Parameters:

  • path (required): Absolute path to the directory to crawl
  • crawlNow (optional): If true, immediately starts crawling the new directory (default: false)

Returns:

  • success: Boolean indicating operation success
  • message: Confirmation message
  • totalDirectories: Updated count of configured directories
  • directories: Updated list of all directories
  • crawlStarted (optional): Present if crawlNow=true, indicates crawl was triggered

Validation:

  • Directory must exist and be accessible
  • Path must be a directory (not a file)
  • Duplicate directories are prevented
  • Fails if LUCENE_CRAWLER_DIRECTORIES environment variable is set

Example:

Ask Claude: "Add /Users/yourname/Documents as a crawlable directory"
Ask Claude: "Add /path/to/research and crawl it now"

Configuration Persistence: The directory is immediately saved to ~/.mcplucene/config.yaml and will be automatically crawled on future server restarts.

removeCrawlableDirectory

Remove a directory from the crawler configuration.

Parameters:

  • path (required): Absolute path to the directory to remove

Returns:

  • success: Boolean indicating operation success
  • message: Confirmation message
  • totalDirectories: Updated count of configured directories
  • directories: Updated list of remaining directories

Important Notes:

  • This does NOT remove already-indexed documents from the removed directory
  • To remove indexed documents, use startCrawl(fullReindex=true) after removing directories
  • Fails if LUCENE_CRAWLER_DIRECTORIES environment variable is set
  • The directory must exist in the current configuration

Example:

Ask Claude: "Stop crawling /Users/yourname/Downloads"
Ask Claude: "Remove /path/to/old/archive from the crawler"

Index Info Tools (group: info)

getIndexStats

Get statistics about the Lucene index, including lemmatizer cache performance metrics, query runtime percentiles (p50-p99), and per-field facet computation timing.

Returns:

  • documentCount: Total number of documents in the index
  • indexPath: Path to the index directory
  • schemaVersion: Current index schema version
  • softwareVersion: Server software version
  • buildTimestamp: Server build timestamp
  • dateFieldHints: Min/max date ranges for date fields (created_date, modified_date, indexed_date) in ISO-8601 format — useful for building date range filters
  • sortableFields: Map of dynamically registered sortable dbmeta_* fields from JDBC metadata enrichment to their sort type ("numeric" or "keyword"). Null when no JDBC enrichment has registered sortable fields. Use this to discover which dbmeta_* fields can be passed as sortBy. Native fields (file_size, created_date, modified_date) are always sortable and not listed here.
  • lemmatizerCacheMetrics: Performance metrics for the OpenNLP lemmatizer caches (one per language: German and English)
    • language: Language code (de or en)
    • hitRate: Cache hit rate as a percentage (e.g., "85.3%")
    • totalHits: Number of times a token was found in the cache
    • totalMisses: Number of times a token required lemmatization
    • cacheSize: Current number of entries in the cache
    • evictions: Number of cache entries evicted due to size limits
  • queryRuntimeMetrics: Aggregate search query performance statistics (null before any searches are executed)
    • totalQueries: Total number of search queries executed since server start
    • averageDurationMs: Average query duration in milliseconds (e.g., "12.5")
    • minDurationMs: Fastest query duration in milliseconds
    • maxDurationMs: Slowest query duration in milliseconds
    • averageHitCount: Average number of matching documents per query (e.g., "42.3")
    • p50Ms / p75Ms / p90Ms / p95Ms / p99Ms: Query duration percentiles in milliseconds (computed from last 1000 queries)
    • averageFacetDurationMs: Average facet computation time per query in milliseconds (e.g., "0.125")
    • perFieldAverageFacetDurationMs: Per-field average facet computation time in milliseconds (e.g., {"language": "0.031", "file_extension": "0.028", ...})

Lemmatizer Cache Performance:

The server uses single-token caching for OpenNLP lemmatization to reduce CPU usage during indexing and querying. Each language analyzer (German and English) maintains a shared LRU cache with up to 1,500,000 entries, shared across all Lucene indexing threads. The cache uses case-insensitive keys for common words (e.g., "Vertrag" and "vertrag" share the same cache entry) while keeping proper nouns case-sensitive (e.g., "Berlin" vs "berlin").

Key Metrics:

  • Hit Rate: Higher is better. 85-95% is typical after indexing a few thousand documents. Higher hit rates mean less CPU usage.
  • Cache Size: Current number of cached (token, POS tag) to lemma mappings. Grows up to 1,500,000 entries per language.
  • Evictions: How many entries have been removed to make room for new ones. Some evictions are normal with large document sets.

Performance Impact: Without caching, lemmatization can consume 70-80% of CPU during indexing. With caching, CPU usage typically drops to 20-30%, resulting in 2-3x faster indexing throughput for document sets with repetitive vocabulary.

listIndexedFields

List all field names present in the Lucene index.

Returns:

  • fields: Array of field names available for searching and filtering

Example response:

{
  "success": true,
  "fields": [
    "file_name",
    "file_path",
    "title",
    "author",
    "content",
    "language",
    "file_extension",
    "file_type",
    "created_date",
    "modified_date"
  ]
}

getDocumentDetails

Retrieve all stored fields and full content of a document from the Lucene index by its file path. This tool retrieves document details directly from the index without requiring filesystem access - useful for examining indexed content even if the original file has been moved or deleted.

Parameters:

  • filePath (required): Absolute path to the file (must match exactly the file_path stored in the index)

Returns:

  • success: Boolean indicating operation success
  • document: Object containing all stored fields:
    • file_path: Full path to the file
    • file_name: Name of the file
    • file_extension: File extension (e.g., pdf, docx)
    • file_type: MIME type
    • file_size: File size in bytes
    • title: Document title
    • author: Author name
    • creator: Creator application
    • subject: Document subject
    • keywords: Document keywords/tags
    • language: Detected language code
    • created_date: Creation timestamp
    • modified_date: Modification timestamp
    • indexed_date: Indexing timestamp
    • content_hash: SHA-256 hash of content
    • content: Full extracted text content (limited to 500KB)
    • contentTruncated: Boolean indicating if content was truncated
    • originalContentLength: Original content length (only present if truncated)

Content Size Limit: The content field is limited to 500,000 characters (500KB) to ensure the response stays safely below the 1MB MCP response limit. Check the contentTruncated field to determine if the full content was returned.

Example:

Ask Claude: "Show me the indexed details of /Users/yourname/Documents/report.pdf"
Ask Claude: "What content was extracted from /path/to/contract.docx?"

Example response:

{
  "success": true,
  "document": {
    "file_path": "/Users/yourname/Documents/report.pdf",
    "file_name": "report.pdf",
    "file_extension": "pdf",
    "file_type": "application/pdf",
    "file_size": "125432",
    "title": "Annual Report 2024",
    "author": "John Doe",
    "language": "en",
    "indexed_date": "1706540400000",
    "content_hash": "a1b2c3d4...",
    "content": "This is the full extracted text content of the document...",
    "contentTruncated": false
  }
}

Observability Tools (group: observability)

suggestTerms

Suggest index terms matching a prefix. Useful for discovering vocabulary, finding German compound words, exploring author names, or auto-completing field values.

Parameters:

  • field (required): Field name to suggest terms from (e.g. content, author, file_extension)
  • prefix (required): Prefix to match terms against (e.g. ver to find vertrag, version)
  • limit (optional): Maximum number of terms to return (default: 20, max: 100)

Notes:

  • For analyzed fields (content, title, etc.), the prefix is automatically lowercased to match indexed tokens
  • For StringFields (file_extension, language), the prefix is used as-is (exact match)
  • Numeric/date fields (file_size, modified_date, etc.) are not supported — use getIndexStats for date ranges
  • Returns terms sorted by document frequency (most common first)
  • Returns empty results for nonexistent fields (not an error)

Example — discover German compound words:

{
  "field": "content",
  "prefix": "vertrag",
  "limit": 10
}

Example response:

{
  "success": true,
  "field": "content",
  "prefix": "vertrag",
  "terms": [
    {"term": "vertrag", "docFreq": 45},
    {"term": "vertrags", "docFreq": 23},
    {"term": "vertragsklausel", "docFreq": 8},
    {"term": "vertragsbedingungen", "docFreq": 5}
  ],
  "totalTermsMatched": 4
}

getTopTerms

Get the most frequent terms in a field. Useful for understanding index vocabulary, discovering common values (languages, file types, authors), and identifying dominant terms.

Parameters:

  • field (required): Field name to get top terms from (e.g. content, author, file_extension)
  • limit (optional): Maximum number of terms to return (default: 20, max: 100)

Notes:

  • Returns terms sorted by document frequency (most common first)
  • For large content fields (>100K unique terms), a warning is included suggesting suggestTerms instead
  • Numeric/date fields are not supported — use getIndexStats for date ranges
  • Returns empty results for nonexistent fields (not an error)

Example — explore file types in index:

{
  "field": "file_extension",
  "limit": 10
}

Example response:

{
  "success": true,
  "field": "file_extension",
  "terms": [
    {"term": "pdf", "docFreq": 234},
    {"term": "docx", "docFreq": 156},
    {"term": "txt", "docFreq": 89},
    {"term": "md", "docFreq": 45}
  ],
  "uniqueTermCount": 12
}

Example — explore content vocabulary:

{
  "field": "content",
  "limit": 20
}

Admin Tools (group: admin)

indexAdmin

An MCP App that provides a visual user interface for index maintenance tasks directly inside your MCP client (e.g. Claude Desktop). When invoked, the app is rendered inline in the conversation and offers one-click access to administrative operations without requiring manual tool calls.

Index Administration App

Available actions:

  • Unlock Index -- Removes a stale write.lock file after an unclean shutdown (equivalent to calling unlockIndex with confirm=true)
  • Optimize Index -- Merges index segments for improved search performance (equivalent to calling optimizeIndex)
  • Purge Index -- Deletes all documents from the index (equivalent to calling purgeIndex with confirm=true)

Each action shows inline status feedback (success, error, or progress details) directly in the app UI.

Example:

Ask Claude: "Can you invoke the indexAdmin tool please?"

optimizeIndex

Optimize the Lucene index by merging segments. This is a long-running operation that runs in the background.

Parameters:

  • maxSegments (optional): Target number of segments after optimization (default: 1 for maximum optimization)

Returns:

  • success: Boolean indicating the operation was started
  • operationId: UUID to track the operation
  • targetSegments: The target segment count
  • currentSegments: The current segment count before optimization
  • message: Status message

Behavior:

  • Returns immediately after starting the background operation
  • Use getIndexAdminStatus to poll for progress
  • Cannot run while the crawler is actively crawling
  • Only one admin operation can run at a time

Example:

Ask Claude: "Optimize the search index"
Ask Claude: "What's the status of the optimization?"

Performance Notes:

  • Optimization improves search performance by reducing the number of segments
  • Temporarily increases disk usage during the merge
  • For large indices, this can take several minutes to hours

purgeIndex

Delete all documents from the Lucene index. This is a destructive, long-running operation that runs in the background.

Parameters:

  • confirm (required): Must be set to true to proceed. This is a safety measure.
  • fullPurge (optional): If true, also deletes index files and reinitializes (default: false)

Returns:

  • success: Boolean indicating the operation was started
  • operationId: UUID to track the operation
  • documentsDeleted: Number of documents that will be deleted
  • fullPurge: Whether a full purge was requested
  • message: Status message

Behavior:

  • Returns immediately after starting the background operation
  • Use getIndexAdminStatus to poll for progress
  • Only one admin operation can run at a time

Purge Modes:

  • Standard purge (fullPurge=false): Deletes all documents but keeps index files. Disk space is reclaimed gradually during future merges.
  • Full purge (fullPurge=true): Deletes all documents AND index files, then reinitializes an empty index. Disk space is reclaimed immediately.

Example:

Ask Claude: "Delete all documents from the index - I confirm this"
Ask Claude: "Purge the index completely and reclaim disk space - I confirm this"

Warning: This operation cannot be undone. All indexed documents will be permanently deleted. You will need to re-crawl directories to repopulate the index.

unlockIndex

Remove the write.lock file from the Lucene index directory. This is a dangerous recovery operation - only use if you are certain no other process is using the index.

Parameters:

  • confirm (required): Must be set to true to proceed. This is a safety measure.

Returns:

  • success: Boolean indicating operation success
  • message: Confirmation message
  • lockFileExisted: Boolean indicating if a lock file was present
  • lockFilePath: Path to the lock file

When to use: Use this tool when the server fails to start with a LockObtainFailedException after an unclean shutdown. See Troubleshooting for details.

Example:

Ask Claude: "Unlock the Lucene index - I confirm this is safe"

Warning: Unlocking an index that is actively being written to by another process can cause data corruption. Only use this when you are certain the lock is stale.

getIndexAdminStatus

Get the status of long-running index administration operations (optimize, purge).

Parameters: None

Returns:

  • success: Boolean indicating the status was retrieved
  • state: Current state: IDLE, OPTIMIZING, PURGING, COMPLETED, or FAILED
  • operationId: UUID of the current/last operation
  • progressPercent: Progress percentage (0-100)
  • progressMessage: Human-readable progress message
  • elapsedTimeMs: Time elapsed since operation started (in milliseconds)
  • lastOperationResult: Result message from the last completed operation

Example response (during optimization):

{
  "success": true,
  "state": "OPTIMIZING",
  "operationId": "a1b2c3d4-...",
  "progressPercent": 45,
  "progressMessage": "Merging segments...",
  "elapsedTimeMs": 12500,
  "lastOperationResult": null
}

Example response (idle after completion):

{
  "success": true,
  "state": "IDLE",
  "operationId": null,
  "progressPercent": null,
  "progressMessage": "No admin operation running",
  "elapsedTimeMs": null,
  "lastOperationResult": "Optimization completed successfully. Merged to 1 segment(s)."
}

Example:

Ask Claude: "What's the status of the index optimization?"
Ask Claude: "Is the purge operation complete?"

Index Field Schema

When documents are indexed by the crawler, the following fields are automatically extracted and stored:

Content Fields

  • content: Full text content of the document (analyzed, searchable)
  • content_reversed: Reversed tokens of the content (analyzed with ReverseUnicodeNormalizingAnalyzer, not stored). Used internally for efficient leading wildcard queries -- not directly searchable by users.
  • content_lemma_de: Lemmatized tokens using German OpenNLP lemmatizer (analyzed with OpenNLPLemmatizingAnalyzer, not stored). ALWAYS present for ALL documents regardless of detected language to enable mixed-language matching. Used internally for lemmatization-based search -- not directly searchable by users.
  • content_lemma_en: Lemmatized tokens using English OpenNLP lemmatizer (analyzed with OpenNLPLemmatizingAnalyzer, not stored). ALWAYS present for ALL documents regardless of detected language to enable mixed-language matching. Used internally for lemmatization-based search -- not directly searchable by users.
  • content_translit_de: German transliteration shadow field that maps umlaut digraphs (ae→ä, oe→ö, ue→ü) before standard Unicode normalization (analyzed with GermanTransliteratingAnalyzer, not stored). ALWAYS present for ALL documents. Enables ASCII digraph queries like "Mueller" to match umlaut-containing documents like "Müller". Used internally -- not directly searchable by users.
  • passages: Array of highlighted passages returned in search results (see Search Response Format below)

File Information

  • file_path: Full path to the file (unique ID)
  • file_name: Name of the file
  • file_extension: File extension (e.g., pdf, docx)
  • file_type: MIME type (e.g., application/pdf)
  • file_size: File size in bytes

Document Metadata

  • title: Document title (extracted from metadata)
  • author: Author name
  • creator: Creator/application that created the document
  • subject: Document subject
  • keywords: Document keywords/tags

Language & Dates

  • language: Auto-detected language code (e.g., en, de, fr)
  • created_date: File creation timestamp
  • modified_date: File modification timestamp
  • indexed_date: When the document was indexed

Technical

  • content_hash: SHA-256 hash for change detection

Search Response Format

Search results are optimized for MCP responses (< 1 MB) and include:

{
  "success": true,
  "documents": [
    {
      "score": 0.85,
      "file_name": "example.pdf",
      "file_path": "/path/to/example.pdf",
      "title": "Example Document",
      "author": "John Doe",
      "language": "en",
      "passages": [
        {
          "text": "...relevant <em>search term</em> highlighted in context...",
          "score": 1.0,
          "matchedTerms": ["search term"],
          "termCoverage": 1.0,
          "position": 0.12,
          "source": "keyword"
        },
        {
          "text": "...another occurrence of <em>search</em> in a later section...",
          "score": 0.75,
          "matchedTerms": ["search"],
          "termCoverage": 0.5,
          "position": 0.67,
          "source": "keyword"
        }
      ]
    }
  ],
  "totalHits": 42,
  "page": 0,
  "pageSize": 10,
  "totalPages": 5,
  "hasNextPage": true,
  "hasPreviousPage": false,
  "searchTimeMs": 12,
  "facets": {
    "language": [
      { "value": "en", "count": 25 },
      { "value": "de", "count": 12 },
      { "value": "fr", "count": 5 }
    ],
    "file_extension": [
      { "value": "pdf", "count": 30 },
      { "value": "docx", "count": 8 },
      { "value": "xlsx", "count": 4 }
    ],
    "file_type": [
      { "value": "application/pdf", "count": 30 },
      { "value": "application/vnd.openxmlformats-officedocument.wordprocessingml.document", "count": 8 }
    ],
    "author": [
      { "value": "John Doe", "count": 15 },
      { "value": "Jane Smith", "count": 10 }
    ]
  },
  "_search": {
    "query": "contract",
    "filters": [],
    "page": 0,
    "pageSize": 10
  },
  "_actions": [
    {
      "type": "nextPage",
      "tool": "simpleSearch",
      "parameters": { "query": "contract", "filters": [], "page": 1, "pageSize": 10 }
    },
    {
      "type": "drillDown",
      "tool": "simpleSearch",
      "parameters": { "query": "contract", "filters": [{ "field": "language", "operator": "eq", "value": "en" }], "page": 0, "pageSize": 10 },
      "hits": 25
    }
  ]
}

Each document in documents[] also carries its own _actions block:

{
  "score": 0.85,
  "file_path": "/path/to/example.pdf",
  "_actions": [
    {
      "type": "fetchContent",
      "tool": "getDocumentDetails",
      "parameters": { "filePath": "/path/to/example.pdf" }
    }
  ]
}

HATEOAS-Style _actions (LLM Chaining):

Every search response includes two pre-computed action blocks that enable an LLM to chain tool calls without reasoning about parameter mapping:

  • _search -- Captures the exact search state (query, filters, page, pageSize) used to produce this response. Useful for introspection and for constructing follow-up queries.

  • Response-level _actions -- Contains ready-to-use tool calls for navigating the result set:

    Action typeWhen presentDescription
    prevPagepage > 0Go to the previous result page. Pass parameters directly to the named tool.
    nextPagehasNextPage = trueGo to the next result page. Pass parameters directly to the named tool.
    drillDownfacets availableNarrow results by adding a facet value as a filter. The hits field shows expected result count. Limited to the top 2 values per facet dimension; only values not already active as filters are included.

  • Document-level _actions -- Each document in documents[] includes:

    Action typeDescription
    fetchContentFetch the full document text and metadata using getDocumentDetails. The filePath is pre-filled.

To use an action, call the tool named in the action with the parameters map passed as-is — no transformation required.

Key Features:

  • Search Performance Metrics: Every search response includes searchTimeMs showing the exact execution time in milliseconds, enabling performance monitoring and optimization.

  • Passages with Highlighting: The full content field is NOT included in search results to keep response sizes manageable. Instead, each document contains a passages array with up to max-passages (default: 3) individually highlighted excerpts. Each passage is a separate sentence-level excerpt (not a single joined string), ordered by relevance (best first). Long passages are truncated to max-passage-char-length (default: 200) centred around the highlighted terms, trimming irrelevant leading/trailing text. Each passage includes:

    • text -- The highlighted excerpt with matched terms wrapped in <em> tags.
    • score -- Normalised relevance score (0.0-1.0), derived from Lucene's BM25 passage scoring. The best passage scores 1.0; other passages are scored relative to the best.
    • matchedTerms -- The distinct query terms that appear in this passage (extracted from the <em> tags). Useful for understanding which parts of a multi-term query a passage satisfies.
    • termCoverage -- The fraction of all query terms present in this passage (0.0-1.0). A value of 1.0 means every query term matched. LLMs can use this to prefer passages that address the full query.
    • position -- Location within the source document (0.0 = start, 1.0 = end), derived from the passage's character offset. Useful for citations or for understanding document structure.
    • source -- Indicates how this passage was produced: "keyword" means the BM25 highlighter found term matches in the indexed document text; "semantic" means the best-matching vector chunk was used as the snippet (relevant because the document was retrieved via vector similarity, even if the exact query words do not appear in the text).

  • Lucene Faceting: The facets object uses Lucene's SortedSetDocValues for efficient faceted search. It shows actual facet values and document counts from the search results, not just available fields. Only facet dimensions that have values in the result set are returned.

  • Facet Dimensions: The following fields are indexed as facets:

    • language - Detected document language (ISO 639-1 code)
    • file_extension - File extension (pdf, docx, etc.)
    • file_type - MIME type
    • author - Document author (multi-valued)

Faceted Search Examples

Use facets to build drill-down queries and refine search results:

# Filter by file type using facet values
filters: [{ field: "file_extension", value: "pdf" }]

# Filter by language using facet values
filters: [{ field: "language", value: "de" }]

# Filter by author using facet values
filters: [{ field: "author", value: "John Doe" }]

# Combine search query with facet filter
query: "contract agreement"
filters: [{ field: "file_extension", value: "pdf" }]

Facet-Driven Workflow:

  1. Perform initial search with broad query
  2. Review facets in response to see available refinement options
  3. Apply filters using facet values to narrow results
  4. Iterate to drill down into specific subsets

Document Crawler Features

Automatic Crawling

The crawler starts automatically on server startup (if crawl-on-startup: true) and:

  1. Discovers files matching include patterns in configured directories
  2. Extracts content using Apache Tika (supports 100+ file formats)
  3. Detects language automatically for each document
  4. Extracts metadata (author, title, dates, etc.)
  5. Indexes documents in batches for optimal performance
  6. Monitors directories for changes (create, modify, delete)

Incremental Indexing (Reconciliation)

By default (reconciliation-enabled: true), every crawl that is not a full reindex performs an incremental pass first. This makes repeated crawls significantly faster because unchanged files are never re-processed.

How it works:

  1. Index snapshot -- All (file_path, modified_date) pairs are read from the Lucene index.
  2. Filesystem snapshot -- The configured directories are walked and the current (file_path, mtime) pairs are collected (no content extraction at this stage).
  3. Four-way diff is computed:
    • DELETE -- paths in the index that no longer exist on disk (orphans).
    • ADD -- paths on disk that are not yet in the index.
    • UPDATE -- paths where the on-disk mtime is newer than the stored modified_date.
    • SKIP -- paths that are identical; these are never touched.
  4. Orphan deletions are applied first (bulk delete via a single Lucene query).
  5. Only ADD and UPDATE files are crawled, extracted, and indexed.
  6. On successful completion, the crawl state (timestamp, document count, mode) is persisted to ~/.mcplucene/crawl-state.yaml.

Fallback behaviour: If reconciliation fails for any reason (I/O error reading the index, filesystem walk failure, etc.) the system automatically falls back to a full crawl. No data is lost and no manual intervention is required.

Disabling incremental indexing: Set reconciliation-enabled: false in application.yaml to always perform a full crawl. Alternatively, pass fullReindex: true to startCrawl to force a single full crawl without changing the default.

Persisted state file:

~/.mcplucene/crawl-state.yaml

This file records the last successful crawl's completion time, document count, and mode. It is written only after a crawl completes successfully.

Schema Version Management

The server tracks the index schema version to detect when the schema changes between software updates. This eliminates the need for manual reindexing after upgrades.

How it works:

  1. Each release embeds a SCHEMA_VERSION constant that reflects the current index field schema.
  2. The schema version is persisted in Lucene's commit metadata alongside the software version.
  3. On startup, the server compares the stored schema version with the current one.
  4. If they differ (or if a legacy index has no version), a full reindex is triggered automatically.

What triggers a schema version bump:

  • Adding or removing indexed fields
  • Changing field analyzers
  • Modifying field indexing options (stored, term vectors, etc.)

Checking version information: Use getIndexStats to see the current schema version, software version, and build timestamp.

Real-time Monitoring

With directory watching enabled (watch-enabled: true):

  • New files are automatically indexed when added
  • Modified files are re-indexed with updated content
  • Deleted files are removed from the index

Performance Optimization

Multi-threading:

  • Crawls multiple directories in parallel (configurable thread pool)
  • Each directory is processed by a separate thread

Batch Processing:

  • Documents are indexed in batches (default: 100 documents)
  • Reduces I/O overhead and improves indexing speed

NRT (Near Real-Time) Optimization:

  • Normal operation: 100ms refresh interval for fast search updates
  • Bulk indexing (>1000 files): Automatically slows to 5s to reduce overhead
  • Restores to 100ms after bulk operation completes

Progress Notifications:

  • Timer-based: updates every 30 seconds (configurable via progress-notification-interval-ms)
  • Shows throughput (files/sec, MB/sec), progress, and currently processing filenames
  • Non-blocking: Appear in system notification area without interrupting workflow
    • macOS: Notifications appear in Notification Center (top-right corner)
    • Windows: Toast notifications in system tray area
    • Linux: Uses notify-send for desktop notifications

Error Handling

  • Failed files are logged but don't stop the crawl
  • Statistics track successful vs. failed files
  • Large documents are fully indexed (no truncation by default)
  • Corrupted or inaccessible files are skipped gracefully