Labsco
Automattic logo

qa

โ˜… 470

by automattic ยท part of automattic/studio

Compare extracted WXR content against the original source site page by page. Find missing text, headings, images, and links. Fix by patching the WXR or re-extracting individual pages. Produces a health score and structured report. Use when asked to "qa", "check extraction", "compare content", or "verify extraction quality".

๐Ÿงฐ Not standalone. This skill ships with automattic/studio and only works together with that tool โ€” install the tool first, then add this skill.

This is the playbook your agent receives when the skill activates โ€” you don't need to read it to use the skill, but it's here to audit before installing.

QA: Compare โ†’ Fix โ†’ Verify

You are a QA engineer for content migrations. Compare every page in a WXR file against its original source URL โ€” check that text, headings, images, and links made it through extraction intact. When you find gaps, fix them by patching the WXR or re-extracting the page. Produce a structured report with before/after evidence.

Workflow

Phase 1: Initialize

  1. Locate the WXR file
  2. Read it with readWxr() from src/lib/wxr-reader.ts
  3. Count pages/posts with _source_url โ€” these are testable
  4. Count pages/posts without _source_url โ€” these are skipped (warn the user)
  5. Start timer for duration tracking

Phase 2: Compare

For each page/post with a _source_url:

  1. Fetch the origin page via HTTP
  2. Parse both โ€” origin HTML and WXR content โ€” into a content model (text, headings, images, links) using parseContent() from src/lib/content-parser.ts
  3. Diff using diffContent() from src/lib/content-differ.ts
  4. Grade the page:
    • pass (>90% weighted match) โ€” content faithfully extracted
    • warn (70-90%) โ€” minor gaps
    • fail (<70%) โ€” significant content missing
    • error โ€” fetch failed or page unreachable
  5. Document immediately โ€” don't batch. Log each result.

Per-page checks:

DimensionWhat to checkWeight
TextWord-level similarity (Jaccard on word sets)50%
Headingsh1-h6 count, text, order match20%
ImagesCount match, missing images by filename20%
LinksCount match, missing hrefs10%

Depth judgment: Spend more attention on pages that fail โ€” these need investigation. Pass pages just get logged.

Phase 3: Compute Health Score

Content Health Score (0-100):

  Text fidelity (50%):
    All pages pass     โ†’ 100
    1-2 pages warn     โ†’ 80
    1-2 pages fail     โ†’ 50
    3+ pages fail      โ†’ 20

  Heading fidelity (20%):
    0 missing headings โ†’ 100
    Each missing       โ†’ -10 (min 0)

  Image fidelity (20%):
    0 missing images   โ†’ 100
    Each missing       โ†’ -15 (min 0)

  Link fidelity (10%):
    0 missing links    โ†’ 100
    Each missing       โ†’ -10 (min 0)

  score = ฮฃ (dimension_score ร— weight)

Phase 4: Report (Before Fixes)

Show the comparison report to the user:

Per-page results:

Page: /about (https://www.example.com/about)
  Text:     98% โœ“
  Headings: 3/3 โœ“
  Images:   2/3 โš  missing: hero-banner.jpg
  Links:    5/5 โœ“
  Grade:    warn

Summary:

Content QA: 10 pages checked, 2 skipped (no source URL)
  8 pass  1 warn  1 fail  0 error
  Health score: 74/100

  Top issues:
  1. /project-3 [fail] โ€” text similarity 42%, 3 missing images
  2. /about [warn] โ€” 1 missing image (hero-banner.jpg)

Phase 5: Triage

Sort issues by severity, then decide which to fix based on tier:

  • Quick: Fix fail grade only. Mark warn as deferred.
  • Standard: Fix fail + warn. (default)
  • Exhaustive: Fix all, including pages with minor discrepancies.

Mark pages with error grade (fetch failed) as deferred โ€” can't fix what you can't compare.

Phase 6: Fix Loop

For each fixable page, in severity order (fail first, then warn):

6a. Assess

Read the diff details. What's missing?

  • Missing alt text on existing images โ†’ patchable
  • Missing images entirely โ†’ needs re-extraction
  • Missing text sections โ†’ needs re-extraction
  • Minor text differences โ†’ acceptable, skip

6b. Fix

Level 1: Patch the WXR (for minor fixes)

  • Run runQa({ wxrFile, fix: true }) which patches missing alt text and minor gaps directly in the WXR

Level 2: Re-extract (for major gaps)

  • If content is too far off (text similarity <50%), the page needs full re-extraction
  • Flag it for the user: "Page /project-3 needs re-extraction โ€” text similarity is 42%"
  • If the user approves, re-run extraction for just that URL through the adapter

6c. Verify

After fixes, re-run the comparison on fixed pages:

const result = await runQa({ wxrFile, fix: false });

Check: did the fix improve the grade? If a fix made things worse, revert the WXR from the backup.

6d. Self-Regulation

After every 5 fixes, evaluate:

  • Are the remaining issues actually fixable from the WXR?
  • Are we making things better or just churning?
  • If all remaining issues are warn with >80% similarity, stop โ€” that's good enough.

Hard cap: 20 fix attempts. After 20, stop and report.

6e. Escalate to /diagnose

If after fixing, pages still have fail grades that can't be patched โ€” especially if the failures share a pattern (e.g. all blog posts fail, all product pages are empty) โ€” suggest running /diagnose to investigate the root cause. QA finds the symptoms; diagnose finds the cause.

Phase 7: Final Report

After all fixes:

Content QA Complete โ€” 10 pages checked

  Before: 74/100  โ†’  After: 92/100

  Fixed:
    /about โ€” patched missing alt text on hero-banner.jpg (warn โ†’ pass)
    /project-3 โ€” re-extracted (fail โ†’ pass)

  Deferred:
    /project-5 โ€” origin returns 404, cannot compare

  Health score: 74 โ†’ 92 (+18)

Include:

  • Total pages checked
  • Fix count (patched: X, re-extracted: Y)
  • Deferred issues with reasons
  • Health score delta: before โ†’ after

Using the Code

import { runQa } from './src/lib/qa-runner.js';

// Compare only (no fixes)
const result = await runQa({ wxrFile: '<outputDir>/output.wxr' });

// Compare and fix
const fixResult = await runQa({ wxrFile: '<outputDir>/output.wxr', fix: true });

The QaResult contains:

  • pages[] โ€” per-page results with slug, sourceUrl, grade, diff details
  • skipped โ€” count of pages without _source_url
  • summary โ€” { pass, warn, fail, error, fixed }

The QA log is written to qa-log.jsonl alongside the WXR file.

Important Rules

  1. Compare before fixing. Always show the report first. Ask user before applying fixes.
  2. Minimal fixes. Patch what's safe (alt text, minor gaps). Flag major gaps for re-extraction.
  3. Verify after fixing. Re-run comparison on fixed pages. If the fix made things worse, revert.
  4. No WordPress site needed. QA compares the WXR against the origin site directly.
  5. Log everything. Every comparison and fix goes to qa-log.jsonl.
  6. Don't over-fix. Some text differences are acceptable (navigation, footers, cookie banners). Focus on the main content.
  7. Pages without _source_url can't be QA'd. Warn the user if many pages lack source URLs โ€” they need re-extraction with a newer version that records source URLs.
  8. Self-regulate. Stop after 20 fix attempts or when remaining issues are minor.
  9. Log discoveries. If you find a pattern of content loss specific to a platform (e.g. "Squarespace always drops image captions"), add it to DISCOVERIES.md so future extractions can be improved.