Chapter 17: Migrating 1000+ Pages off WordPress

Practical extract recipe for 1,000+ pages:

1. Take a mysqldump snapshot first — your immutable ground truth and your rollback anchor.
2. Run WP-CLI on the server for resilience: wp post list --post_type=any --format=json --fields=ID,post_type,post_status,post_name,post_date,post_parent > index.json. This gives you a complete entity inventory and the redirect-map seed.
3. Pull rendered content + meta via the REST API with a paginating script that writes one JSON file per post (raw/posts/{id}.json). Add retry/backoff; a 1,000-page pull is thousands of requests.
4. Download the actual media files — WXR will not. Crawl /wp-content/uploads/ via the Media REST endpoint (/wp/v2/media) to enumerate every attachment + its source_url and metadata, then download originals to raw/media/. Capture alt_text, caption, title, and the image-size variants WP generated.

The "four parsers" warning from the Data Liberation project is real: a single field can be a URL inside JSON inside an HTML comment inside the WXR XML, requiring an XML parser, an HTML parser, a JSON parser, and a WHATWG URL parser to fully resolve (WordPress/data-liberation). Plan for nested decoding, not regex.

17.3 Stage 2 — Parse: normalize to an Intermediate Representation

Do not transform straight from WordPress shapes to target shapes. Insert an Intermediate Representation (IR) — a flat, boring JSON record per entity — so the extract side and the load side evolve independently.

// raw → IR, one file per entity: ir/post-1234.json
{
  "legacyId": 1234,
  "type": "post",                // post | page | cpt:product | ...
  "status": "publish",
  "slug": "how-we-cut-latency",
  "legacyUrl": "/2019/04/how-we-cut-latency/",
  "title": "How we cut latency",
  "html": "<!-- raw post_content / rendered HTML -->",
  "excerpt": "...",
  "author": "jdoe",
  "publishedAt": "2019-04-03T09:00:00Z",
  "modifiedAt": "2023-11-02T14:21:00Z",
  "parent": 12,
  "terms": { "category": ["engineering"], "post_tag": ["performance","cdn"] },
  "meta": { "acf_hero_subtitle": "...", "_yoast_wpseo_metadesc": "..." },
  "mediaRefs": [ {"legacyId": 5567, "url": "https://old.site/wp-content/uploads/2019/04/graph.png"} ]
}

At this stage you also decode the WordPress-isms: PHP-unserialize meta_value (carefully — guard against the double-serialize corruption in Trac #10619), expand Gutenberg block comments (<!-- wp:paragraph -->), and inventory every shortcode ([gallery], [caption], builder shortcodes) so nothing surprises you in transform. Produce a shortcode-census.csv (shortcode name → count) early; the long tail is where migrations die.

17.4 Stage 3 — Transform: HTML → structured content (the LLM-assisted core)

This is the chapter's centre of gravity. Your target CMS stores structured content — an array of typed blocks (Portable Text, Gutenberg-style blocks, Strapi blocks, MDX, or your own JSON schema) — not a blob of HTML. WordPress hands you a blob of HTML. Bridging that gap has two layers.

Layer A — Deterministic parsing (do this first, for ~90% of content)

A unified/AST pipeline converts well-formed HTML to structured blocks deterministically and for free — no LLM, no per-token cost, fully reproducible. The JavaScript ecosystem standard is unified + rehype: rehype-parse turns HTML into a HAST syntax tree; you then map nodes to your target shape. For Sanity specifically, rehype-portable-text (by rexxars) outputs an array of Portable Text blocks directly; rehype-remark converts HTML → Markdown for MDX/Git targets (unifiedjs.com; remarkjs/remark-rehype). In Python, BeautifulSoup/lxml + a custom visitor does the same job.

Map known structures deterministically:

Anything you map deterministically is provably correct and needs no review. Maximize this layer.

Layer B — LLM-assisted, schema-constrained conversion (for the messy tail)

The remaining content — page-builder soup, custom HTML embeds, semantically-meaningful-but-structurally-ambiguous markup, "this two-column div is really a callout" — is where an LLM earns its keep. The 2026 best practice is schema-constrained extraction: never ask the model for free-form output; force it to emit JSON that conforms to your block schema.

• Define the target as a Pydantic model or JSON Schema and use constrained decoding (e.g. the Outlines library, or vendor JSON-mode / structured-output APIs) so the model is guided token-by-token and physically cannot emit a block type or field that violates the schema (Merchynt LLM-ready data generator; quoleady). This converts "hope the LLM returns valid JSON" into a guarantee.
• The research dataset ScrapeGraphAI-100k (arXiv:2602.15189, 93,695 real prompt/schema/HTML/response examples from Q2–Q3 2025) confirms schema-constrained HTML→structured extraction is now a well-characterized task with measurable complexity, not a science project.
• Prompt pattern: give the model (1) the HTML fragment, (2) the JSON Schema of allowed blocks, (3) a hard instruction: "Preserve all text verbatim. Do not summarize, rewrite, or invent content. Map structure to the nearest allowed block. If unsure, emit a rawHtml fallback block and set needsReview: true." The verbatim-preservation clause is non-negotiable — an LLM's instinct to "improve" prose is a content-integrity bug at migration scale.
• Always validate the model's output against the schema and against a text-fidelity check (see 17.6). The LLM is a transformer, not an authority.

Cost/throughput at 1,000+ pages: route by difficulty. If 90% goes through Layer A (deterministic, $0) and only the ~10% hard tail hits an LLM, you are calling the model ~100–150 times, not 1,000+. Use a cheap fast model (e.g. a Haiku/Flash-class tier) for the bulk and reserve a frontier model only for the few documents that fail validation. Batch APIs cut cost further for non-interactive runs. The deterministic-first split is what makes LLM migration affordable.

17.5 Stage 3b — Media migration

Media is the most underestimated workstream because WXR omits the actual files (WordPress/data-liberation). Plan it as its own pipeline:

1. Enumerate every asset from /wp/v2/media (URL, mime, alt, caption, dimensions) into media-manifest.json. Cross-reference with mediaRefs harvested from content so you catch images referenced inline but not in the Media Library, and flag library assets referenced by nothing (candidates to drop).
2. Download originals to disk (originals only — regenerate derivatives on the new platform; do not migrate WP's -300x200 thumbnail explosion).
3. Upload to the new home and capture the returned asset ID for each. Options:
   • DAM/CDN with upload API — Cloudinary's upload API ingests from local paths, remote HTTP/HTTPS, S3 URLs, data URIs, even FTP, so you can often upload directly from the old URL without a download hop; its CLI batches and supports "lazy migration" (upload hot assets now, the long tail on first request) (Cloudinary docs/blog).
   • Object storage (S3/R2/GCS) + your own URL scheme.
   • CMS-native asset store (Sanity assets, Strapi upload, Payload media).
4. Build a media ID map: legacyAttachmentId → {newAssetId, newUrl}. This map is consumed by Stage 3 so every image/gallery block resolves to the migrated asset, and by Stage 4's redirect logic for hot-linked image URLs.
5. Preserve metadata: alt text is an accessibility (WCAG 2.2) and SEO asset — never drop it. Carry alt, caption, title, credit.

Pitfall: inline images often hard-code the full old domain (https://old.site/wp-content/uploads/...). A find-replace pass over content using the media ID map must run after upload, or you ship a site that hot-links the soon-to-be-dead origin.

17.6 Stage 4 — Validate & QA: schema, fidelity, links, taxonomy

Validation has four independent gates; a document ships only if it passes all four.

1. Schema validation — every target document validates against the CMS schema (required fields, allowed block types, reference integrity). Fail closed: a schema-invalid doc never reaches the loader.
2. Content-fidelity diffing — the highest-value check. Strip both source HTML and target blocks to normalized plain text and diff them. Word-count delta and token-level diff catch the two worst LLM failure modes: silently dropped paragraphs and "helpfully" rewritten sentences. Flag any document where text similarity < ~98% for human review. (This is why you forbid summarization in the prompt.)
3. Link & media-reference integrity — every internal link resolves to a migrated slug or a redirect; every image block points to a real new asset; zero references to the old domain remain.
4. Taxonomy mapping — verify every legacy term mapped to a target term (see 17.8) and no entity lost its categories/tags.

For site-level QA, crawl-diff staging against production. Screaming Frog (or Sitebulb) supports crawl comparison with URL Mapping, letting you compare two different URL structures (different hostnames/directories) and surface every status-code change, missing title/H1/meta, broken canonical, and lost hreflang; its Content tab reports near-duplicate similarity so a faithfully migrated page shows ~100% match and a mangled one stands out (Screaming Frog tutorials). Baseline a full pre-migration crawl (every URL, status, canonical, title, H1, meta, internal-link count) so you have something to diff against (BrightEdge 2025 migration guide). For rendered/visual proof, layer screenshot-based regression (e.g. a Playwright visual-diff pass, or Litmus for email/render checks) on the top-traffic templates.

17.7 URL mapping & redirects (protecting SEO and AI equity)

This is where migrations win or lose traffic. A 301 redirect map — a spreadsheet of oldUrl → newUrl — preserves link equity and ranking power across the move; 301s pass PageRank and apply equally to server-level or CMS-integrated redirects (Duplicator; PS Branding). Enterprise sites that map 1:1 and monitor Search Console recover 95%+ of rankings within ~90 days; generic "redirect everything to the homepage" is the classic traffic-killer (BrightEdge; americaneagle).

Build the map programmatically, not by hand:

1. Inventory every live old URL — combine your WP index.json, the XML sitemap(s), and a full Screaming Frog crawl (catches paginated archives, attachment pages, feed URLs, and pretty-permalink variants).
2. Derive new URLs from the IR slug + target route structure. Because each IR record carries legacyUrl and the new slug, most of the map auto-generates.
3. Handle the WordPress-specific tail: date-based permalinks (/2019/04/slug/), ?p=ID ugly permalinks, /category/.../, /tag/.../, /author/.../, attachment pages, and feed URLs (/feed/). Each pattern needs a rule.
4. Map dead/merged content to the closest live page, never to the homepage; document deliberate 410s for genuinely retired URLs.
5. Preserve AI equity (2025–26 concern): inventory existing schema.org markup (Article, FAQPage, Product, BreadcrumbList) and re-emit it on the new pages — losing structured data can make AI search engines stop citing you (BrightEdge; orangeseo). Carry over or upgrade llms.txt if present, and keep canonical tags consistent.

Implement redirects at the edge/server for performance (Nginx/Apache rules, Vercel/Netlify _redirects, Cloudflare Rules) rather than a slow per-request CMS plugin, and keep redirect chains to a single hop.

17.8 Taxonomy & relationship mapping

WordPress taxonomy (category, post_tag, custom taxonomies) and parent/child page hierarchies must be reconstructed in the target model. Build a taxonomy crosswalk (legacyTermId → targetTermId) before loading content, because content references terms by ID. This is also the cleanup opportunity: 1,000-page WP sites typically carry hundreds of near-duplicate, single-use, or typo tags. Use the migration to consolidate and rationalize the taxonomy — but make the merge decisions explicit in the crosswalk so they are reviewable and reversible. Resolve post_parent chains into your target's hierarchy/reference fields, and recreate author records (or map them to a byline field if authors aren't first-class in the new model).

17.9 Stage 5 — Load: idempotent, keyed, resumable

Push to the target via its ingest path — Contentful Management API, Sanity dataset import / mutation API, Strapi REST/GraphQL, Payload local API, or direct DB seed (thepublive; Sanity Learn). Non-negotiables:

• Upsert by legacyId, never blind-create — so re-runs converge instead of duplicating.
• Two-pass load for references: pass 1 creates all documents (so IDs exist), pass 2 wires internal links and cross-references. This avoids the chicken-and-egg of A linking to not-yet-created B.
• Resumable & rate-limited — checkpoint progress; APIs throttle, and 1,000+ docs + assets will hit limits.
• Dry-run mode that diffs intended vs existing target state before writing.

17.10 Phased cutover & rollback

Do not big-bang a 1,000-page site. Sequence it:

1. Rehearse on staging — run the full pipeline into a staging copy of the target, repeatedly, until the QA diff is clean. The cutover itself should be short and mostly verification (itmonks; greengeeks).
2. Content freeze on WordPress (or a short freeze window) so no new posts appear after your final extract; if a freeze is impossible, plan a delta-sync pass for content created during the window (extract is idempotent, so a final incremental run is cheap).
3. Phase by risk — migrate low-traffic archive content first to validate the pipeline in production, then high-value pages. Optionally run both sites in parallel with a reverse proxy routing migrated paths to the new system and the rest to WordPress.
4. Cutover during a low-traffic window, flip redirects and DNS, then immediately re-crawl to confirm zero new 404s/500s and that redirects fire (Screaming Frog post-migration crawl).
5. Rollback plan, documented and tested: keep the WP host live and backed up, keep the mysqldump, document DNS reversion steps and TTLs (lower DNS TTL before cutover so reversion is fast), and define the abort criteria and the comms matrix (greengeeks; itmonks). Because your loader is idempotent and your source is an immutable snapshot, "re-run the pipeline" is itself part of the rollback toolkit.
6. Post-launch monitoring — watch Search Console coverage/crawl-stats and server logs for 404 spikes; expect ranking dips to recover within ~90 days if 301s are clean (BrightEdge). Note the mid-2025 Search Console caveat that AI Mode inclusion changed total counts, so baseline carefully (zcmarketing).

17.11 Pitfalls at scale (the long-tail killers)

Key Takeaways

• Treat the migration as an idempotent ETL pipeline (extract → parse → IR → transform → validate → load) where every stage writes to disk and every target doc carries a legacyId for upserts — you will run it 20+ times.
• No single WordPress source is complete: combine WXR (structure), REST API (rendered HTML + meta), and a DB dump (ground truth); WXR notably does not export media files.
• Convert HTML to structured blocks deterministically first with a unified/rehype (or BeautifulSoup) AST pipeline — this handles ~90% for free and reproducibly; route only the messy tail to an LLM.
• LLM conversion must be schema-constrained (Pydantic/JSON Schema + constrained decoding like Outlines or vendor structured-output) with a hard "preserve text verbatim, never summarize" instruction.
• Validate on four gates — schema, text-fidelity diff (catches dropped/rewritten content), link/media integrity, taxonomy — and fail closed; nothing ships that fails any gate.
• Media is its own pipeline: enumerate via /wp/v2/media, download originals only, upload to a DAM/CDN/object store, and rewrite inline URLs via a legacyAttachmentId → newAsset map.
• A programmatic 1:1 301 redirect map (old→new, never old→homepage) protects link equity and AI-search citations; preserve schema.org markup and canonical tags during the move.
• Use crawl-diffing (Screaming Frog URL Mapping/Content similarity) to baseline before and compare staging vs. live after; visual regression on top templates.
• Phase the cutover, lower DNS TTL in advance, keep WordPress + its mysqldump warm as a tested rollback, and re-crawl immediately post-launch; clean 301s typically recover 95%+ of rankings within ~90 days.

Key References

• WordPress / Data Liberation. Let's talk about WXR! (Discussion #56). 2024–2025. https://github.com/WordPress/data-liberation/discussions/56 — Authoritative discussion of WXR's limits: media is referenced not exported, and nested URL-in-JSON-in-HTML-in-XML needs four parsers.
• Sanity. Migrating Content from WordPress to Sanity (Sanity Learn course). 2025. https://www.sanity.io/learn/course/migrating-content-from-wordpress-to-sanity — REST API extraction and HTML→Portable Text conversion as a worked migration.
• rexxars. rehype-portable-text. 2024–2025. https://github.com/rexxars/rehype-portable-text — rehype compiler that turns parsed HTML (HAST) into an array of Portable Text blocks; the deterministic transform backbone.
• unified / remarkjs. remark-rehype & rehype-parse. 2025. https://github.com/remarkjs/remark-rehype , https://unifiedjs.com/explore/package/rehype-parse/ — The AST pipeline (HTML→HAST→target) for deterministic structured-content conversion.
• ScrapeGraphAI. ScrapeGraphAI-100k: A Large-Scale Dataset for LLM-Based Web Information Extraction. arXiv:2602.15189, 2026. https://arxiv.org/pdf/2602.15189 — 93,695 real schema-constrained HTML→structured extraction examples; characterizes the LLM-conversion task.
• Merchynt. LLM-Ready Structured Data Generator: Top 3 Best 2025. 2025. https://www.merchynt.com/post/llm-ready-structured-data-generator — Describes constrained decoding (Outlines, Pydantic/JSON Schema) as a guardrail forcing schema-valid LLM output.
• Cloudinary. Migration Guide / Programmatically Uploading & Migrating Media to the Cloud. 2025. https://cloudinary.com/documentation/migration — Upload API ingest from local/HTTP/S3/FTP, CLI batching, and lazy-migration for large media libraries.
• Duplicator. How to Create a 301 Redirect Map for Website Migrations. 2025. https://duplicator.com/301-redirect-map/ — The redirect-map spreadsheet method (old→new) for preserving SEO during migration.
• BrightEdge. 2025 Guide to a Successful Site Migration: Protect Your SEO in the Era of AI Search. 2025. https://www.brightedge.com/blog/2025-guide-successful-site-migration-how-protect-your-seo-and-grow-era-ai-search — Baseline crawling, AI-equity/schema preservation, 95%+ ranking recovery within 90 days.
• Screaming Frog. How To Use The SEO Spider In A Site Migration and How To Compare Crawls. 2025. https://www.screamingfrog.co.uk/seo-spider/tutorials/how-to-use-the-seo-spider-in-a-site-migration/ , https://www.screamingfrog.co.uk/seo-spider/tutorials/how-to-compare-crawls/ — Crawl baselining and staging-vs-live URL-mapping/content-similarity diffing for QA.
• WordPress Trac. Ticket #10619 — Importing WXR breaks serialized postmeta value. WordPress core. https://core.trac.wordpress.org/ticket/10619 — Canonical bug for double-serialized/corrupted post-meta on WXR import.
• DreamHost Knowledge Base. Importing and exporting a WXR file in WordPress. 2025. https://help.dreamhost.com/hc/en-us/articles/360050852091 — Mechanics and scope of the built-in WXR export (posts, pages, terms, authors, comments, custom fields).
• thePublive. Migrate from WordPress to Headless CMS: Complete Guide for Publishers. 2025. https://www.thepublive.com/feeds/blog/migrate-wordpress-headless-cms — Per-target ingest paths (Contentful Management API, Sanity dataset import, Strapi REST/GraphQL) and scale considerations.
• itmonks. WordPress Migration Checklist: Pre, During, Post-Launch Steps. 2025. https://itmonks.com/blog/wp-development/wordpress-migration-checklist/ — Phased cutover, content freeze, low-traffic window, and documented rollback procedure.