Bring your existing docs to Thally with a URLBring your existing docs to Thally with a URL

You can now migrate your existing documentation site to Thally using its public URL.

Thally already supports migration from a source GitHub repository through Thally Cloud, the CLI, and MCP. We've extended the same migration engine to support web URLs.

How to migrate

1. Enter your docs URL. Create a site in Thally Cloud, choose URL import, and enter a URL such as https://docs.example.com/getting-started.
2. Run the import. Thally discovers pages and navigation, captures content, and converts it into a Thally project.
3. Verify. Thally builds and checks the imported site. Failed checks block publication and show recovery guidance.
4. Review and publish. Confirm the content and GitHub destination, then publish. Thally creates the repository and starts the managed deployment.

How we built it

URL migration runs on @thallylabs/migrate, the open-source engine used by Cloud, the CLI, and MCP. Cloud adds durable jobs, capture providers, sandbox verification, and publication.

1. Discover the docs

   Use the URL scope, navigation, sitemaps, and llms.txt to build an inventory.

2. Capture the source

   Native HTTP by default, or Firecrawl /v2/scrape for raw HTML when configured.

3. Convert and generate

   Cheerio, Turndown, and remark produce a bundle, then MDX, docs.json, and hashed assets.

4. Build and verify

   Daytona builds the Cloudflare artifact. Playwright checks it through workerd with Firefox.

   Checks fail: report the issue and retry using saved progress. Publication stays blocked.

5. Review and publish

   Passing verification binds the saved files to their manifest hash before publication.

6. GitHub repository and Thally site

   Publish the verified content into a repository you control and a managed site.

Discovery

Thally builds a page inventory from navigation links, sitemaps, llms.txt, and supported publisher metadata. Discovery stays within the selected origin and path scope, retaining top-level tabs and sidebar groups. Thally owns this inventory regardless of the capture provider.

Capture with native HTTP or Firecrawl

Native HTTP fetching is the default. When configured for rendered capture, our Firecrawl adapter calls POST /v2/scrape for each selected page with these key options:

{
  "url": "https://docs.example.com/getting-started",
  "formats": ["rawHtml"],
  "onlyMainContent": false,
  "onlyCleanContent": false,
  "maxAge": 0,
  "storeInCache": false
}

Full raw HTML preserves sidebars, tab relationships, and embedded page data that a cleaned Markdown response can lose. Both providers feed the same conversion pipeline.

The adapter bounds response sizes, checks target status, and validates the returned URL against the original origin and public-host policy. Credentials go only to Firecrawl. We send no click or execute actions, since page controls can run arbitrary logic. Unavailable interactive content is reported as an import issue.

Conversion to MDX

Cheerio identifies article structure and supported tabs, cards, tables, and API fields. Turndown converts HTML into Markdown; unified and remark tooling parse and validate MDX.

For supported publishers, literal React Flight payloads and declared Markdown alternatives can supply code examples absent from the initial HTML. Embedded source must agree with the visible article before use. The parser never evaluates its scripts, imports, or MDX expressions. Retained code payloads let conversion detect changed or dropped examples.

Supported images are checked and stored under content-hashed filenames. Internal links and asset references are rewritten for the destination. The renderer emits MDX pages, docs.json, and local assets from the migration bundle.

For unfamiliar layouts, an optional Anthropic-backed planner proposes content and navigation selectors from a bounded DOM sketch. Technical payloads stay outside the model. Deterministic coverage checks validate extraction; the model selects source nodes rather than rewriting documentation.

Recovery

Captures, converted records, assets, and checkpoints are persisted with SHA-256 hashes. A page is acknowledged only after its evidence and output are saved. Retries reconcile stored artifacts and reuse completed work. Capture and converter versions prevent incompatible checkpoint reuse.

Build verification

A dedicated Daytona sandbox installs the pinned project and builds the managed Cloudflare artifact. It serves the build through workerd. Playwright with sandboxed Firefox checks imported routes, code examples, internal destinations, assets, the entry route, and navigation readiness.

The verification report is bound to the generated files' manifest hash and pinned scaffold release. Publication consumes those saved, verified files. Failed checks block publication; retry can verify the saved output again.

Current limits

• Public sites only. Authentication walls, blocked requests, and undiscoverable pages can prevent complete capture.
• Up to 1,000 pages in the web flow. Byte and execution budgets also apply. Larger sites should use the CLI, which has its own limits.
• Custom presentation needs review. Bespoke layouts, fonts, and CSS may need work. Unsupported components, unresolved lazy content, and incomplete examples can block verification.
• Snapshot import. Later changes to the source site aren't continuously synchronized. Supported videos retain their original public URLs.

Try URL migration

Create a site in Thally Cloud and choose URL import.