On This Page
We shipped a new feature this month that we think will matter more and more as discovery shifts toward AI-mediated experiences: DataJelly now generates a clean, structured Markdown version of every snapshot and serves it to AI crawlers.
DataJelly has always focused on a simple promise: humans get your site unchanged, and bots get the version they actually need. For search engines, that means fully-rendered HTML snapshots. For AI systems, HTML is often the wrong transport format.
The Problem: AI Systems Don't "Browse," They Extract
Modern JavaScript sites tend to produce HTML that's optimized for browsers, not retrieval:
- Heavy "div soup" and styling scaffolding
- Repetitive nav/footer/UI chrome
- Components that bury the primary content
- Huge token overhead before the model even reaches the point
Even when the content is present, it can be difficult for AI crawlers to consistently isolate what matters.
What We Added: Markdown as a Bot Delivery Format
When a snapshot completes, DataJelly now produces two bot-ready outputs from the same rendered page:
- Rendered HTML for search crawlers (Google, Bing, etc.)
- Clean Markdown for AI crawlers (ChatGPT, Claude, Perplexity, other LLM-based agents)
The Markdown output is designed to preserve meaning and structure while stripping markup noise:
- Headings and hierarchy preserved
- Links normalized and retained
- Main content extracted (excluding nav/header/footer)
- Consistent snapshot header metadata (crawl date, source, URL)
Why Markdown Helps
Markdown is a better "transport format" for AI retrieval because it's:
- Token efficient — less markup, fewer wasted tokens
- Structurally explicit — headings/lists are clear
- Cleaner for chunking + embeddings — better downstream retrieval
- Less ambiguous — less UI noise mixed into content
This isn't a claim that "Markdown magically solves AI SEO." It's a practical way to ensure AI systems receive your real content in a form they can reliably process.
Measurable Impact: Token Reduction
We built this to be measurable, not a vibe.
For example, on one DataJelly page the AI Markdown output reduced token usage from ~42,112 tokens (HTML) to ~3,704 tokens (Markdown) — a ~91% reduction — while preserving content and structure.
You can see this per route in the dashboard, including total tokens saved across your domain.
Bot Delivery Transparency in the Dashboard
Alongside the feature, we added dashboard views that show exactly what each system receives:
- Bot Delivery view across all snapshots (Human / Search / AI)
- Snapshot Details with direct "See what AI bots see" and "See what search crawlers see"
- HTML vs Markdown Bake-Off comparison with token reduction and a Markdown quality score
The goal is full transparency: you should be able to inspect the real delivered output, not guess.
Control: Enable/Disable Per Domain
AI Markdown is enabled per domain and can be toggled at any time:
Domain Details → AI Markdown Response → Enable/Disable
If Markdown generation fails for any reason, DataJelly falls back to serving the normal rendered HTML snapshot.
Who This Is For
This feature is most useful for teams shipping:
- JavaScript-heavy marketing sites and apps
- AI-generated sites built with tools like Lovable/Bolt
- Content where AI visibility (retrieval, citations, answers) matters alongside classic SEO
We'll keep iterating on extraction quality, structure preservation, and scoring as we see more real-world domains and edge cases.
For a deeper technical dive into how AI Markdown extraction works, see our full guide: AI Markdown View Guide.
Related Reading
AI Markdown View Guide
Technical deep dive into how AI Markdown extraction works.
Understanding the Bots
The three types of bots and what each one needs.
DataJelly Edge
Edge rendering that delivers the right format to every bot.
Bot Test Tool
See what specific crawlers receive from your pages.
HTTP Debug Tool
Compare raw vs rendered responses across user agents.
AI Visibility Infrastructure
Whitepaper on token efficiency and multi-format delivery.