A growing share of discovery now happens inside AI answers, not blue links. Optimizing for that is called AEO (Answer Engine Optimization) or GEO (Generative Engine Optimization) — and your Next.js architecture directly affects it.
What AI crawlers change about the game:
- Most don't execute JavaScript. GPTBot, ClaudeBot, PerplexityBot read the initial HTML. If your content is client-rendered, you're invisible to them — rendering strategy just became twice as important.
- They parse structure, not design. Semantic HTML (
<article>,<h2>hierarchy,<table>, lists) and JSON-LD are what they can reliably extract and quote. - They quote answer-shaped content. A section that opens with a direct one-paragraph answer to a specific question gets cited; a wall of marketing copy doesn't.
Concrete steps in a Next.js project:
// app/robots.ts — decide explicitly which AI crawlers you welcome
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{ userAgent: "*", allow: "/" },
{ userAgent: "GPTBot", allow: "/" }, // OpenAI
{ userAgent: "ClaudeBot", allow: "/" }, // Anthropic
{ userAgent: "PerplexityBot", allow: "/" }, // Perplexity
{ userAgent: "Google-Extended", allow: "/" }, // Gemini training
],
sitemap: "https://example.com/sitemap.xml",
};
}
- Consider an
llms.txtat the site root — an emerging convention (a curated markdown index of your key pages) that some AI systems already read. - Add FAQPage schema to pages that answer real questions — the same markup powers both Google rich results and AI citations.
- Keep dates honest and visible (
datePublished/dateModifiedin JSON-LD + on the page). AI engines strongly prefer verifiably fresh sources. - Write quotable paragraphs: question as heading, direct answer in the first two sentences, detail after.
This is a deep topic — we offer it as a dedicated service (Generative Engine Optimization), but the checklist above covers the technical foundation.
Checklist:
- All indexable content present in initial HTML (no JS required)
- AI crawlers explicitly allowed (or blocked — but decide, don't default)
- FAQ/HowTo schema on question-answering pages
- Key pages open sections with direct, citable answers