What is SEO?
Search visibility, crawl and structured data
SEO stands for search engine optimisation: improving a website so search engines can crawl, index and understand its pages, and so people are more likely to find a relevant result and choose it. In practice that means useful content, sound technical setup, clear titles, working internal links, good page experience and steady maintenance. These same foundations now also govern whether your pages can appear inside AI features such as AI Overviews and AI Mode.
Reviewed by Jackie, Head of Learning & Development, Levellers · Last reviewed 8 June 2026
What this means
SEO is the practice of improving a website so search engines can find, crawl, index and understand it, and so people are more likely to choose your result when it appears. It is not a trick. It is the combination of useful pages, sensible site structure, clean technical setup, clear titles and descriptions, working internal links, good page experience and regular maintenance. Those basics still matter as search interfaces become more AI-led, because the AI features built into search draw on the same index and the same eligibility rules as ordinary results.
Why it matters
Search remains a practical route into trusted commercial information. People still need to find definitions, service pages, comparisons, opening hours, contact details and proof that you are credible. The fundamentals of SEO remain the entry ticket for AI features in search as well, because a page that is not indexed or not eligible to show with a snippet cannot be drawn into an AI Overview either. For a smaller business, SEO is normal operating hygiene. If your best pages are hard to crawl, duplicated, weakly titled, poorly linked or factually stale, you make it harder for buyers and for search systems to understand what you do. There is also a commercial backdrop worth planning around: recognised analyst forecasting has warned that traditional search engine volume could fall by 25% by 2026 as people turn to AI chatbots and other virtual agents, so the quality and clarity of your pages matters more, not less.
How it works
Search works in three broad stages: crawling, indexing and serving results. SEO improves your pages across all three.
Crawling
Crawling is the automated fetching of pages by software known as crawlers or bots. You influence it with internal links, an XML sitemap (a machine-readable list of the URLs you want found) and a robots.txt file. Robots.txt is the long-standing file at the root of a site that tells crawlers which paths they may request. It was formalised as an internet standard in September 2022, and it is a crawl-control tool, not a removal tool: a path you disallow can still be listed in results if other sites link to it.
Indexing
Indexing is the analysis and storage of a page so it can be retrieved later. Keep important content in text rather than locked inside images or scripts, set a clear canonical preference where near-duplicates exist (the canonical tag names your preferred version of a page), and avoid accidental blocks. To keep a page out of results entirely, use a noindex rule in a meta robots tag or HTTP header, and make sure the page is still crawlable so the rule can be seen.
Serving and ranking
When someone searches, the system selects and orders pages it judges most relevant and useful. Clear title links and meta descriptions, sensible internal linking, and good page experience all help. Structured data, which is standardised machine-readable markup describing a page, can give explicit clues but must match what is visible, and it does not guarantee any special result. The meta keywords tag plays no part: it has been ignored in Google's web search ranking since 2009, when Google confirmed it does not use the keywords meta tag.
Examples
Check that key service pages return a normal success status and are not blocked from indexing. Write clear, distinct title text and meta descriptions. Link related explainers with descriptive anchor text rather than "click here". Review outdated pages, merge overlapping ones, and set canonical preferences so the strongest version wins. Submit an XML sitemap and keep it current. Keep claims supported, dated and accurate, because stale pages erode trust with both readers and systems.
Common misunderstandings
"SEO is about tricks" (no, it is durable hygiene). "SEO is dead because AI search exists" (no, the same foundations decide whether a page is even eligible for AI features). "robots.txt removes a page from Google" (no, it controls crawling; use noindex to remove). "Meta keywords help rankings" (no, they have been ignored for over a decade). "Structured data guarantees a rich result" (no, it makes a page eligible at most).
Risks and boundaries
Following best practice does not guarantee crawling, indexing or serving; none of these is promised even for a technically perfect page. SEO is ongoing, not a one-off project. Page experience is not a single switch. Structured data does not guarantee rich results. And display features can be withdrawn: FAQ rich results, the expandable question-and-answer listings that some sites relied on, stopped appearing in Google Search on 7 May 2026, with the search appearance, rich result report and Rich Results Test support being retired in June 2026 and Search Console API support in August 2026. Treat any single search feature as borrowed, not owned.
What to do next
Audit the ten pages that matter most to revenue or enquiries. For each, confirm crawlability, indexability, internal linking, canonical consistency, clear titles, accurate descriptions and factual currency. Fix the blockers first, then improve clarity. Set a quarterly review so the work compounds rather than decays.
Have a question or a suggestion, or want to understand how we research and review these guides? Read about our editorial standards and how to reach us.
FAQs
Is SEO still worth doing now that AI search is growing?
Yes. The same foundations decide whether your pages can be found at all, including inside AI features, so SEO is the entry ticket rather than a legacy tactic.
Does SEO guarantee rankings?
No. Good practice improves your chances, but crawling, indexing and serving are never guaranteed for any page.
Do meta keywords help?
No. The meta keywords tag has been ignored in Google's ranking since 2009 and is not worth your time.
Should I block weak pages with robots.txt to remove them?
No. Robots.txt controls crawling, not indexing. To remove a page from results, use a noindex rule and keep the page crawlable so the rule can be read.
What is the difference between crawling and indexing?
Crawling is fetching the page; indexing is analysing and storing it. A page can be crawled but not indexed, and occasionally listed without being crawled.
Does structured data improve rankings?
Not directly. It helps systems understand a page and can make it eligible for certain features, but it must match visible content and guarantees nothing.
How long does SEO take to show results?
It varies by site and competition. Treat it as steady maintenance measured over months, not a one-off fix.
What should a small business prioritise first?
Make your most commercially important pages crawlable, indexable, clearly titled and genuinely useful. That foundation supports both classic results and AI features.
Sources
RFC 9309: Robots Exclusion Protocol (IETF (RFC Editor)). robots.txt as a crawl-control standard and the fact that disallowed URLs are not thereby removed from results.
sitemaps.org Protocol (sitemaps.org). XML sitemaps as a way to inform crawlers of available URLs.
Schema.org (Schema.org). structured data vocabulary that describes page content for machines.
Introduction to Information Retrieval (Stanford NLP Group (Manning, Raghavan, Schutze)). academic grounding for crawling, indexing and ranking.
Gartner Predicts Search Engine Volume Will Drop 25% by 2026, Due to AI Chatbots and Other Virtual Agents (Gartner). analyst forecast on shifting search volume.