What is the robots meta tag?

It is written as a meta element naming robots, or naming one crawler specifically, with values like noindex and nofollow. Because it lives in the page's own HTML, it applies to that page alone, which makes it the right tool for keeping a single thank-you page or an internal search result out of an index without touching anything else.

The distinction from robots.txt is the one that causes most of the trouble. A robots.txt disallow asks a crawler not to fetch the page. A noindex in the robots meta tag asks it not to index a page it has already fetched. Those are different requests and they interact badly: a page blocked in robots.txt is never fetched, so a noindex inside it is never read, and the page can persist in an index on the strength of links alone. If the goal is removal, the page has to stay fetchable long enough for the tag to be seen.

For AI answer engines the same split applies with an extra wrinkle. An answer crawler that is refused at robots.txt never reaches the page, so nothing in the head matters. One that is allowed through reads the tag like any other crawler. Neither directive decides whether an assistant chooses to quote a page it is permitted to read, which is a separate question this tag has no opinion about.