Create a robots.txt with allow/disallow rules.
With User-agent = *, Disallow paths = The quick brown fox jumps over the lazy dog., Sitemap URL = https://www.example.com/sitemap.xml:
A plain-text file at the root of a website (example.com/robots.txt) that tells web crawlers which paths they may not request.
Not fully. Google cannot index the content of a disallowed page, but the URL itself can still appear in search results without a snippet. To keep a page out of the index, use a noindex directive instead.
At the domain root - for example, https://yourdomain.com/robots.txt.