How to use the Robots.txt Generator.
Robots rules control crawling, not reliable removal from search results. Match the correct crawler syntax and never use robots.txt to protect confidential data.
Make the workflow fit your task.
Specify which crawler group and paths the rules should address. Draft the file with those boundaries, then test representative allowed and disallowed URLs before applying it to the intended host.
- What you provide
- Explicit crawler and path requirements.
- What you get
- Robots rules with explanatory examples.
See the input and the result.
Illustrative input and output · a teaching example, not a live WebAct run
Example input
Allow public pages; ask crawlers not to crawl /internal-preview/.
Completed example
User-agent: * Disallow: /internal-preview/ Confirm that private resources also require access control.
Load this input into the prompt, then copy it to WebAct to try the task. Your result may differ from the illustration.
Decisions and troubleshooting.
Can robots.txt keep private pages confidential?
No. Crawling instructions are not access control. Private content needs appropriate authentication or another actual access restriction.
Why does a blocked URL still appear in search?
A crawl restriction is not a reliable indexing removal mechanism. Review the intended indexing controls separately and ensure they can be discovered.
Reference for this workflow.
Try it with your own source.
Replace the example with your material in the task prompt. Keep the requirements you need, then copy the task into WebAct.
Customize and copy the task ↑