About robots.txt
Submit your updated robots.txt to Google
The Submit function of the robots.txt Tester tool allows you to easily put in place and ask Google to more quickly crawl and index a new robots.txt
file for your site. Update and notify Google of changes to your robots.txt
file by following the steps below.
1. Click Submit in the bottom-right corner of the robots.txt
editor. This action opens up a Submit dialog.
2. Download your edited robots.txt
code from the robots.txt Tester page by clicking Download in the Submit dialog.
3. Upload your new robots.txt
file to the root of your domain as a text file named robots.txt
(the URL for your robots.txt
file should be /robots.txt
).
If you do not have permission to upload files to the root of your domain, you should contact your domain manager to make changes.
For example, if your site home page resides under subdomain.example.com/site/example/
, you likely cannot update the robots file subdomain.example.com/robots.txt
. In this case, you should contact the owner of example.com/
to make any necessary changes to the robots.txt
file.
4. Click Verify live version to see that your live robots.txt
is the version that you want Google to crawl.
5. Click Submit live version to notify Google that changes have been made to your robots.txt
file and request that Google crawl it.
6. Check that your newest version was successfully crawled by Google by refreshing the page in your browser to update the tool’s editor and see your live robots.txt
code. After you refresh the page, you can also click the dropdown above the text editor to view the timestamp of when Google first saw the latest version of your robots.txt
file.