googlebot access to wp-content folder
The problem I have with letting Googlebot run rampant through my plugins and themes is they have a tendency to end up in the index for any hacker to exploit. Too many times in the past I have found different plugins and themes listed in the Google index when I do a site search. So until Google stops indexing what it finds in the wp-content folder on wordpress sites, I will go for security rather than a happy Googlebot. thank you.
Yes I am interested.
In case you are worried that some of these files get indexed, you can place this code into your site's main .htaccess file (or into a .htaccess file placed directly into these directories)
<Files ~ "\.(css|js)";>
Header set X-Robots-Tag "noindex"
On Sunday, March 15, 2015 at 3:01:20 PM UTC+1, flyfishingcolorado wrote:
So you're saying use htaccess to manage access to the wp-content folders instead of robots.txt. Will I then see a lot of 403 crawl error messages on my webmaster tools dashboard?
There are two separate things, blocking stuff in robots.txt , and returning status 403 Forbidden for URL of folders without index files.
Look at a folder URL without index fike, for example /images/ or /wp-content/ , if you see a list with links of all files and sub-folders, this is where Google and other search engines find and collect URLs that you think cannot be found.
If the folder URL returns status 403 (you can see the status with Fetch as Googlebot), then Google will not index or process the URL, it will just see status 403 and go crawling other URLs. But this is if the URL is not blocked in robots.txt, if it is blocked Google cannot see the status 403. If there are web crawl errors 403 in Webmaster Tools, you can see where the URL is linked from and fix the link.
One more question. If I am on shared hosting. If I run a master robots.txt for all my add-on domain sites, won't I see messages on webmaster tools about not having a robots.txt in each add-on site? Or would the main robots.txt file have to contain code blocks for each add-on site in my account?
I could open up the wp-includes folder of course. along with the total wp-content folder. Then put an htaccess no index file in each folder. If I did that how do I know Googlebot or any other major bot would honor this. Doing it this way defeats all of the grand filters in my main htaccess file I would think.
Some community members might have badges that indicate their identity or level of participation in a community.
Member levels indicate a user's level of participation in a forum. The greater the participation, the higher the level. Everyone starts at level 1 and can rise to level 10. These activities can increase your level in a forum:
- Post an answer.
- Having your answer selected as the best answer.
- Having your post rated as helpful.
- Vote up a post.
- Correctly mark a topic or post as abuse.
Having a post marked and removed as abuse will slow a user's advance in levels.
View profile in forum?
To view this member's profile, you need to leave the current Help page.
Report abuse in forum?
This comment originated in the Google Product Forum. To report abuse, you need to leave the current Help page.
Reply in forum?
This comment originated in the Google Product Forum. To reply, you need to leave the current Help page.