/webmasters/community?hl=en
This content is likely not relevant anymore. Try searching or browse recent questions.
-
Remove 404 pages that were indexed but no longer in sitemap 1 Recommended Answer 6 Replies 6 Upvotes
1 Recommended Answer
$0 Recommended Answers
404 pages remain in the search index long after they were removed.
Our sitemap is regularly crawled but pages that have been returning a 404 error for far longer remain in the index. The sitemap doesn't include these URLs anymore.

In the case of this URL https://www.union.edu/art-history/faculty-staff/lorraine-cox
Search Console says it was "Indexed, not submitted in sitemap" and it's last crawl was March 27.  Google read our sitemap on June 26.  

How can we get the index to update itself to remove these pages if they are not mentioned in the sitemap because they don't exist?  

We've done the part of providing a current sitemap.  What can we do to clean out the 404 results in the index from URLs not in the sitemap?

Manual removal is not an option because we don't know how many are in the index.  The Search Console coverage report does not mention any errors related to pages not found
All Replies (6)
Hello
 
This type of pages usually go away within the time - sometimes it's faster, sometimes it's slower. It's very good that they are not in the sitemap anymore.
 
About Google Search Console you can use the Live Test to see what's the current state of an URL. 
 
 
Dido
marked this as an answer
Hello,
 
Do you have any update on your situation?
 
Dido
marked this as an answer
-
So there is nothing that we can do to try and clean out these 404 search results?  3+ months is a long time for a site that is regularly crawled by Google.  

Meanwhile it's the first search result and the correct URL that has been submitted via the sitemap and was last crawled on June 15th is nowhere in the search results.

marked this as an answer
Hi,

We are running a classified website listing apartments. As apartments get bought or rented, they need to be removed from our site because of GDPR reasons. These happen in the hundreds if not thousands each day, so manually removing them is not an option.

Based on the comments in this page, we are planning to build a page, where we will list all the apartment pages that have been closed, so Google would index them faster and see they are 404/no index,  and thus remove them from Google search. 

Do you think this is the best solution? There no API where we could just push the apartment URLs and ask google to remove them from search?  Can this https://developers.google.com/search/apis/indexing-api/v3/quickstart be used for this purpose? It only mentions job ads though?

Thanks for your help!

Br,
Tommi
marked this as an answer
Well the page might help a tiny bit. 
 
... but in general doubt Google would end up crawling it very often. It contains so many broken links!
And as such it wont rank very well, nor show to users. And as such google won't assign much extra priority to recrawling the pages in the page. Meaning it not really helping. 
 
I would say that a sitemap with proper lastmod dates (ie the date the content was deleted!) would be best. 
... could also use a RSS feed style sitemap. 
 
 
In theory could be pro-active with unavailable_after 
... telling bots the page will quickly expire, eg always return say 7 days in future, so in thery if a page is not recrawled within 7 days (and extend the time!!), the page will disappear from results. 
Again maybe manage lastmod times in sitemaps to encourage re-crawling to update the expiry. 
 
 
 
Could also try the indexing API to notify. Although the quota is 200/day. 
... yes, its meant for jobs/liveevents. But the worse taht can happen is get blocked. And then no worse off. 
marked this as an answer
This question is locked and replying has been disabled.
Discard post? You will lose what you have written so far.
Write a reply
10 characters required
Failed to attach file, click here to try again.
Discard post?
You will lose what you have written so far.
Personal information found

We found the following personal information in your message:

This information will be visible to anyone who visits or subscribes to notifications for this post. Are you sure you want to continue?

A problem occurred. Please try again.
Create Reply
Edit Reply
Delete post?
This will remove the reply from the Answers section.
Notifications are off
Your notifications are currently off and you won't receive subscription updates. To turn them on, go to Notifications preferences on your Profile page.
Report abuse
Google takes abuse of its services very seriously. We're committed to dealing with such abuse according to the laws in your country of residence. When you submit a report, we'll investigate it and take the appropriate action. We'll get back to you only if we require additional details or have more information to share.

Go to the Legal Help page to request content changes for legal reasons.

Reported post for abuse
Unable to send report.
Report post
What type of post are you reporting?
Google takes abuse of its services very seriously. We're committed to dealing with such abuse according to the laws in your country of residence. When you submit a report, we'll investigate it and take the appropriate action. We'll get back to you only if we require additional details or have more information to share.

Go to the Legal Help page to request content changes for legal reasons.

Reported post for abuse
Unable to send report.
This reply is no longer available.
/webmasters/threads
//accounts.google.com/ServiceLogin
You'll receive email notifications for new posts at
Unable to delete question.
Unable to update vote.
Unable to update subscription.
You have been unsubscribed
Deleted
Unable to delete reply.
Removed from Answers
Marked as Recommended Answer
Removed recommendation
Undo
Unable to update reply.
Unable to update vote.
Thank you. Your response was recorded.
Unable to undo vote.
Thank you. This reply will now display in the answers section.
Link copied
Locked
Unlocked
Unable to lock
Unable to unlock
Pinned
Unpinned
Unable to pin
Unable to unpin
Marked
Unmarked
Unable to mark
Reported as off topic
/webmasters/profile/0?hl=en