B
Briann
Member since 8/12/2022

Achievements

To start earning badges, sign up to be a Product Expert in our Product Expert Program .

Activity

No recent activity
Post history
Thanks.

So about about those pages that have been deleted/redirected but still appear as indexed and show in search? How do I force Google to visit those pages and remove them from search?
Hi,

My question was if I should cancel the 2nd request or leave it as it is.

I check the server logs every day, nothing is wrong.
Every redirect is working and done perfectly. 

Here's the GSC data maybe it says something to you:


​But why duplication appeared? May be you have used 302 instead of 301? Or something another is incorrectly migrated. Or may be something else ... without URls it is difficult to me to answer.
The redirect is done correctly. Google reports these URLs as duplicate on the old domain (they appear fine on the new domain, indexed and crawled) because it didn't crawl them not even once after the domain migration. Like, I did the domain migration on July and the last crawl on these particular pages on the old domain has been in May and June. Hence, Google hasn't seen the redirect and the new canonical and marked the pages as Duplicate because at the time of the crawl the "User-declared canonical" was the old domain and now Google chose the canonical URL the new domain. Obviously, the canonical URL is the new domain but because Google didn't crawl the page it didn't see it.

I keep trying the manually index these pages every week but the report does not update.

​Check internal linking - have you changed it to new URLs? During migration all internal linking should be changed.

Yes, I have updated all internal links with the new domain.

​About these duplication: check canonical tag implemented by you (user) and what Google took as a canonical - same address or another?

The current canonical for these pages is the new domain. Howerver in the report it states that the "User-declared canonical" is the old domain and Google choose the new domain as canonical so it marked the pages as duplicate. The problem is that Google hasn't crawled these pages after I done the migration so it didn't confirmed the redirect and the new canonical. Again, this is only happening if I inspect the URLs on the old domain. On the new domain, these pages are fine.

How can I solve this? How can I force Google to crawl these pages. No matter how many times I trigger the "manual indexing" nothing happens. The report does not update and these pages are still marked as duplicate on the old domain.


Hi gonz0, I'd prefer not to. I'd love the have these questions answered by you:

  1. Do you think it is still possible to recover after 6 months?
  2. Do you think this duplicate issue could impact the migration?
  3. Should I trigger another "Change of address"?
Yes, everything redirects to the new domain 1:1 except for the sitemap. I have left the old sitemap in place on the old domain for crawling's sake.

*I have updated all internal links
*Fixed all broken links
*Updated all social media links
*Updated the most impactful backlinks

Everything looks flawless in GSC except for 10 pages on the old domain. 

These 10 pages are marked as "Duplicate content" by Google because Google hasn't crawled these pages not even once after these 6 months post domain migration. The report states these 10 pages have been last crawled 7 months + ago and Google saw the "User-declared canonical URL" as the old URL at that time in the meanwhile he choose the canonical URL the new domain but it didn't updated the "user-declared canonical", hence the duplicate content.

No matter how many times I trigger the "manual crawling" in GSC for these pages the report does not update (even though I can clearly see in logs that Google entered these pages on the old domain and it has been redirected to the new domain).

I also requested the "Validation" for these 10 pages marked as Duplicate like 2 months ago but literally, nothing happened. It likes the old property in GSC is frozen, nothing updates anymore.

  1. Do you think it is still possible to recover even after 6 months?
  2. Do you think this duplicate issue could impact the migration?
  3. Should I trigger another "Change of address" so that Google would crawl these 10 pages again?
Bump, please does anyone has an answer for this?
As an offtopic, I was temporarly banned from the platform because of this thread, I don't know why? If I do anything wrong I don't intend to, so please let me know if I violate any of the policies. I was a bit surprised but it looks like the "ban" was reversed. My continues threads about my current problems with the migration could look like spam, maybe? But I'm really desperate with the situation and I try to keep asking questions until I find a solution.

Ontopic: I understand everything you say but the domain migration I did was just to move to the plural version of the domain. It's basically the same brand name. I did everything by the book and still lose the traffic and have big problems recovering it. 

And it looks like the domain migration is a big deal just for Google. Other search engines have no problem with it. I don't understand where all these concerns come from. A domain migration should come with no compromises if done correctly. If it's 100% the same site there shouldn't be any logic explanations for hindering the recovery process. 

Anyway, thanks a lot for your input. So it's normal to have no "moving site" reported in the GSC for the www. version of the new URL since I triggered the change of address to the non www version.
But I seen people and an actual case study that claims it's super important to trigger the Change of address on all URLs. And I don't know exactly if I should change the www version of the old domain to the www version of the new one or to the one I prefer to have indexed.

It's super annoying that Google does not have a comprehensive guide on every detail about domain migration and also that does not notify you if anything goes wrong. I say this because I noticed and also saw almost everyone saying that Google is more problematic with domain migration than Bing or any other search engine. Literally every small detail that's not mentioned in the guidelines if done wrong can hinder the recovery process. The process is in essence super straightforward, it should be enough to redirect and request the change of address, which is for other search engines, but it looks like Google needs more than that to take the move for granted. 

Saw plenty of cases where the migration was done flawless but still took a long time to recover completely, and in others, never recovered. This is alarming. 

And those people that say traffic is usually recovered after 1 month or so probably never did a domain migration in their life or have moved very small sites or authority ones but no in between.
Hi,

May I ask what are your thoughts about the thing that Google does not read/consider the old sitemap? I think if I manage to make Google crawl the sitemap things will improve considerably.

Why is that?

- Is see on the old domain that some pages have been last crawled on June (right after I did the domain migration). Some of these pages are also marked as "Crawled not indexed" or "Page is not indexed: Duplicate, Google chose different canonical" instead of "Page with redirect" since they haven't been crawled for ages. This is another signal that Google does not read the sitemap.
- The crawling rate on images is deadly slow (3-5 crawls per day) and I have more than 5000 images on the old domain that needs to be crawled and marked as redirect. Considering that Google has no sitemap, it has no source to see the old URLs of images and confirm the redirect.
- Google keep crawling the category pages on the old domain over and over again. If it would read the sitemap I'm sure it would visit inner pages and images as well.
- All old URLs are missing the "Referring page" or "Sitemap" in the "Inspection URL" report. That clearly indicates that Google visit the old URLs from completely other sources and this may considerably slow down that crawling process.

I saw few people claiming that is a super bad idea and could break the process if you redirect the sitemap too after domain migration (which I did). Is it true? 

So for the first three months after migration, all old sitemaps have redirected to the new sitemaps, so basically I had no mapping for the old URLs at that time. 

In September, however, I added an old sitemap on the old domain and leave it for one week after I deleted it (I was scared that it would create a conflict between the old and new URLs). After doing that I saw a considerable increase in the crawling rate on the old domain and I also noticed that the sitemap appeared as "Reffering page" for a couple of old URLs when I inspected them. This indicates that Google indeed read the sitemap at that time and crawled a part of the URLs.

I tried to do the trick again on Octomber and again on November but it didn't worked at all. Google has stopped read the sitemap after I deleted it in September.

What do you think?
Before the domain migration, the website was ranking on plenty of keywords. Day 4-5 after the migration 95-98% of pages have dropped from the search and only 2-5% have kept their position (the exact same pages that rank now and have ranked all the time after migration).

I think it has nothing to do with algo changes. I haven't seen any movement in clicks/impressions during the updates so they didn't impact the website at all.

And even if it was like you said, the 95% of pages that are vanished from Google are clearly good enough to be at least in the top 100 to receive impressions. Some pages that are now 1st on Google have poorer quality and more aggressive competition than other pages that haven't seen the light after migration. 

Also, if the website would have been affected by any update I don't think Google would crawl the new pages in a few hours.
Is there any chance for Google to encounter errors crawling/indexing/viewing the site without notifying in GSC or show the logs in the Coverage report? Because as I said, the Coverage report for both old and new domain is as clean as possible (all pages appear as "Page with redirect" for the old domain and "Indexed" for the new domain). I don't see any error in the "Crawl stats" either, 100% of crawls on the old domain are 301 and on the new 200 or 304.

Another "good" sign is that new content on the new domain is indexed in 1-2 hours. 

But I really don't get why Google prefers to only show 5-10 pages in search (and rank them on the first page) and what stops it from showing the rest. If there would be something wrong I doubt those pages would rank at all, but they do. And this behavior of Googlebot to only visit the category pages on the old domain drives me crazy. 
I'm really hopeless. I keep asking people on all platforms but no one can figure out what is going wrong. I have almost no saving lefts because the current earnigs bearly support the costs of the website and my living (the website is practically all I do for living) and I feel so down knowing that I can't recover it from something that simple and stupid. I don't even have the money to hire someone to audit the migration..
The old site received around 1000 clicks per day, the new site receives only 70-80. I'm saying the new site has the exact traffic it had Day1 after domain migration. It feels like the traffic and ranking keywords are capped to a static value. The new site is ranking with only a couple of keywords (those keywords haven't changed their position not once, they are literally taking the first position) while the rest 500 keywords + are vanished, don't receive a single impression. How could Google rank those pages so well on the new domain and vanish the rest?

---------------

I'm almost 90% sure that it has to do with the crawling rate on the old domain and with the fact that Google didn't checked every redirect, especially on images. It's super strange that Googlebot keeps accessing the same category URLs on the old domain over and over again while the inner pages and images are rarely crawled. Could this indicate a server problem? On Bing and other search engines I fully recovered the traffic and rankings after two weeks but Google seems to be a bit slower with the process. If there were a technical error I think I would have problems recovering the traffic on other search engines as well

Hi,

Thanks for your input.

The website has the exact traffic and number of "ranking pages" as it was right after changing domain so I don't think it was affected by any algorithm update. I can give you the domain and server logs (in private) if you are willing to take a look at it.
Need help recovering after domain migration. Google does not read/crawl the old sitemap?
Google Search Central11/29/2022
I have opened a couple of threads about my issues with domain migration and I'm still in the same position (clueless and hopeless). Opening a new one with a more specific question.

TL;DR - Did a domain migration 3-4 months ago and traffic is still 80% less from what was before. I still have a couple of keywords ranking on 1-3st and generating the 20% traffic, but the rest of pages are basically vanished (receive 0 impression on them, like they don't even rank in the first 200 searches).

Done a lot of research on this. There's literally no error in server logs for both old and new domain. Everything looks perfect.

Howerver, I started noticing Google crawling rate on the old domain is very slow. 

80% of crawls are wasted on category pages and very few crawl the inner pages and images. My guess is that this is the reason for slow recovery because Google didn't had the chance to confirm all the redirects.

It seems like Google does not read the old sitemap so it basically has no source to take all the old URLs (inner pages and images) and confirm the redirect. I think this is also the reason Google keeps crawling the category pages because it has no other source where to view the other links. The fact that there are only 5-6 requests per day on the "Crawl requests: Image" report confirms my theory that Google didn't saw the redirect on all the images (it would take years to check every image with that crawl rate)

Most old URLs also have no "Referring page" stated in the "URL Inspection" report. If Google would see the sitemap, shouldn't all URLs have it in the "Reffering page" or under the "Sitemap" section in the report?

Other than that what could possibly be the reason for the slow recovery? I have checked everything countless times and I couldn't find anything that would hinder the process.

I had 900-1000 clicks per day from Google, now I bearly have 70-80. I'm already thinking of quitting this one but I put a lot of work into it for the last 4 years and it would be super sad to waste it on such a stupid move.

I'm sure there's something to be done but I have no idea what.

I can provide server logs, GSC reports or anything else if someone is willing to take a closer look into what's going on with Google.
11 replies
1 Recommended answer
Google Search Central11/29/2022
Bumping this one!
I'm also considering the time factor but I'm trying to do my best to ensure nothing hinders the process.

In your case, how is the crawling on the old domain? Because in mine, 90% of crawls are on the category pages and the rest of inner pages and images. I don't understand why it has this behavior, what it sees on these category pages to keep it crawling like that (sometimes I see 20 requests in a single day on a specific category). I think this is a huge factor that slows down the recovery because Google wastes a lot of time crawling category pages that don't actually change. I'd love to see how the crawling is going on your old domain.
Thanks. Honestly I can't ingore the old domain because I have a feeling something is wrong. There's no reason not to be recovered already since we already have all new URLs indexed.

What do you say about the fact that Google most of the times (>90%) when visits the old domain crawls the URLs of the category pages and forget about inner pages? Sometimes it crawls one category URL 20 times in a single day although nothing has changed. Why it keep visiting those URLs over and over again? What makes it trigger this behavior? I feel like something is going on.
Thanks!

And it's OK to "revamp" the old sitemap after two months post migration?

And regarding the robots.txt, it has the same rules but I was thinking to leave the old file in place so that I can point to the old sitemap. What do you think?
Google Search Central11/19/2022
Even if those redirects have been changed years ago?
Hi, thanks for your answer, really informative.

What do you think about the fact that Google keep crawling the category pages but not the inner pages on the old domain. Given that some URLs haven't been crawled for more than a month (as per the inspection tool) shouldn't Google check the redirects again?
Hi Barry, yes I wouldn't be worried about that if traffic was recovered, which is not the case. Very few pages appear in search and generate traffic, while the rest (which the website used to get traffic on) receive ZERO impressions like they don't even exist

Also, is it normal that even after 2-3 months Google still chooses the old URLs as canonical for some pages? I'm thinking this is creating a conflict between the old and new URLs which causes Google not to rank any of our pages until it figure out which one is the real canonical.

(and even when Google has picked the old URL as canonical for a specific page, it still indexes both the new and old URLs of that page, is this normal?)

Another thing I have noticed is a discrepancy between the server logs and GSC reports. If I manually inspect an old URL that I see in logs that Google has recently visited, the report states it was last crawled 2 months ago (around the time when I triggered the domain change).

Any thoughts?

Should I give it more time?




This is what I don't understand too.

I have deleted the subdomain long time ago but still since I have done the domain migration I see on the old property that Google keeps crawling this old subdomain ads.txt file. In the Crawl stats report there are two requests every day on this file and all return DNS error. Strangely I never uploaded an ads.txt file on that subdomain.

If I insepct the URLs there's no "Referring page" stated in the report.


So nothing about https://www.stylecraze.com , https://bestreviews.com/ , https://my-best.net/ , https://www.findthisbest.com/

Those sites are ruining Google searches.
Google Search Central8/18/2022
Hi Kusshwaha, 

The whole website has been redirected 1:1, every new URLs matching the old ones.

However, I have noticed for some time that in GSC, we have quite a few pages "not indexed" because they are pages with redirects. This is because:
- For some pages, Google finds both versions of the URL with "/" and without it, and since the version with the trailing slash redirects to the version without (which is also the version in the sitemap), Google sees it with a redirect. However, the versions without the trailing slash are indexed and good, that's why I thought I shouldn't worry about it. The strange thing is how Google found those versions of the URL ending with trailing slash. We literally have no internal links pointing to them. We also tried to inspect the URLs to see the referring pages, but we couldn't find any URL ending with "/" on those pages.
- Also, since we have migrated the domain, Google sees all the redirects we did on the old domain 
and shows them in Coverage as excluded due to "Page with redirect." I think this is normal since those URLs have been actually redirected. The actual and final versions of the URLs are indexed. But why does Google prefer to show those old URLs that have been previously redirected on the old domain quite a long time ago and display them as excluded pages in GSC? 
Google Search Central8/15/2022
@Lanh Đồng Of course we followed every guideline from Google. 
6 months sounds insanely long for a domain migration. I hope that is not the case as we can barely "survive" that long without generating revenue. 

Any insights from other webmasters that actually did a domain migration? What was your experience?
Google Search Central8/12/2022
@MartinMoore Yes, this is exactly what I have noticed too. The traffic appears capped and Google choose to only rank few pages on top of the search while the rest remain out of 100. 

I mean, out of 300 pages, 8 pages are first on Google and the rest out of 100. How Google can rank only 3% of my pages, and all those 3% be 1st while the rest can't be found anywhere on search.

And I'm sorry to hear your rankings haven't been recovered. This is really worrying.
Question regarding domain migration
Google Search Central8/12/2022
Hello webmasters,

On 27 June, we did a domain migration, and everything went smooth for the first week. The website didn't suffer any downtime, and for the first week, the rankings were stable, and the whole traffic was redirected to the new domain. 

However, after a week, we saw most of our keywords thrown out of searches, and the traffic went down from 800 to 200 users per day. 

We expected that behavior, but we didn't expect it would take so long for the rankings to recover. The problem is that even after a month and a half, the traffic is still down, and most rankings haven't recovered yet. 

Now we are wondering if we did something wrong in the process, and our only guess is that we messed up by wrongly redirecting the sitemap. We found on the internet that this shouldn't be a problem, but I still want to see your input on this.

So the thing is, we redirected everything from the old domain to the new domain, including the sitemap. So at the moment, there's no sitemap on the old domain with the old links. If you enter olddomain.com/sitemap.xml, it will redirect to the newdomain.com/sitemap.xml that would have all the new links. Is this an actual problem? Should I disable this particular redirect and leave the old sitemap in place? If so, in which property should I link the old sitemap? To the property of the new domain or old domain?

To be noted that there's no error in GSC related to Coverage. On the new domain property, almost all pages have been marked as "Submitted and indexed," while for the old domain, most pages are shown as "Page with redirect" in Coverage. 

Other concerns we have that add more context to our issue:

- This is the strangest thing we have noticed. So before the migration, we had more than 300 ranking keywords, from which 30 keywords were in the top 3. After the migration, only a handful of keywords (like 5-6) have kept their rankings in the top 3, while the rest of the keywords have been thrown out of 100. Even at this moment, those 5-6 pages are still on top of the rankings and generate small traffic, while more than 200-300 keywords that previously had stable rankings are still out of 100, with few keywords bouncing from day to day. We would expect to see a slight and progressive increase in the number of stable keywords, but it looks like there's no big change, and still, most pages look like they have vanished from searches. How is it possible for Google to leave more than 300 pages out of 100 (pages that previously have been ranking pretty well) from a domain that sees it as good enough to rank it in the first position for other keywords? I mean, this whole time, those 5 pages are first on Google, and the rest vanished. This discrepancy is too significant.

- Google is very slow in crawling our website. At the beginning of the migration, most pages on the new domain were marked as "Mobile-friendly pages." However, after a week or so, we have started to see a slight decline in the number of mobile-friendly pages without seeing any increase in errors. So from 300+ pages marked as mobile-friendly, now, after more than a month, they are only 150. Our only guess is that Google is not crawling our website enough to get the necessary data for these metrics. Obviously, all pages are mobile-friendly and load under 1s. (We pass all core web vitals)

- We see strange behavior in GSC > Coverage for the new domain property. That is, we have an increase in the number of excluded pages due to "Page with redirect". This is because Google founds the version of the URLs that end with "/" for some pages, and those redirect to the URLs without "/", hence the issue. We have no idea how Google can find those URLs ending with "/" as we have removed them from sitemap and any other location. The number of affected pages for this particular error hasn't increased for a week or so.

- We are also planning to update our homepage. Should we wait until the domain migration is completely finished?

- Also, during this domain migration, Google has launched one or two updates. Is there any chance those updates can mess up the migration due to a faulty algorithm?

Any help is highly appreciated.

Thanks!
3 upvotes
10 replies
false
5337628125281838947
true
Search Help Center
true
true
true
true
true
102095
Search
Clear search
Close search
Main menu
false