Oct 16, 2023
Issues related to website crawling and indexing
Informational notification.
This question is locked and replying has been disabled.
Community content may not be verified or up-to-date. Learn more.
Oct 18, 2023
These structures have not been designed yet. I'm worried that Google spiders will get lost coming into my site. Or get a bad impression of my site structure. May I ask what I shouldI do not know what's holding you from completing the design. I just wanted to tell you to complete it. It's a brand new website, so what bad impression can Googlebot take? If you block pages, that can also be a bad signal- i.e., you are hiding things from the Googlebot.
Last edited Oct 18, 2023
Original Poster Mason5666 marked this as an answer
Oct 18, 2023
Problem 1: Some of the pages that were there when Google crawler came to crawl last time, disappeared when Google crawler came to crawl next time (because these pages were deleted by me). Will this have a negative impact on my SEO?
These pages should return 404 Not Founf or 410 Gone status. Google will drop those pages from index over a certain period of time.
Question 2: While the site is still in the process of being built. Should I turn off the ability for the site to be found and indexed?
You should not have made the site live and available to search engine before building the minimum prototype. Now blocking Googlebot in the middle can have detrimental effects. Blocking Googlebot for new sites is not a good idea. You can apply noindex or even better you just ignore (as I mentioned in my previous response) until the site is minimum prototype. Google will not index all pages even if you allow it do so.
For the time being, you can ignore "Discovered-currently not indexed" and "Crawled-currently not indexed". In short, you should put your full effort on making the site "ready".
Original Poster Mason5666 marked this as an answer
Oct 16, 2023
Original Poster Mason5666 marked this as an answer
All Replies (3)
Oct 16, 2023
To fix the crawling and indexing error on your website, you should:
- Identify the technical problem: This may involve using a tool like Google Search Console to check for errors in your website's robots.txt file or sitemap.xml file. You may also want to use a crawling tool to see how search engine bots are crawling and indexing your website.
- Fix the technical problem: Once you have identified the technical problem, you should fix it as soon as possible. This may involve updating your robots.txt file, submitting a new sitemap.xml file, or fixing any other technical issues on your website.
- Request a recrawl: Once you have fixed the technical problem, you can request a recrawl of your website in Google Search Console. This will help Google to update its index of your website and fix any crawling or indexing errors.
Here are some additional tips for improving your website's crawlability and indexability:
- Make sure that your website has a clear and concise site structure.
- Use descriptive and keyword-rich titles and meta descriptions for your pages.
- Use internal links to connect your pages together.
- Avoid using duplicate content on your website.
- Make sure that your website is mobile-friendly.
By following these tips, you can help to ensure that your website is crawled and indexed correctly by search engines.
It is important to note that it may take some time for Google to recrawl and index your website after you have fixed the technical problem. In the meantime, you can continue to work on improving your website's content and structure.
Now there is an error in the crawling and indexing process due to a technical problem.
- What kind of error you saw? Can you provide the screenshot?
Original Poster Mason5666 marked this as an answer
Original Poster Mason5666 marked this as an answer
Oct 18, 2023
Oct 19, 2023
@Izaz Arefin @Ujjawal Poonia
OK mate, I've got your point. Keeping my site open to google crawlers now. And try to finish my site. Thanks for your reply!