/webmasters/community?hl=en
/webmasters/community?hl=en
1/2/11
Original Poster
cris39maglia

robots.txt remove

I have read the FAQs and checked for similar issues: YES / NO
My site's URL (web address) is: www.cristina-yarnworld.com
Description (including timeline of any changes made):
Happy New Year !!!!!
 
Please help me ! I don`t know what to do, I put for mistake ( I do not speak so good english) a robots.txt on my web site , and now I have this crawl error:
# robots.txt for http://www.cristina-yarnworld.com
 
User-agent: *
Disallow: */shared_files/
Disallow: */contents/
Disallow: */contents/styles/
Disallow: */contents/*/changecurrency.html
Disallow: */contents/*/conf.html
Disallow: */contents/*/customerdtl.html
Disallow: */contents/*/load_*.html
Disallow: */contents/*/login.html
Disallow: */contents/*/orderterms.html
Disallow: */contents/*/ordertotal.html
Disallow: */contents/*/search.html
Disallow: */contents/*/search_result.html
Disallow: */contents/*/shipping_charge.html
Disallow: */contents/*/thankyou.html
Disallow: */contents/*/unsuccessful.html
Disallow: */contents/*/V6*.html
Disallow: */contents/*/website.html
Disallow: */contents/*.js
Allow: */contents/*/*
Sitemap: http://www.cristina-yarnworld.com/sitemap.xml
 
Can someone help me please?2011-01-02
 
URL Detail Detected
http://www.cristina-yarnworld.com/contents/de/load_index1.html URL restricted by robots.txt Dec 29, 2010
http://www.cristina-yarnworld.com/contents/de/load_index2.html URL restricted by robots.txt Dec 29, 2010
http://www.cristina-yarnworld.com/contents/extra.html URL restricted by robots.txt Dec 29, 2
Community content may not be verified or up-to-date. Learn more.
All Replies (53)
Tim Abracadabra
1/2/11
Tim Abracadabra
Hi cris39maglia,

This all seems OK to me.

Your robots.txt seems pretty standard for a CMS based shopping cart site.

You typically want these files restricted as they provide no value to the site or are duplicates of other pages you prefer indexed.

By restricting these pages you tend to optimize the site for "better" indexing
(better is not to be confused with rank) and maybre a better user experience\privacy.

These are not crawl "errors"
The URL's you show restricted by robots.txt are exactly the types you have defined
in the Disallow's in your robots.txt.

Are you saying you do not want them restricted? Why if I may ask?
Have you looked at those pages themselves and determined you want them indexed?

By the way, Is this a robots.txt that comes with your site package or was it recommended?

If you don't want these pages restricted for some reason you can either modify
the existing robots.txt accordingly (remove the Disalow: statements) and Google will update itself the next time robots.txt is re-crawled.

Note: It may take time for this to be reflected in the Google Webmaster tools (A month or so maybe) but rest assured, Google will follow the new crawl directives right away.

And Happy New Year to you too!

Hope that helps,
Tim

1/2/11
Original Poster
cris39maglia
Thank you Tim,
 
I'm not sure exactly what the robots.txt file is. I was researching/experimenting with my website and uploaded this file by mistake. I would like to delete the file to return my site to the configuration it was in before I uploaded the file. How can I remove the Disalow statments? I don`t want restricted URL`s on my web-site beacuse I just open an on-line store and I try to sell some product. I have also a 404 error....
 
Thanks for your help,
 
Cristina
Tim Abracadabra
1/3/11
Tim Abracadabra
Hi Christina.

"I'm not sure exactly what the robots.txt file is."
OK, robots.txt  is a file that resides in the root of your site (Typically where your home page is located)

It typical use is to inform search engines (like Google) what pages to Disallow: Allow: according to pattern matching specifications.

It can also be used to specify the name\location of your sitemap.xml file.

I would suggest you read up on robots.txt as it can be quite useful to help your site index efficiently and somewhat enhance security\privacy. It has worthwhile uses.

If you want you typically can just delete the robots.txt from the root folder\directory of the site's webspace and that will be fine. Just be sure you really don't need\want it first.

See the link below for more information on robots.txt.

But put simply, If Google does not find a robots.txt Googlebot crawls freely as no nothing is Disallowed).

-----------------------------------------
"I have also a 404 error.... "

Hmm, We can look into that for you.

Can you please provide the 404 error details including any date and exactly where you are seeing this error?

Thanks,
Tim



1/3/11
Original Poster
cris39maglia
Tim,
 
I think I mess up everything on my web-site, and I don`t know enymore what should I do. I cannot find the robots.txt in the root folder/directory on my site. How can attach you a 404 error?
Thank you so much for your patience
Tim Abracadabra
1/4/11
Tim Abracadabra
[Edit]

Hi Cristina,

>> "I think I mess up everything on my web-site, and I don`t  know enymore what should I do."

In what way is it "mess up"? Google has indexed at least 53 pages and has crawled and updated the home page on Jan 2, 2011 14:53:32 GMT.


>> "I cannot find the robots.txt in the root folder/directory on my site."

It is there somewhere as I just downloaded it from your website!.
I'm looking at it right now :-)

__________________________________________________________

# robots.txt for http://zj-8437-uq.shopfactory.com/

User-agent: *
Disallow: */shared_files/
Disallow: */contents/
Disallow: */contents/styles/
Disallow: */contents/*/changecurrency.html
Disallow: */contents/*/conf.html
Disallow: */contents/*/customerdtl.html
Disallow: */contents/*/load_*.html
Disallow: */contents/*/login.html
Disallow: */contents/*/orderterms.html
Disallow: */contents/*/ordertotal.html
Disallow: */contents/*/search.html
Disallow: */contents/*/search_result.html
Disallow: */contents/*/shipping_charge.html
Disallow: */contents/*/thankyou.html
Disallow: */contents/*/unsuccessful.html
Disallow: */contents/*/V6*.html
Disallow: */contents/*/website.html
Disallow: */contents/*.js
Allow: */contents/*/*
Sitemap: http://zj-8437-uq.shopfactory.com/sitemap.xml
______________________________________________________________

Anyway, As I have no way of knowing how you work with your website and its
actual structure, I can not tell you how to find it. But it is there somewhere.

Is possibly shopfactory.com is your website package provider?
Maybe check with them.

Here is the link to their support forum:
http://support.shopfactory.com/


>> " How can  attach you a 404 error?"

You don't need to attach anything.

Just copy the error information from where ever you are seeing it and paste
it to your post.

Hope that helps clarify,
Tim
Tim Abracadabra
1/4/11
Tim Abracadabra
Additionally:

Cristina, please note that the robots.txt I see is not exactly
the robots.txt in your original post.

It might be possible the robots.txt upload you attempted
did not do ... what you think it did ??

 ;-)

Anyway, post your 404 error information and details of why you find
your site messed up and we will go from there.

Thanks,
Tim
1/4/11
Original Poster
cris39maglia
Hello Tim,
 
I don`t know what I did, I saw this morning that this robors.txt is not the same. Yesterday I tryed to open another shop, and I think is the robots.txt for that one ( this one does not have www.cristina-yarnworld.com name).
Here is the 404 error
 
http://www.cristina-yarnworld.com/)?%5B%5E/%5D*$ 404 (Not found) 19 pages Dec 29, 2010
http://www.cristina-yarnworld.com/contents/de-ch/index.html 404 (Not found) unavailable
Dec 30, 2010
http://www.cristina-yarnworld.com/contents/de/d15_Gr%FCndl__Wolle.html 404 (Not found) 2 pages Dec 30, 2010
http://www.cristina-yarnworld.com/contents/de/d29.html 404 (Not found) 1 pages Dec 31, 2010
http://www.cristina-yarnworld.com/contents/de/yarn_by_fiber.cfm?info_type=2&info_id=14 404 (Not found) 1 pages Dec 30, 2010
http://www.cristina-yarnworld.com/contents/de/yarn_by_fiber.cfm?info_type=2&info_id=19 404 (Not found) 1 pages Dec 30, 2010
http://www.cristina-yarnworld.com/contents/de/yarn_by_fiber.cfm?info_type=2&info_id=6 404 (Not found) 1 pages Dec 30, 2010
http://www.cristina-yarnworld.com/contents/de/yarn_by_gauge.cfm?info_id=12 404 (Not found) 1 pages Dec 30, 2010
http://www.cristina-yarnworld.com/contents/de/yarn_by_info.cfm?info_type=1&info_id=50 404 (Not found) 1 pages Dec 30, 2010
http://www.cristina-yarnworld.com/contents/de/yarn_by_info.cfm?info_type=3&info_id=27 404 (Not found) 1 pages Dec 30, 2010
http://www.cristina-yarnworld.com/contents/de/yarn_by_info.cfm?info_type=4&info_id=58 404 (Not found) 1 pages Dec 30, 2010
http://www.cristina-yarnworld.com/contents/en-us/d15_Gr%FCndl__Wolle.html 404 (Not found) 2 pages Dec 30, 2010
http://www.cristina-yarnworld.com/contents/en-us/d29.html 404 (Not found) 1 pages Jan 1, 2011
http://www.cristina-yarnworld.com/contents/en-us/yarn_by_fiber.cfm?info_type=2&info_id=14 404 (Not found) 1 pages Dec 30, 2010
http://www.cristina-yarnworld.com/contents/en-us/yarn_by_fiber.cfm?info_type=2&info_id=19 404 (Not found) 1 pages Dec 30, 2010
http://www.cristina-yarnworld.com/contents/en-us/yarn_by_fiber.cfm?info_type=2&info_id=6 404 (Not found) 1 pages Dec 30, 2010
http://www.cristina-yarnworld.com/contents/en-us/yarn_by_gauge.cfm?info_id=12 404 (Not found) 1 pages Dec 30, 2010
http://www.cristina-yarnworld.com/contents/en-us/yarn_by_info.cfm?info_type=1&info_id=50 404 (Not found) 1 pages Dec 30, 2010
http://www.cristina-yarnworld.com/contents/en-us/yarn_by_info.cfm?info_type=3&info_id=27 404 (Not found) 1 pages Dec 30, 2010
http://www.cristina-yarnworld.com/contents/en-us/yarn_by_info.cfm?info_type=4&info_id=58 404 (Not found) 1 pages Dec 30, 2010
http://www.cristina-yarnworld.com/de-ch/ 404 (Not found) 4 pages Dec 30, 2010
 
 
 
THANK you so much Tim !!!!!!!!!!
1/4/11
Original Poster
cris39maglia
Show URLs: In Sitemaps ‎(2)‎ Not found ‎(21)‎ Restricted by robots.txt ‎(5)‎
URL Detail Linked From Detected
http://www.cristina-yarnworld.com/contents/de/d15_Gr%FCndl__Wolle.html 404 (Not found) 2 pages Dec 30, 2010
http://www.cristina-yarnworld.com/contents/en-us/d15_Gr%FCndl__Wolle.html 404 (Not found) 2 pages Dec 30, 2010
1/4/11
Original Poster
cris39maglia
Tim Abracadabra
1/4/11
Tim Abracadabra
Hi Cristina,

I looked at what you posted, thanks for providing that information!

I do not believe you currently have anything to be worried about.

Here is some detail for the three groups of errors you posted:

--------------------------------------------------------------------------
These are not errors, They just indicate your robots.txt is working as it should.
Show URLs: robots.txt ‎(5)

URL Detail Detected
http://www.cristina-yarnworld.com/contents/de/load_index1.html  URL restricted by robots.txt Dec 29, 2010 <OK>

http://www.cristina-yarnworld.com/contents/de/load_index2.html  URL restricted by robots.txt Dec 29, 2010  <OK>

http://www.cristina-yarnworld.com/contents/en-us/load_index1.html  URL restricted by robots.txt Dec 30, 2010  <OK>

http://www.cristina-yarnworld.com/contents/en-us/load_index2.html  URL restricted by robots.txt Dec 30, 2010  <OK>

http://www.cristina-yarnworld.com/contents/extra.html <OK>

Likely these URL references are junk generated by your CMS/Shopping cart back end.
-----------------------------------------------------------------

Show URLs: In Sitemaps ‎(2)‎

These errors seem to have come crawls of URLs found in your Sitemap file.

http://www.cristina-yarnworld.com/contents/de/d15_Gr%FCndl__Wolle.html  404 (Not found) 2 pages Dec 30, 2010

http://www.cristina-yarnworld.com/contents/en-us/d15_Gr%FCndl__Wolle.html  404 (Not found) 2 pages Dec 30, 2010

This means your Sitemap was wrong\ not consistant with the current website.
(Your site may have changed since the Sitemap was last uploaded\referenced)

This is generally a warning only, will not affect Google ranking. May indicate a problem with Sitemap generation. If it continues contact shopfactory who seem to be generating the Sitemap.
-----------------------------------------------------------------------------

Not found ‎404 (21) These are just outright 404's

Yes, These pages do return a 404\does not exist for me too.

I won't re-list the detail here for brevity but it seems that some work was being done on the site around the end of the month possibly with adding pages for different languages and
somehow Google was made aware of\tried to crawl these pages and it could not find them.

I can't say what was being done but you probably know.

See the date on these 404 errors? Dec 29, 2010,  Dec 30, 2010.

Keep and eye on those dates. If they increment like  Jan 05, 2011, etc..
Then you might want to look for the cause.
Otherwise these were a one time occurrence and should disappear from the error information within a month or two.

Bottom Line:

I see nothing here that will cause any problem with Google indexing and ranking your site.

Further, if these errors continue to occur (With incrementing dates) then
I would suggest opening up a support ticket with your CMS\Shopping cart provider shopfactory.com and have them explain why their software is generating these URL's or faulty\untimely Sitemaps and if they will fix it.

Hope that helps and all the best,
Tim

42 MORE
1/11/11
Original Poster
cris39maglia
Hello Tim,
 
Finaly I receive the respons for Sf. Here it is:"Yes, we do support mod_rewrite. And if you are running ShopFactory it’s already done for you, if you are doing custom pages or similar it should just work as per any of the instructions you can find on the internet."
Were these replies helpful?
How can we improve them?
 
This question is locked and replying has been disabled. Still have questions? Ask the Help Community.

Badges

Some community members might have badges that indicate their identity or level of participation in a community.

 
Expert - Google Employee — Googler guides and community managers
 
Expert - Community Specialist — Google partners who share their expertise
 
Expert - Gold — Trusted members who are knowledgeable and active contributors
 
Expert - Platinum — Seasoned members who contribute beyond providing help through mentoring, creating content, and more
 
Expert - Alumni — Past members who are no longer active, but were previously recognized for their helpfulness
 
Expert - Silver — New members who are developing their product knowledge
Community content may not be verified or up-to-date. Learn more.

Levels

Member levels indicate a user's level of participation in a forum. The greater the participation, the higher the level. Everyone starts at level 1 and can rise to level 10. These activities can increase your level in a forum:

  • Post an answer.
  • Having your answer selected as the best answer.
  • Having your post rated as helpful.
  • Vote up a post.
  • Correctly mark a topic or post as abuse.

Having a post marked and removed as abuse will slow a user's advance in levels.

View profile in forum?

To view this member's profile, you need to leave the current Help page.

Report abuse in forum?

This comment originated in the Google Product Forum. To report abuse, you need to leave the current Help page.

Reply in forum?

This comment originated in the Google Product Forum. To reply, you need to leave the current Help page.