/programmable-search/community?hl=en
This content is likely not relevant anymore. Try searching or browse recent questions.
CSE Domain Limit Reduced from 5,000 to 20
0
Summarizing findings to date. Google has reduced the domain limit on their Custom Search Engine product from 5,000 to 20. Engines that index more than 20 effectively become non-functional.

Some legacy engines with >20 domains remain unaffected but it’s unclear for how long. Open questions to Google:
  • When was this change originally implemented?
  • How long will older engines remain grandfathered in under the old rules?
  • Will changes to an older engine’s annotation or context file trigger the new restriction?
This support page updated on March 16, 2019 explains: “If you happen to add more than 20 unique sites, your CSE may sometimes display fewer results.”


Google has confirmed that “fewer” actually means 0-20 total results, as demonstrated below with the search term "test" submitted to each engine:
Test engine B includes the same 20 sites as A plus 1. Test engine D includes the same 20 sites as C plus 1. Exact result counts vary.
Details
Pinned
Locked
Latest Update Latest Updates (0)
Relevant Answer Relevant Answers (0)
All Replies (7)
Relevant Answer
Hi Rick,

Q1) When was this change originally implemented? How long will older engines remain grandfathered in under the old rules? Will changes to an older engine’s annotation or context file trigger the new restriction?

A1) Previously, this behavior i.e if you add more than 10 unique sites, your CSE engine sometimes displays fewer results did exists. Currently, this number has been increased to 20 unique sites. This behavior is observed across all CSE engine's. There is no changes to annotations or context files. You can still add more than 20 sites to your CSE "Sites to search" configuration just that you may observe your CSE engine (more than 20 sites) will sometimes display fewer results.

Q2) Google has confirmed that “fewer” actually means 0-20 total results, as demonstrated below with the search term "test" submitted to each engine:
Test engine B includes the same 20 sites as A plus 1. Test engine D includes the same 20 sites as C plus 1. Exact result counts vary.

A2) i) Test Engine A (20 sites) and C (20 sites) are serving results as expected.

ii) Test Engine B (21 sites), accessed the "Public URL" page and performed search with the term "atlantic" and got "About 680,000,000 results" served and when further navigated the results pagination to page 3, it gave me "About 21 results". Screenshot attached (Rick-serving and Rick-serving-1).

This behavior is observed because the result number in CSE results that shows up initially is just an estimated number. This estimate may not be accurate. This is done to serve the results faster. As you navigate to next results set you will start seeing the exact result number. 

Then performed search with the term "test" and it served 4 results. Screenshot attached (Rick-serving-2). This is serving results as expected and you are seeing the behavior as documented i.e due to more than 20 sites:

iii) Same as ii) is observed with your Test Engine D (21 sites). Performed search with the term "college" and got "About 4,970,000,000 results" served and when further navigated the results pagination to page 3, it gave me "About 23 results". Screenshot attached (Rick-serving-3 and Rick-serving-4). Same reasons as explained in ii) above.

Then performed search with the term "jack" and it served 10 results. Screenshot attached (Rick-serving-5). Same reasons as explained in ii) above.

To sump-up, you are seeing these two behavior's i) the result number in CSE results that shows up initially is just an estimated number. This estimate may not be accurate. This is done to serve the results faster. As you navigate to next results set you will start seeing the exact result number and ii) If you happen to add more than 20 unique sites, your CSE may sometimes display fewer results.

Thanks
marked this as an answer
Relevant Answer
Hi Rick,

Screenshots attached.

Thanks
marked this as an answer
Relevant Answer
Hi Rick,

Screenshots attached.

Thanks
marked this as an answer
Relevant Answer
Hi Deepak,

Thank you very much for the additional information. Your screenshots accurately reflect the behavior of Google CSE breaking after the 21st site is added to the index. Two follow-up questions...

We see that some engines with >20 sites continue to work perfectly (return accurate result count estimates and 100 high quality results), while others break in the fashion shown above. Do you have any insight into why some work and others don't? Is it just that Google switches to a more limited search index that excludes long tail results measured by PageRank or is there some other handicap being put in place once the 20 site threshold is crossed?

I guess the critical question that I'm trying to answer is: For large scale CSEs that include thousands of domains that work perfectly today, is there a risk that they will break (stop returning 100 legitimate results) in the near future as a result of unannounced platform changes?
marked this as an answer
Relevant Answer
Hi,

Q1) Do you have any insight into why some work and others don't?
A1) Thats why its mentioned "If you happen to add more than 20 unique sites, your CSE may sometimes display fewer results.".

Q2) For large scale CSEs that include thousands of domains that work perfectly today, is there a risk that they will break (stop returning 100 legitimate results) in the near future as a result of unannounced platform changes?
A2) Currently, there is no changes to this. Will not able to comment about future.

Thanks
marked this as an answer
Relevant Answer
What ? Google initially estimated results to be above half a billion and than when navigated some pages comes to real calculation of around 20 results ? Google applies estimation when results are way more to be calculated accurately not when results are way less (around 20) than it does not needs to apply those estimation stuff. The real issue here is what the OP has stated that when more than 20 domains are added to CSE, it does not function correctly & from correctly here it means showing way less results even though way more results are actually in their for search term in those sites, this is not expected behaviour rather this is faulty behaviour if the google has put in the virtual limit (by saying results may not be shown) of 20 domains than it should atleast gave a warning which it doesn't. Hope someone having accurate information from the google team in this context address the concern. Till than Google CSE with more than 20 sites is fu*ked up.
marked this as an answer
Relevant Answer
Moreover if their is now virtual limit of 20 domains than atleast it should show results from first 20 sites as added to CSE but it does not show rather just show few results from 1/2/3 sites.
marked this as an answer
This question is locked and replying has been disabled.
Discard post? You will lose what you have written so far.
Write a reply
10 characters required
Failed to attach file, click here to try again.
Discard post?
You will lose what you have written so far.
Personal information found

We found the following personal information in your message:

This information will be visible to anyone who visits or subscribes to notifications for this post. Are you sure you want to continue?

A problem occurred. Please try again.
Create Reply
Edit Reply
This will remove the reply from the Answers section.
Notifications are off
Your notifications are currently off and you won't receive subscription updates. To turn them on, go to Notifications preferences on your Profile page.
Report abuse
Google takes abuse of its services very seriously. We're committed to dealing with such abuse according to the laws in your country of residence. When you submit a report, we'll investigate it and take the appropriate action. We'll get back to you only if we require additional details or have more information to share.

Go to the Legal Help page to request content changes for legal reasons.

Reported post for abuse
Unable to send report.
Report post
What type of post are you reporting?
Google takes abuse of its services very seriously. We're committed to dealing with such abuse according to the laws in your country of residence. When you submit a report, we'll investigate it and take the appropriate action. We'll get back to you only if we require additional details or have more information to share.

Go to the Legal Help page to request content changes for legal reasons.

Reported post for abuse
Unable to send report.
This reply is no longer available.
/programmable-search/threads
//accounts.google.com/ServiceLogin
You'll receive email notifications for new posts at
Unable to delete question.
Unable to update vote.
Unable to update subscription.
You have been unsubscribed
Deleted
Unable to delete reply.
Removed from Answers
Removed from Updates
Marked as Recommended Answer
Marked as Update
Removed recommendation
Undo
Unable to update reply.
Unable to update vote.
Thank you. Your response was recorded.
Unable to undo vote.
Thank you. This reply will now display in the answers section.
Link copied
Locked
Unlocked
Unable to lock
Unable to unlock
Pinned
Unpinned
Unable to pin
Unable to unpin
Marked
Unmarked
Unable to mark
Reported as off topic
Known Issue
Fixed
Marked Fixed
Unmarked Fixed
Unable to mark fixed
Unable to unmark fixed
/profile/0
false
Search
Clear search
Close search
Google apps
Main menu
Search Help Center
true
0
false