Introduction
GGC Batch Repairs is a system for ISP Partners to self-manage selected repairs for the GGC nodes they host.
The differences between the way GGC repairs have traditionally been managed, and the self-management enabled by GGC Batch Repairs are described below.
The GGC Batch Repairs system is in a restricted Beta testing phase.
How GGC repairs have been managed traditionally
- An internal ticket is opened with GGC Support for every fault detected on a GGC machine.
- GGC Support attempts to fix the fault remotely.
- If GGC Support cannot fix the fault remotely:
- The ISP Partner may be asked to assist with a visual inspection.
- The ISP Partner may be asked to help with diagnostics.
- If the fault remains unfixed:
- GGC Support opens a vendor Service Repair ticket.
- GGC Support inform the ISP Partner that a vendor Service Repair is required.
- GGC Support may ask the ISP Partner to confirm or update your contact information for coordination of
a vendor site-visit, or the receipt of parts. - The vendor will contact the ISP Partner directly to arrange a schedule if a site-visit is required.
- GGC support will receive repair-status updates from the vendor.
- GGC Support will not contact the ISP Partner again about the fault unless a problem arises that needs their
assistance. - When the fault is confirmed fixed, GGC Support closes all relevant tickets.
How GGC Batch Repairs are managed
- The ISP Partner will not receive an email from GGC Support for every fault that cannot be remotely fixed.
- GGC Support attempts to fix the fault remotely.
- If GGC Support cannot fix the fault remotely:
- The Fault will be triaged by Google, until sufficient Faults have accumulated in a metro that our service is vulnerable
- A single Google Global Cache Batch Repair ticket will be sent to the ISP that will provide an interface to interact with all individual Faults
- When the ISP accepts this Batch Repair ticket the ISP Partner will be able to choose which faults are most important to the ISP Partner to repair.
- The ISP Partner will have the option to choose multiple faults to repair as a batch if that is most convenient.
- Shipping information for replacement parts is provided via the ticketing system. Shipping information is not emailed to the ISP Partner.
- Escalations for individual Faults are handled inline via the Google Global Cache Batch Repair ticket (see screenshots).
- The Google Global Cache Batch Repair ticket will resolve automatically when all of the Faults it is tracking are resolved.
What types of fault will be presented for self-managed repairs?
- At this time disk failures and power supply failures are the only type of fault for GGC Batch Repairs
- As the self-managed repairs system is proved, more fault types will gradually be added to it.
- Google has not yet set a timeline for moving more faults to the self-managed repairs system.
- Google hope eventually to handle most faults in the new system.
How will faults be aggregated?
- Initially, all faults for one ISP-name in one airport-code location will be displayed together in one GGC Batch Repairs ticket.
- As an example, "HugeISP" has 3 GGC nodes near Denver (DEN), and 3 more near Salt Lake City (SLC).
- All current faults for the hugeisp-den1, hugeisp-den2 and hugeisp-den3 GGC nodes will be shown in one Google Global Cache Batch Repair ticket.
- All current faults for the hugeisp-slc1, hugeisp-slc2 and hugeisp-slc3 GGC nodes will be shown in another Google Global Cache Batch Repair ticket.
- In future, current faults for all GGC nodes "HugeISP" host may be displayed in one Google Global Cache Batch Repair ticket, if this development proves practical.
- We are also considering creating GGC Batch Repair tickets based on GNL.
The self-managed repairs process
- When sufficient Faults have accumulated, GGC Support will send the ISP Partner an email notifying them that a new Google Global Cache Batch Repair ticket has been created. The email will include a link to the ticket.
- The Google Global Cache Batch Repair ticket shows a list of current faults, showing the fault type, and the repair’s Current Status.
- The ISP Partner can choose any number of faults from the list, depending on what will be most convenient for them.
- Vendor Service Repair tickets will immediately be opened for the faults the ISP Partner selected.
- The Current Status of the repair will update.
- The ISP Partner may receive an email asking them to confirm or update the shipping contact information for the GGC node, if the information has not been recently verified.
- GGC support will receive repair-status updates from the vendor.
- The vendor will contact the ISP Partner directly if it is necessary to schedule a site-visit.
- GGC Support will not contact the ISP Partner about the repair unless a problem arises that needs their assistance.
- The Google Global Cache Batch Repair ticket is a snapshot of all relevant faults at the time it is opened. Faults that are created after the Google Global Cache Batch Repair ticket is created are not automatically appended. The expectation is that they will be part of a future batch ticket.
Fault notifications or status updates from ISP Partners
- If an ISP Partner notices a fault on a GGC machine, should they ignore it?
- Yes. Google’s monitoring systems will detect the fault and deal with it.
- Can a GGC Partner open a Google Global Cache Batch Repairs ticket?
- Not presently, although this is being considered as an configurable option in the ISP Portal (https://isp.google.com/).
- Should an ISP Partner try to make fault-related inquiries or updates in the Google Global Cache Batch Repair ticket?
- Any updates or inquiries should be made via the escalation button specific to the Fault. This escalation mechanism is accessed through the Google Global Cache Batch Repair interface, but it is not an email reply to the Google Global Cache Batch Repair ticket itself.
Screen-shots
The following screen-shots were taken from Google test and development systems. They are representative of what an ISP Partner will see, but not identical.
GGC Support will send the ISP Partner an email stating that Faults have been triaged and must be repaired. The email will include a URL for the relevant Google Global Cache Batch Repair ticket.
The Google Global Cache Repair Trip ticket will show a list of current faults. The ISP Partner can choose which faults they want to repair by clicking the check-boxes at the left of the fault description.
The ISP Partner should select all the faults they want to repair immediately, then click the “PROCEED” button.
The ISP Partner may be emailed asking them to confirm address information. This must be contact information for
someone who is aware that repairs to the GGC node are required, and is able to receive the necessary spare parts.
When the contact details are correct, they should click the “CONFIRM ADDRESS(ES)” button.
The Google Global Cache Batch Repair display will change to show a list of the repairs in progress. Note that in the example ticket, the“Current Status” has changed to “Replacement disk ordered”.