Known Issue: "Error: Could not validate quota limits"

Background

For every new deployment or storage nodes addition operations, the Elastifile Management Server validates the available quotas, for many required resources such as pd-ssd, cpus, memory, etc.

As part of GCP changes to the Local SSD quota, which described in this article, a new issue introduced when Elastifile queries the response from the GCP API. 

 

Issue

GCP Compute API changed the way they return the response, and the Elastifile Management Server unable to parse this new response.

Hence, whenever an add node operation is executed, the following error might be shown and the user could not add the new resources:

"Error: Could not validate quota limits"

In the /elastifile/log/ecp/cloud_provider.log , you should find the following error:

2023-04-14 15:19:13,528 - thrift.server.TServer - ERROR - integer out of range for 'q' format code
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/thrift/server/TServer.py", line 167, in serveClient
    self.processor.process(iprot, oprot)
  File "/elastifile/ecp/app/api/ecp_api/ECPService.py", line 1226, in process
    self._processMap[name](self, seqid, iprot, oprot)
  File "/elastifile/ecp/app/api/ecp_api/ECPService.py", line 1423, in process_quota_get
    result.write(oprot)
  File "/elastifile/ecp/app/api/ecp_api/ECPService.py", line 4811, in write
    self.success.write(oprot)
  File "/elastifile/ecp/app/api/ecp_api/ttypes.py", line 3361, in write
    iter92.write(oprot)
  File "/elastifile/ecp/app/api/ecp_api/ttypes.py", line 1888, in write
    oprot.writeI64(self.limit)
  File "/usr/lib64/python2.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 114, in writeI64
    buff = pack("!q", i64)
error: integer out of range for 'q' format code

 

Solution

New Deployment

The following steps should be taken:

1. SSH to the EMS

user@patch ~$ sudo su -
root@patch ~# . elfs_admin
[root@patch ~(elfs_admin)]#

2. Edit the file named /elastifile/ecp/app/lib/ecp_gce.py and comment out the following lines:

# elif item.get('metric') == 'LOCAL_SSD_TOTAL_GB':
#   quota = dict(metric='local-ssd',
#     usage=int(item.get('usage')),
#     limit=int(item.get('limit')))
#   quotas.append(quota)

3. Restart the ECP service: systemctl restart ecp

4. Complete the new deployment

5. Apply the patch which described in the next section: Adding Nodes on Running System

 

Adding Nodes on Running System

Elastifile offers the following patch that can be installed by the user in order to solve the issue.

For that, follow the steps below:

1. SSH to the EMS

user@patch ~$ sudo su -
root@patch ~# . elfs_admin
[root@patch ~(elfs_admin)]#

2. Download the required patch

gsutil cp gs://elastifile-software-repo/Patches/local-ssd-unlimited-quota/elastifile-upgrade-patch-3.2.1.51-64368.d61e0733322f.tar /tmp/

3. Upload the patch to EMS software repo

elfs-cli sw_package upload --content /tmp/elastifile-upgrade-patch-3.2.1.51-64368.d61e0733322f.tar

4. Install the patch 

elfs-cli system  upgrade_start --id 1 --type patch --admin-passwd <EMS_PASSWORD>

 

Validation

In order to validate that the patch installed successfully and the issue is resolved, run the following command and expect a valid output:

[root@patch ~(elfs_admin)]# ecp-cli quota get
{
    "meta": {
        "msg": []
    },
    "quotas": [
        {
            "limit": 600,
            "metric": "cores",
            "usage": 24
        },
        {
            "limit": 102400,
            "metric": "external-hdd",
            "usage": 300
        },
        {
            "limit": 575,
            "metric": "ip-addresses",
            "usage": 3
        },
        {
            "limit": 40960,
            "metric": "external-ssd",
            "usage": 2820
        },
        {
            "limit": 6000,
            "metric": "instances",
            "usage": 6
        },
        {
            "limit": 4094,
            "metric": "internal-ip-addresses",
            "usage": 8
        },
        {
            "limit": 500,
            "metric": "firewalls",
            "usage": 61
        },
        {
            "limit": 300,
            "metric": "routes",
            "usage": 35
        }
    ]
}