Skip to content

Object Storage

Object Storage offers a scalable storage service which can be used by applications compatible with the Amazon S3 protocol.

The CERN Object Store is built upon Ceph and operated by the IT Storage Group.

The Ceph Service makes no additional backups and there is no provision within the service for disaster recovery. Users are therefore responsible for maintaining independent backups of their objects where they judge it important.

Quota

Access to the Object Storage service requires an explicit quota request for shared projects. Please use the "Request quota change" button on the Horizon web interface to request resources. Note that quota will not be granted for "Personal" OpenStack tenants.

Space and Quota utilization

It is possible to get information about space and quota utilization for the configured project using the /?usage=true query parameter on the S3 endpoint. This will return an XML document with usage information for the project associated to the provided credentials.

Using curl

The following curl command will return an XML document with usage information for the project associated to the provided credentials with the s3.cern.ch S3 endpoint:

Note

The only parts that need to be changed in the command below are the credentials (setting $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY) and the S3 base endpoint URL (s3.cern.ch) if you want to check the usage for a different project or S3 endpoint, the rest of the command should be left unchanged.

curl --aws-sigv4 "aws:amz:cern-geneva:s3" --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" -H "x-amz-content-sha256: UNSIGNED-PAYLOAD" "https://s3.cern.ch/?usage=true"

Output will be an XML document with the following structure:

<?xml version="1.0" encoding="UTF-8"?>
<Usage>
  <Entries></Entries>
  <Summary>
    <QuotaMaxBytes>60473139527680</QuotaMaxBytes>
    <QuotaMaxBuckets>1000</QuotaMaxBuckets>
    <QuotaMaxObjCount>-1</QuotaMaxObjCount>
    <QuotaMaxBytesPerBucket>-1</QuotaMaxBytesPerBucket>
    <QuotaMaxObjCountPerBucket>-1</QuotaMaxObjCountPerBucket>
    <TotalBytes>5082583315976</TotalBytes>
    <TotalBytesRounded>5083388264448</TotalBytesRounded>
    <TotalEntries>376617</TotalEntries>
  </Summary>
  <CapacityUsed>
    <User>
      <Buckets>
        <Entry>
          <Bucket>test-bucket</Bucket>
          <Bytes>735</Bytes>
          <Bytes_Rounded>4096</Bytes_Rounded>
        </Entry>
        ...
      </Buckets>
    </User>
  </CapacityUsed>
</Usage>

Using the provided cephs3usage binary

We provide here a binary program (cephs3usage) to retrieve usage information in a more user-friendly manner. The program is available for Linux and MacOS, and architectures amd64 and arm64. The usage is straightforward, just provide the S3 endpoint URL as an argument (which defaults to https://s3.cern.ch if not provided):

./cephs3usage_linux_amd64 -endpoint https://s3.cern.ch

cephs3usage assumes credentials to be configured in any way that awscli expects, usually ~/.aws/credentials. In case multiple profiles are configured, it is possible to use environment variables (as supported by AWS CLI Configuration) to pick the desired one:

AWS_PROFILE=cern_s3 ./cephs3usage_linux_amd64 -endpoint https://s3.cern.ch -json

The output will be a JSON/XML document with the following structure:

{
  "QuotaMaxBytes": 60473139527680,
  "QuotaMaxBuckets": 1000,
  "QuotaMaxObjCount": -1,
  "QuotaMaxBytesPerBucket": -1,
  "QuotaMaxObjCountPerBucket": -1,
  "TotalBytes": 5082583315976,
  "TotalBytesRounded": 5083388264448,
  "TotalEntries": 376617,
  "Buckets": [
    {
      "Bucket": "test-bucket",
      "Bytes": 735,
      "Bytes_Rounded": 4096
    },
    ...
  ]
}

Advanced

More advanced users can also use the /?usage=true query parameter directly with their S3-compatible application, as long as it supports custom query parameters. The structure of the returned XML document is described in the Ceph documentation.

Credentials

Credential activation time

Please note that although the generation of the credentials is instantaneous, it can take up to 15 minutes for the key to be propagated; you will be denied access in the meantime.

After quota has been granted, S3 credentials can be created via the openstack CLI.

First, configure the openstack environment using the appropriate openstack project: 

unset OS_PROJECT_ID;
unset OS_TENANT_ID;
unset OS_TENANT_NAME;
export OS_PROJECT_NAME="PUT PROJECT HERE";

Make sure that your environment is configured for the correct region, i.e. cern or pdc.

export OS_REGION_NAME="cern" # either "cern" or "pdc";

Next, create and view the keys: 

S3_HOST=$(openstack catalog show s3 -f json -c endpoints | jq --raw-output ".endpoints[] | select(.interface==\"public\") | select(.region_id==\"$OS_REGION_NAME\") | .url" | cut -f3 -d/)
ACCESS_KEY=$(openstack ec2 credentials create -f value -c access)
SECRET_KEY=$(openstack ec2 credentials show $ACCESS_KEY -f value -c secret)

echo $S3_HOST
echo $ACCESS_KEY
echo $SECRET_KEY

Use the above host and keys in your S3-compatible application.

As mentioned above, the propagation of your credentials can take up to 15 minutes.

Using Object Storage

Here are some examples using object storage from the command line:

Privacy Policy

Object Storage is provided by Ceph, whose Privacy Policy is available here.