Load Balancing

Load balancing is a mechanism to distribute workloads between several (backend) systems, typically servers or applications, with the goal to make the offered service more performant and scalable.

Load Balancing as a Service (LBaaS) described here offers IP based load balancing - as an alternative to DNS based load balancing -, and the documentation will cover how to create, configure, and manage a load balancer in the CERN cloud.

The upstream documentation is available for additional details.

Service Load Balancing vs Service Availability

Load-balancing can help to increase service availability, but it is not by itself a mechanism to build highly available applications. For instance, the LBaaS offering described here relies on a single load balancer instance (a virtual machine) through which traffic is routed. The unavailability of this instance will render the backend service inaccessible.

Access to Load Balancers

All shared projects have access to the service but have a default quota of 0.

There is no quota for personal projects for load balancing.

Quota for shared projects can be requested through the standard quota update request form (in the LoadBalancer section).

Concepts

Load Balancer: load balancer instance occupies a neutron network port and has an IP address assigned from a subnet.
Listener: load balancers can listen for requests on multiple ports. Each one of those ports is specified by a listener.
Pool: a pool holds a list of members that serve content through the load balancer. Pools are attached to listeners.
Member: a load balancer backend, member of a pool.
Health monitor: the health monitor keeps track of healthy members in a pool.
Amphora: a virtual machine running HAProxy. This is what does the actual load balancing, and each load balancer gets one.

                                    +---------------+
                                    |               |
                                    | Load Balancer |
                                    | 137.138.6.18  |
                                    |               |
                                    +-------+-------+
                                            |
                              +-------------+--------------+
                              |                            |
                              |                            |
                       +------v-------+           +--------v-------+
                       |              |           |                |
                       |   Listener   |           |    Listener    |
                       | Port 80/HTTP |           | Port 443/HTTPS |
                       |              |           |                |
                       +------+-------+           +--------+-------+
                              |                            |
                              |                            |
                              |                            |
+-----------------+       +---v----+                   +---v----+      +-----------------+
|                 |       |        |                   |        |      |                 |
|  Health Monitor +-------+ Pool 1 |                   | Pool 2 +------+  Health Monitor |
|                 |       |        |                   |        |      |                 |
+-----------------+       +---+----+                   +---+----+      +-----------------+
                              |                            |
                              |                            |
                              |                            |
  +---------------------------v----------+       +---------v----------------------------+
  |                                      |       |                                      |
  | +--------+ +--------+     +--------+ |       | +--------+ +--------+     +--------+ |
  | |Member 1| |Member 2| --- |Member N| |       | |Member 1| |Member 2| --- |Member N| |
  | +--------+ +--------+     +--------+ |       | +--------+ +--------+     +--------+ |
  |                                      |       |                                      |
  +--------------------------------------+       +--------------------------------------+

In the example below we use:

mylb as the load balancer name
mylistener as the listener name
mypool as mylistener's default pool
myhealthmonitor as mypool's health monitor
137.138.53.95 and 188.185.80.141 as the IPs of the backends (members)

Examples

Basic HTTP Load Balancer

The network id should always be CERN_NETWORK.

In this case we're setting up a basic HTTP based load balancer on port 80.

openstack loadbalancer create --name mylb --vip-network-id CERN_NETWORK

After creating a loadbalancer instance, we will test the network reachability by sending ICMP ping requests. In worst case, loadbalancer instance will respond 60 seconds after creation.

ping <loadbalancer-virtual-ip>

Next we create a loadbalancer listener for protocol HTTP and port 80.

openstack loadbalancer listener create --name mylistener \
                                       --protocol HTTP \
                                       --protocol-port 80 mylb

Next we create a pool to host the members, specifying the load balancing algorithm as ROUND_ROBIN. Supported options for load balancing algorithm are ROUND_ROBIN, SOURCE_IP, SOURCE_IP_PORT and LEAST_CONNECTIONS.

openstack loadbalancer pool create --name mypool  \
                                   --lb-algorithm ROUND_ROBIN \
                                   --listener mylistener \
                                   --protocol HTTP

Next we add the loadbalancer members. The port is the port the backend is listening on, which may be different from the port of the listener above.

openstack loadbalancer member create --name server-1 --address 137.138.53.95 --protocol-port 80 mypool
openstack loadbalancer member create --name server-2 --address 188.185.80.141 --protocol-port 80 mypool

Next, we create an HTTP type health monitor, our back-end servers have been configured with a health check at the URL path /healthcheck. Supported types for health monitors are HTTP, HTTPS, and TCP. In the case of a TCP type health monitor, TCP service port for backend servers is periodically probed. Please note that the health monitor is an optional resource.

openstack loadbalancer healthmonitor create --name http-monitor \
                                            --delay 7 \
                                            --timeout 5 \
                                            --max-retries 3 \
                                            --url-path /healthcheck \
                                            --expected-codes 200,201 \
                                            --type HTTP mypool

TCP type healthmonitor can be created by using the following command.

openstack loadbalancer healthmonitor create --name tcp-monitor \
                                            --delay 7 \
                                            --max-retries 3 \
                                            --timeout 5 \
                                            --type TCP mypool

Finally, we can verify our loadbalancer by sending requests to virtual IP.

curl http://<loadbalancer-virtual-IP>

Some response sent by backend

TCP Load Balancer

This is generally suitable when load balancing a non-HTTP TCP-based service. The following example creates a load balancer for ssh connections. Note that the load balancer listens for ssh connections on port 5555 and backend servers are using port 22 (you can use other ports as appropriate).

openstack loadbalancer create --name lb --vip-network-id CERN_NETWORK
openstack loadbalancer listener create --name tcp-listener --protocol TCP --protocol-port 5555 lb
openstack loadbalancer pool create --name tcp-pool --lb-algorithm ROUND_ROBIN --listener tcp-listener --protocol TCP --session-persistence type=SOURCE_IP
openstack loadbalancer member create --name server-1 --address 137.138.53.95 --protocol-port 22 tcp-pool
openstack loadbalancer member create --name server-2 --address 188.185.80.141 --protocol-port 22 tcp-pool

#137.138.6.16 is load balancer's IP address
ssh root@137.138.6.16 -p 5555
Last login: Mon May 25 10:55:56 2020 from lbaas-69e19c65-6d30-48f6-a3e3-04ffe7442a54.cern.ch
[root@delete-me ~]# hostname -i 
188.185.80.141

UDP Load Balancer

UDP load balancing is currently experimental.

TLS termination

This section gives an example of how to create a load balancer to serve TLS terminated traffic. By using this setup, the load balancer instance becomes responsible for doing the HTTPS encryption, such that your back ends are free to operate with HTTP only. This way there is only one place one needs to manage certificates and their rotation.

For additional details, please consult the upstream cookbook.

1: Create the load balancer

Pick a name for your LB and associated resources

# To adapt for your setup
export MYAPP="my-lb-app"  # This results in an LB for my-lb-app.cern.ch

Create the load balancer instance:

openstack loadbalancer create --name ${MYAPP}-tls-term-lb --vip-network-id CERN_NETWORK --tag "landb-alias=${MYAPP}" --wait

2: Prepare the certificate

Alt. A: Using CERN Host certificates

This option is the easiest if the service is exposed to CERN-users only.

After creating the load balancer, wait for the alias to be registered on LanDB.
Go to the CERN Certificate Authority and request a new host certificate for MYAPP (without certificate password)

Alt. B: Using CERN commercial certificates

It is also possible to use commercial certificates managed through CERN, such as those provided by Sectigo. In this case, you can proceed to the next step using the provided p12 or .pfx file.

Alt. C: Using Let's Encrypt

Alternatively, certificates accepted by internal and external users can be obtained for free from Let's Encrypt using HTTP-01 challenges.

This method, however, requires a few extra step:

Create a VM that can runcertbot and answer certificate challenges:

# This can be a small VM, both m2 and m4 flavor will work. Note down its IP for later.
openstack server create --flavor m2.small --key-name <your_ssh_key> --image 'ALMA10 - x86_64' ${MYAPP}-le-certnode

While it is starting up, configure the load balancer for Lets Encrypt:

openstack loadbalancer listener create --name ${MYAPP}-listener-http --protocol HTTP --protocol-port 80 ${MYAPP}-tls-term-lb --wait
openstack loadbalancer pool create --name ${MYAPP}-letsencrypt-pool1 --lb-algorithm ROUND_ROBIN --listener ${MYAPP}-listener-http --protocol HTTP --wait
openstack loadbalancer member create --address <IP_of_certnode> --protocol-port 80 --wait ${MYAPP}-letsencrypt-pool1

openstack loadbalancer l7policy create --action REDIRECT_TO_POOL --redirect-pool ${MYAPP}-letsencrypt-pool1 --name ${MYAPP}-letsencrypt-redirection ${MYAPP}-listener-http --position 1
openstack loadbalancer l7rule create --compare-type STARTS_WITH --type PATH --value /.well-known/acme-challenge ${MYAPP}-letsencrypt-redirection

Request an opening in the CERN firewall, such that the challenge can pass through to the VM. Bear in mind that these steps are subject to group membership synchronization times:
1. Go to the groups page and create a new group <PROJECT_NAME>-ACME-LANDBSET
  - Add your project responsible group as the administrators of the group.
  - After creating it, add the group openstack-landb-set-access as a member to this group.
2. In a new page, create a generic landb-set for the above VM and:
  - Add a desired Name <PROJECT_NAME>-ACME.
  - In the field Responsible, set the responsible to the group created above <PROJECT_NAME>-ACME-LANDBSET.
  - In the description add openstack_project=cc059d57-6e98-4688-a3be-aae2b451868b, <PROJECT_ID>.
    - cc059d57-6e98-4688-a3be-aae2b451868b is required in addition to your openstack project id: This is the project where the LB amphora VMs live. Then click "Create".
3. Now, add your load balancer to the LanDB Set.
```
openstack loadbalancer set --tag landb-set="<PROJECT_NAME>-ACME" ${MYAPP}-tls-term-lb
```
4. Finally, open a ticket with the Computer Security team so that can make the appropriate firewall openings.
  - If the service behind the load balancer is intended to be CERN-internal, a request to add your landb-set -ACME into their FIREWALL ALLOW ACME CHALLENGE set should be sufficient: Once accepted, the synchronization and opening of the firewall will take a maximum of 40 minutes. You may use the below as a template for your request ticket on SNOW:
    
    Request for load balancer ACME challenge access
    
    Dear Computer Security team,
    
    Could you please add our LanDB set <PROJECT_NAME-ACME> to the dedicated "FIREWALL ALLOW ACME CHALLENGE" set?
    
    This set contains our TLS-terminated load balancer, which is used for <insert purpose and reason for needing Lets Encrypt here>.
    
    Kind regards,
  - Alternatively, if the service is intended to be accessible from the Internet, you may use this moment to request a firewall opening for port 80 and 443. You can request the opening for the -ACME set above. The opening will also allow you to obtain your certificate. In this case you should directly proceed with the configuration steps below, as the security team will have to evaluate your deployment before making the opening.

Now, once the opening is in place, connect to the certbot VM and request a certificate:

export MYAPP="my-lb-app"
dnf install -y certbot
# We'll need to open the firewall locally too
firewall-cmd --permanent --add-service=http
firewall-cmd --reload
# Do a test run first
certbot --test-cert certonly -d $MYAPP.cern.ch --standalone -m <your_admin_group_email> --agree-tos
# If that works, obtain the production certificate
certbot certonly -d $MYAPP.cern.ch --standalone -m <your_admin_group_email> --agree-tos

Combine the received certificates into one, and prepare the .pk12 for the next steps:

cat /etc/letsencrypt/live/$MYAPP.cern.ch/fullchain.pem /etc/letsencrypt/live/${MYAPP}.cern.ch/privkey.pem | tee /etc/ssl/${MYAPP}.cern.ch.pem
# Leave the password empty when prompted
openssl pkcs12 -export -inkey /etc/ssl/${MYAPP}.cern.ch.pem -in /etc/ssl/${MYAPP}.cern.ch.pem -out /etc/ssl/${MYAPP}.cern.ch.p12

Now securely copy the .p12 certificate file to a machine from which you can access your OpenStack project, and proceed to the next step.
```
scp ...
```

3: Configure the loadbalancer

Store the certificate for the LB in OpenStack:

openstack secret store --name="${MYAPP}_lb_cert" -t 'application/octet-stream' -e 'base64' --payload="$(base64 < ${MYAPP}.cern.ch.p12)"

Create the HTTPS listener which will serve your application:

openstack loadbalancer listener create --protocol-port 443 --protocol TERMINATED_HTTPS --name ${MYAPP}-listener-https --default-tls-container=$(openstack secret list -f json --name ${MYAPP}_lb_cert | jq -r '.[] | ."Secret href"') --wait ${MYAPP}-tls-term-lb
openstack loadbalancer pool create --name ${MYAPP}-https-pool1 --lb-algorithm ROUND_ROBIN --listener ${MYAPP}-listener-https --protocol HTTP --wait
openstack loadbalancer member create --address <IP_of_my_desired_backend> --protocol-port 80 --wait ${MYAPP}-https-pool1

Optionally, create a redirect for HTTP traffic, such that it is upgraded to HTTPS:

# Skip this next command if using Lets Encrypt, as the listener is already created
openstack loadbalancer listener create --name "${MYAPP}-listener-http" --protocol HTTP --protocol-port 80 ${MYAPP}-tls-term-lb
openstack loadbalancer l7policy create --action REDIRECT_PREFIX --redirect-prefix https://${MYAPP}.cern.ch/ --name ${MYAPP}-http-redirect "${MYAPP}-listener-http"
openstack loadbalancer l7rule create --compare-type STARTS_WITH --type PATH --value / ${MYAPP}-http-redirect

At this stage your load balancer should be up and running.

Kubernetes Service Type LoadBalancer

Check the corresponding kubernetes service documentation.

For troubleshooting Kubernetes Service Type LoadBalancer, there is also information in the kubernetes troubleshooting documentation.

Automatic population of members with puppet servers

At the moment there is no service automatically adding or removing nodes when you add them to a hostgroup. However, we prepared a python script that can be run in aiadm (or similar machines with access to the puppet database) in the openstack project of the loadbalancer.

You have to manually set up the loadbalancer, listener, healthmanager and pool. After that with the pool having a description in the following format:

hostgroup=HG;port=PORT

with HG being the hostgroup that should be added to that pool and PORT being the port used for the members.

Example:

$ openstack loadbalancer pool show 44da3db2-dea9-402e-98ca-9ee24b9d2d99 -c name -c description
+-------------+----------------------------------------------------------+
| Field       | Value                                                    |
+-------------+----------------------------------------------------------+
| description | hostgroup=cloud_lbaas/controller/frontend/sdn3;port=9876 |
| name        | port-9876-pool                                           |
+-------------+----------------------------------------------------------+

At the run of the linked script, it will update the members to have all hosts registered in the hostgroup cloud_lbaas/controller/frontend/sdn3 with member ports 9876.

Additional parameters can be used to specify whether you want to register only IPv4 or/and IPv6 as well as whether you want to include all hosts belonging to the hostgroup and subgroups.

By default the script will not apply the changes but rather reports them to you. With --apply it will also update the loadbalancer by first adding the new members and then deleting the old ones.

Layer 7 Load Balancing

A Layer 7 load balancer can be used to make load balancing decisions based on the URI, host, HTTP headers, and other data in the application message. Please have a look at the L7 load balancing guide to find various use-cases of layer 7 load balancers with examples.

Load balancer with Proof of Work (PoW) challenge using Anubis

Experimental

This feature is in an early experimental stage. Feedback and bug reports are very welcome, but use at your own risk for the time being. This initial proof of concept comes with some limitations, notably:

The image will change throughout the trial, as we discover issues and fix them. Perform a failover on an existing LB to get the latest version.
It runs the 'free' tier of Anubis, so you cannot use premium BotStopper features such as custom branding.
Anubis is configured by the service, so you cannot tweak its internals

Anubis is a Web AI Firewall Utility that weighs the soul of your connection using one or more challenges in order to protect upstream resources from scraper bots.

This program is designed to help protect the small internet from the endless storm of requests that flood in from AI companies. Anubis is as lightweight as possible to ensure that everyone can afford to protect the communities closest to them.

In short, Anubis works by requiring a client to present a valid cookie when requesting a protected page. The load balancer sends the client to the Anubis backend until a JS-based challenge is completed and Anubis grants it a valid cookie. After this, when the client presents the obtained cookie, the load balancer forwards requests to the intended backend as usual.

LBaaS load balancers can be deployed with Anubis installed inside of the Amphora VM, thereby providing bot/scraper protection for the backends behind it. Anubis-enabled load balancers work by checking for the presence of a special cookie. If the cookie is present, the request is passed on to the service backend. On the other hand, if the cookie is missing or incorrect, the request is passed on to the Anubis instance, which will require the completion of a challenge before handing out a valid cookie.

As Anubis requires L7 access to modify request headers, the load balancer must be a TLS terminated one when serving HTTPS traffic. Thus, the example below is very similar to the TLS terminated one from above.

To create a TLS-terminated LB with Anubis, follow the steps in the TLS example above, with two important modifications:

When creating the loadbalancer, pass it the --flavor anubis option like so:

openstack loadbalancer create --name ${MYAPP}-tls-term-lb --vip-network-id CERN_NETWORK --tag "landb-alias=${MYAPP}" --flavor anubis --wait

When creating your HTTPS listener, activate Anubis on it using the --tag experimental_anubis option:

openstack loadbalancer listener create --protocol-port 443 --tag experimental_anubis  --protocol TERMINATED_HTTPS --name ${MYAPP}-listener-https --default-tls-container=$(openstack secret list -f json --name ${MYAPP}_lb_cert | jq -r '.[] | ."Secret href"') --wait ${MYAPP}-tls-term-lb

Miscellaneous

Preserve the client-ip for SSL Passthrough / TCP LoadBalancers

For use-cases where you want to do TLS termination on the backend or non-HTTP applications, you can use the TCP protocol for both listener and pool. In this mode, the load balancer can't insert headers (for HTTP applications) to indicate the client's IP address. Therefore for backend servers, traffic will appear to originate from the load balancer.

To preserve the client-ip, the PROXY protocol is proposed and many applications support it. You can create a TCP listener and a pool with protocol PROXY. Note that your backend application must support the PROXY protocol.

There is also support for the PROXYV2 protocol, which uses binary headers.

Load balancer Statistics

Monitoring dashboards for your load balancers can be found under the CLOUD organization on monit-grafana.cern.ch.

A detailed one exists specifically for TCP load balancers.

Setting Load balancer Session Limit

Concurrent session limit for a load balancer can be set by using the following command. Default value is set to -1.

openstack loadbalancer listener set --connection-limit=5000 <listener-name>

Setting Member Weights

The weight of a member determines the portion of requests or connections it services compared to the other members in the pool - the load is proportional to the member weight relative to the sum of all weights. The weight value can range between 0 and 256, defaulting to 1.

For further information take a look at the HA Proxy Documentation, and search for weight section.

Setting weight for a new member:

openstack loadbalancer member create --address 188.185.80.141 --weight 2 --protocol-port 80 pool

Updating weight for an existing member:

openstack loadbalancer member set --weight 2 <pool-name> <member-name>

Loadbalancer pool and member names can be found by executing the following commands:

openstack loadbalancer pool list

openstack loadbalancer member list <pool-name>

Enabling/Disabling Members

During some maintenance activities you might want to disable some members from serving requests. This feature can help in upgrading services with zero downtime. Following commands can be used to enable/disable loadbalancer members:

openstack loadbalancer member set --enable <pool-name> <member-name>
openstack loadbalancer member set --disable <pool-name> <member-name>

Session Persistence

Session persistence is a feature of the load balancing service. It attempts to force connections or requests in the same session to be processed by the same member as long as it is active. The OpenStack LBaaS service supports three types of persistence:

SOURCE_IP (Default):

With this persistence mode, all connections originating from the same source IP address, will be handled by the same member of the pool. Following command can be used to create a pool with session persistence of type SOURCE_IP:
```
openstack loadbalancer pool create --name <pool-name> --lb-algorithm ROUND_ROBIN --listener <listener-id> --protocol HTTP --session-persistence type=SOURCE_IP
```
HTTP_COOKIE:

With this persistence mode, the loadbalancer will create a cookie on the first request from a client. Subsequent requests containing the same cookie value will be handled by the same member of the pool. Following command can be used to create a pool with session persistence of type HTTP_COOKIE:
```
openstack loadbalancer pool create --name <pool-name> --lb-algorithm ROUND_ROBIN --listener <listener-id> --protocol HTTP --session-persistence type=HTTP_COOKIE
```
APP_COOKIE:

With this persistence mode, the loadbalancer will rely on a cookie established by the backend application. All requests carrying the same cookie value will be handled by the same member of the pool. Following command can be used to create a pool with session persistence of type APP_COOKIE:
```
openstack loadbalancer pool create --name <pool-name> --lb-algorithm ROUND_ROBIN --listener <listener-id> --protocol HTTP --session-persistence type=APP_COOKIE,cookie_name=<cookie-name>
```

Backup members

Multiple members can be marked as backups, load balancing will be performed among all backup servers when all normal ones are unavailable. For this feature to work, a healthmonitor resource should be created for the load balancer.

A member can be marked/unmarked as a backup by using the following commands respectively:

openstack loadbalancer member set --enable-backup <pool-name> <member-name>
openstack loadbalancer member set --disable-backup <pool-name> <member-name>

Setting domain name for load balancer

DNS update time

Please note that the domain name will be made available after 15 minutes in the worst case, waiting for the update of the DNS servers.

Domain name can be set for a load balancer by adding tags. Following command can be used to set domain name:

openstack loadbalancer set --tag landb-alias=my-domain-name mylb

ping my-domain-name.cern.ch

Multiple dns aliases can be specified as multiple tags as shown below:

openstack loadbalancer set --tag "landb-alias=my-domain-one" --tag "landb-alias=my-domain-two" --tag "landb-alias=my-domain-three" mylb

Let's say, if you want to remove my-domain-two, then remove the tag with the domain name as shown below:

openstack loadbalancer unset --tag "landb-alias=my-domain-two" mylb

If you want to remove all dns aliases, then simply remove all landb-alias tags

openstack loadbalancer unset --tag "landb-alias=my-domain-one" --tag "landb-alias=my-domain-three" mylb

Adding load balancer to LanDB sets

To add the load balancer to LanDB sets, you can add the tag landb-set=YOUR-SET-NAME to the load balancer.

Note that you will need to configure the LanDB set to allow the load balancer project and our user to have access. See the documentation for the Properties. The UUID for the project that needs to be added is: cc059d57-6e98-4688-a3be-aae2b451868b AND the project id of your project.

As an example: The description of your LanDB set should contain something like this:

openstack_project=cc059d57-6e98-4688-a3be-aae2b451868b,<your_project_id>

The egroup for the Set "Responsible" egroup needs to include the openstack-landb-set-access egroup as a member.

Beware that it is your egroup that needs to include openstack-landb-set-access. If you set the "openstack-landb-set-access" egroup directly as the Set responsible you lose access to the LanDB Set.

Other annotations for a load balancer

We support multiple of the cern specific properties in Properties. Since octavia at the moment does not support properties on the load balancer, we use tags for this purpose. Supported are landb-alias, landb-set, landb-mainuser and landb-ipv6ready with its corresponding values (e.g. the tag: landb-ipv6ready=true).

For example if we want to set the landb-mainuser on a loadbalancer you just need to run:

openstack loadbalancer set --tag "landb-mainuser=my-user-name" mylb

Deleting a load balancer

Load balancer resources should be deleted in the following order:

Members: $ openstack loadbalancer member delete <pool-id> <member-id>
Health monitor: $ openstack loadbalancer healthmonitor delete <healthmonitor-id>
Pool: $ openstack loadbalancer pool delete <pool-id>
Listener: $ openstack loadbalancer listener delete <listener-id>
Loadbalancer: $ openstack loadbalancer delete <loadbalancer-id>

Alternatively loadbalancers can now be deleted fully with:

openstack loadbalancer delete --cascade <LB_UUID>

Load balancers and Kerberos

In order to authenticate to an application using Kerberos behind a load balancer, you will have to obtain a keytab for the load balancer itself.

Getting a keytab for a load balancer

Note

Your load balancer must exist in DNS first. If it was just created you may need to wait a moment.

Log in as root on any CERN-managed device with Kerberos set up and the tool cern-get-keytab installed. This may for instance one of the back-end nodes which are running behind the load balancer.

Generate the keytab:

# lb="fa7e7f60-805c-49f2-afa2-b7aa180a2a11"  # use the ID of your lbaas device

# cern-get-keytab --hostname "lbaas-${lb}.cern.ch"
Waiting for password replication (0 seconds past)
Waiting for password replication (5 seconds past)
Keytab file saved: /etc/krb5.keytab

# klist -k /etc/krb5.keytab  # Validate keytab

For additional details, please read the Kerberos page on the CERN Linux docs.

Getting that keytab using `msktutil`

Open a ticket

This approach is only to be taken in rare cases where the above does not work. You will have to first open a ticket with the Cloud team, such that a special attribute is set on the LB in question.

Note

Please run the following procedure in lxplus/aiadm.

Get the computer name from LDAP for your load balancer:

$ lb="lbaas-fa7e7f60-805c-49f2-afa2-b7aa180a2a11"  # use your lbaas device
$ ldapsearch -x -H "ldap://xldap.cern.ch:389" -b "DC=cern,DC=ch" "cn=$lb" sAMAccountName | grep ^sAMAccountName
sAMAccountName: 48A9A0-52OA4H23E9M8$

Pick a domain controller:

$ host cerndc.cern.ch
cerndc.cern.ch has address 188.184.52.39
$ host 188.184.52.39
39.52.184.188.in-addr.arpa domain name pointer cerndc56.cern.ch.

Generate the keytab:

$ lb="lbaas-fa7e7f60-805c-49f2-afa2-b7aa180a2a11"
$ computer_name="48A9A0-52OA4H23E9M8" # omit the last $ character
$ cerndc="cerndc56.cern.ch"
$ msktutil update -s host --computer-name ${computer_name} --hostname ${lb}.cern.ch --keytab ~/${lb}.keytab --dont-expire-password --dont-update-dnshostname --base OU=Computers --verbose --server ${cerndc} --user-creds-only
$ msktutil update -s host --computer-name ${computer_name} --hostname ${lb}.cern.ch --keytab ~/${lb}.keytab --dont-expire-password --dont-update-dnshostname --base OU=Computers --verbose --server ${cerndc} --dont-change-password
$ klist  -k ~/${lb}.keytab

All commands in a script:

#!/bin/bash

lb="$1"

if [[ $(host "$lb") = *"not found"*  ]] ; then
    echo "${lb} does not exist in DNS, exit."
    exit 1
fi

computer_name=$(ldapsearch -x -H "ldap://xldap.cern.ch:389" -b "DC=cern,DC=ch" "cn=$lb" sAMAccountName | grep ^sAMAccountName  | sed 's/\$//g' | awk '{print $2}')
cerndc=$(dig -t SRV _kerberos._tcp.cern.ch | grep ^cerndc | awk 'NR==1{print $1}')
msktutil update -s host --computer-name ${computer_name} --hostname ${lb}.cern.ch --keytab ~/${lb}.keytab --dont-expire-password --dont-update-dnshostname --base OU=Computers --verbose --server ${cerndc} --user-creds-only
msktutil update -s host --computer-name ${computer_name} --hostname ${lb}.cern.ch --keytab ~/${lb}.keytab --dont-expire-password --dont-update-dnshostname --base OU=Computers --verbose --server ${cerndc} --dont-change-password
klist  -k ~/${lb}.keytab

Failover of a load balancer

Warning

Currently, this will incur a downtime of minimum 20 seconds for the loadbalancer instance. You should know what you are doing or contact the cloud team before doing the operation.

In the current setup, users in the project are able to perform a failover operation, which will spawn a new Amphora VM to host the load balancer by executing:

openstack loadbalancer failover <LB_UUID>

Testing new load balancer images

Load balancers will by default create their Amphora VM using a tested stable image. This image does not change often, but may be updated for reasons such as OpenStack major version upgrades and urgent security fixes.

Service managers who wish to test new images, which may contain more recent software versions and new features, may do so using a dedicated load balancer flavour. The crit_project_004_qa flavour makes use of a QA image, which is built and updated more regularly than the stable default. On the other hand, please be aware that this image is undergoing testing and may not yet be suitable for production services. QA images may after testing eventually be promoted to become the new stable image.

To create a load balancer with an Amphora using the QA image:

openstack loadbalancer create --name mylb-qa --vip-network-id CERN_NETWORK --flavor qa_amphora_image

Then proceed with creating the pools/members/etc. as usual.

Load Balancing

Service Load Balancing vs Service Availability

Access to Load Balancers

Concepts

Examples

Basic HTTP Load Balancer

TCP Load Balancer

UDP Load Balancer

TLS termination

1: Create the load balancer

2: Prepare the certificate

Alt. A: Using CERN Host certificates

Alt. B: Using CERN commercial certificates

Alt. C: Using Let's Encrypt

3: Configure the loadbalancer

Kubernetes Service Type LoadBalancer

Automatic population of members with puppet servers

Layer 7 Load Balancing

Load balancer with Proof of Work (PoW) challenge using Anubis

Miscellaneous

Preserve the client-ip for SSL Passthrough / TCP LoadBalancers

Load balancer Statistics

Setting Load balancer Session Limit

Setting Member Weights

Enabling/Disabling Members

Session Persistence

Backup members

Setting domain name for load balancer

Adding load balancer to LanDB sets

Other annotations for a load balancer

Deleting a load balancer

Load balancers and Kerberos

Getting a keytab for a load balancer

Getting that keytab using msktutil

All commands in a script:

Failover of a load balancer

Testing new load balancer images

Getting that keytab using `msktutil`