Guides

• 1: Envoy Gateway
• 1.1: Direct (PROXY-protocol) mode
• 1.2: Proxy (X-Forwarded-For) mode
• 1.3: Migrating from ingress-nginx
• 2: Cert-manager and Cloudflare demo
• 3: Change PV StorageClass
• 4: Ingress and cert-manager
• 5: Install and expose ArgoCD
• 6: Install and upgrade cert-manager
• 7: Install and upgrade ingress-nginx
• 8: Load balancers
• 9: Migration to Kubernetes CaaS v2
• 10: Node labels and taints with elx-nodegroup-controller
• 11: Nodegroups on a shared network
• 12: Persistent volumes
• 13: Your first deployment

1 - Envoy Gateway

This section introduces Envoy Gateway as the ingress controller in our Elastx Kubernetes CaaS service. We manage and upgrade the controller, the Gateway API CRDs and the cluster-scoped GatewayClass named eg. You create the Gateway API objects that describe your own traffic in your own namespaces.

Envoy Gateway is not installed by default. Ask our support to turn it on for your cluster, and tell us whether you use direct or proxy mode (see below). If you use our managed cert-manager, we turn on its Gateway API support at the same time, so it can issue certificates for your Gateway.

There are companion guides for the two ways traffic typically reaches the cluster. Pick the one that matches your setup:

• Direct (PROXY-protocol) mode: clients connect straight to the load balancer.
• Proxy (X-Forwarded-For) mode: a customer-owned upstream proxy (CDN, WAF, edge proxy) sits in front of the load balancer.

Standard layout: one shared Gateway per cluster

A cluster has one shared Gateway in a dedicated namespace (for example gateway) that serves routes from all your application namespaces through a single OpenStack load balancer and IP. This is the standard setup and the direct equivalent of ingress-nginx, where a single controller fronted every host in the cluster. Each application namespace opts in with a shared-gateway-access: "true" label and contributes its own HTTPRoutes, and never touches the shared Gateway.

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#DAE7EC','primaryBorderColor':'#1E343E','primaryTextColor':'#1E343E','lineColor':'#5A7A8A','clusterBkg':'#EEF3F6','clusterBorder':'#9BB3BF','edgeLabelBackground':'#FFFFFF'}}}%%
flowchart TB
    client(["Clients"]):::client --> lb["OpenStack load balancer<br/>one LB &middot; one IP"]:::lb
    lb --> gw

    subgraph gwns["gateway namespace"]
        gw["Gateway 'shared'"]:::gw
        ctp["ClientTrafficPolicy"]:::policy
        cert["TLS certificates<br/>per hostname"]:::policy
    end
    ctp -.->|attaches to| gw
    cert -.->|terminates TLS| gw

    subgraph ta["team-a namespace"]
        appa["app + HTTPRoute"]:::app
    end
    subgraph tb["team-b namespace"]
        appb["app + HTTPRoute"]:::app
    end
    gw -->|"team-a.example.com"| appa
    gw -->|"team-b.example.com"| appb

    classDef client fill:#FFFFFF,stroke:#1E343E,color:#1E343E;
    classDef lb fill:#DAE7EC,stroke:#1E343E,color:#1E343E;
    classDef gw fill:#FBBD18,stroke:#1E343E,stroke-width:2px,color:#1E343E;
    classDef policy fill:#F5F8FA,stroke:#1E343E,color:#1E343E;
    classDef app fill:#DAE7EC,stroke:#1E343E,color:#1E343E;

Each Gateway provisions its own OpenStack load balancer, so a single shared Gateway keeps your cluster on one load balancer, one IP and one ClientTrafficPolicy: the same single-entry-point model you had with ingress-nginx. TLS is terminated centrally in the gateway namespace, with one certificate per hostname (Envoy serves the right one per request by SNI). The walkthroughs below use this layout throughout. Running more than one Gateway, a separate load balancer for a single namespace via allowedRoutes.namespaces.from: Same, is a non-standard setup for the rare case that genuinely needs an isolated IP or blast radius.

What you create

In the dedicated gateway namespace (once, by whoever owns ingress):

• Gateway: listeners, ports, protocols, TLS; allowedRoutes selecting the shared-gateway-access: "true" label.
• ClientTrafficPolicy: controls PROXY-protocol handling, TLS parameters, timeouts. Must live in the same namespace as the Gateway.
• TLS Certificate / Issuer (cert-manager): one certificate per hostname you serve (HTTP-01 in direct mode, DNS-01 in proxy mode).
• Optionally, an HTTP-to-HTTPS redirect HTTPRoute on the http listener, to match ingress-nginx’s ssl-redirect behaviour for every host.

In each application namespace (per team, self-service):

• The shared-gateway-access: "true" namespace label.
• HTTPRoute, GRPCRoute: routing rules, attached to the shared Gateway via cross-namespace parentRefs.
• BackendTrafficPolicy: retries, circuit breaking.
• SecurityPolicy: JWT, OIDC, CORS.
• BackendTLSPolicy: mTLS toward your backends.

You reference the cluster GatewayClass by its name eg from the Gateway. You do not need to create or modify any cluster-scoped resources.

Which variant fits your setup?

The OpenStack load balancer in front of Envoy runs in TCP mode in both cases. The variants differ in how the real client IP arrives at Envoy, and your ClientTrafficPolicy has to match.

• Direct (PROXY-protocol) mode: clients connect straight to the load balancer. The load balancer is configured with PROXY protocol v2 and prepends a PROXY header carrying the real client IP. Your ClientTrafficPolicy must enable proxy-protocol parsing. See Direct (PROXY-protocol) mode.
• Proxy (X-Forwarded-For) mode: you put your own upstream proxy (CDN, WAF, edge proxy) in front of the load balancer. That upstream injects the real client IP into X-Forwarded-For; the load balancer passes the request through unchanged. Your ClientTrafficPolicy must trust that header with the right hop count. See Proxy (X-Forwarded-For) mode.

Coming from ingress-nginx?

If you used our managed ingress-nginx, the two modes carry over directly; only the names and the resources you write have changed:

As before, the mode is a cluster-level setting: we provision your cluster in one mode or the other; you do not switch it from a manifest. Tell us which fits your setup and we configure the load balancer accordingly. What you do write is a ClientTrafficPolicy that matches that mode (see the two guides above).

Proxy mode requires your own upstream proxy. X-Forwarded-For mode only makes sense when a CDN, WAF, or edge proxy actually sits in front of the load balancer and injects the header. Without one, no real client IP ever reaches Envoy and your backends see only the load balancer. If clients connect directly to Elastx, use direct (PROXY-protocol) mode instead. It carries the client IP for you and needs no CDN.

ingress-nginx has been retired upstream, as announced in the Kubernetes 1.34 changelog. For the full move, see Migrating from ingress-nginx.

TLS

Both walkthroughs terminate TLS on the shared Gateway, with cert-manager issuing one certificate per hostname into the gateway namespace (the same namespace as the Gateway). The validation method differs by mode: direct mode uses ACME HTTP-01 (clients reach the load balancer directly), while proxy mode uses DNS-01 (public DNS points at your upstream proxy, so HTTP-01 cannot reach Envoy). If you need a guide for installing cert-manager, see Install and upgrade cert-manager.

Advanced usage

For more advanced use cases please refer to the documentation provided by each project or contact our support:

• Gateway API
• Envoy Gateway
• cert-manager

1.1 - Direct (PROXY-protocol) mode

This guide walks through setting up Envoy Gateway in a cluster where the OpenStack load balancer is configured in TCP mode with PROXY protocol v2. The load balancer prepends a PROXY header to each incoming connection carrying the real client IP. Envoy parses that header and uses it for access logs, rate limiting and X-Forwarded-For.

Note: Your ClientTrafficPolicy must set proxyProtocol.optional: false. Without it Envoy parses the load balancer’s PROXY-v2 prefix as a malformed HTTP request and every response is HTTP 400 Bad Request.

If you are not sure which variant applies to your cluster, see the Envoy Gateway overview.

The shared-Gateway layout

A cluster runs one shared Gateway in a dedicated namespace that serves routes from all your application namespaces through a single load balancer and IP. Each application namespace opts in with a label and contributes its own HTTPRoutes. This is the standard setup, the same single-entry-point model ingress-nginx gave you, where one controller fronted every host.

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#DAE7EC','primaryBorderColor':'#1E343E','primaryTextColor':'#1E343E','lineColor':'#5A7A8A','clusterBkg':'#EEF3F6','clusterBorder':'#9BB3BF','edgeLabelBackground':'#FFFFFF'}}}%%
flowchart TB
    client(["Clients"]):::client -->|PROXY protocol v2| lb["OpenStack load balancer<br/>TCP mode &middot; one LB &middot; one IP"]:::lb
    lb --> gw

    subgraph gwns["gateway namespace"]
        gw["Gateway 'shared'"]:::gw
        ctp["ClientTrafficPolicy<br/>proxyProtocol.optional: false"]:::policy
        cert["TLS certificate<br/>team-a.example.com"]:::policy
    end
    ctp -.->|attaches to| gw
    cert -.->|terminates TLS| gw

    subgraph ta["team-a namespace (labelled)"]
        appa["app + HTTPRoute"]:::app
    end
    subgraph tb["team-b namespace (labelled)"]
        appb["app + HTTPRoute"]:::app
    end
    gw -->|"team-a.example.com"| appa
    gw -->|"team-b.example.com"| appb

    classDef client fill:#FFFFFF,stroke:#1E343E,color:#1E343E;
    classDef lb fill:#DAE7EC,stroke:#1E343E,color:#1E343E;
    classDef gw fill:#FBBD18,stroke:#1E343E,stroke-width:2px,color:#1E343E;
    classDef policy fill:#F5F8FA,stroke:#1E343E,color:#1E343E;
    classDef app fill:#DAE7EC,stroke:#1E343E,color:#1E343E;

What the shared Gateway gives you:

• One load balancer per cluster: a single LB and IP front all your teams, the way a single ingress controller did before.
• One ClientTrafficPolicy to manage, in the gateway namespace, with TLS terminated there (one certificate per hostname you serve).
• Self-service for app teams: an app team only labels its namespace and creates an HTTPRoute; it never touches the shared Gateway.

Each Gateway provisions its own load balancer. Running more than one (a separate Gateway and load balancer for a single namespace, via allowedRoutes.namespaces.from: Same) is a non-standard setup, for the rare case that genuinely needs an isolated IP or blast radius.

Prerequisites

• Envoy Gateway turned on for your cluster in direct (PROXY-protocol) mode. Ask our support if it is not.
• A dedicated namespace for the shared Gateway. The examples use gateway.
• One or more application namespaces. The examples use team-a.
• A DNS record for each hostname you serve, pointing at the load balancer’s public IP. The examples use team-a.example.com; replace it with your own throughout.
• cert-manager in the cluster. The examples issue certificates with ACME HTTP-01, which needs no DNS-provider credentials. If you are not using our managed cert-manager, install your own with Gateway API support enabled (--enable-gateway-api).

Create the gateway namespace

kubectl create namespace gateway
kubectl label namespace gateway shared-gateway-access=true

The label lets routes created in the gateway namespace itself attach to the shared Gateway. Two gateway-owner routes need this: cert-manager’s short-lived HTTP-01 challenge route, and the HTTP-to-HTTPS redirect below.

Create the shared Gateway

Gateway describes the listeners. Put it in the dedicated gateway namespace and reference the cluster GatewayClass named eg. The allowedRoutes selector is what lets routes in other namespaces attach.

Create a file called gateway.yaml with the following content:

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: shared
  namespace: gateway
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              shared-gateway-access: "true"
    - name: https
      port: 443
      protocol: HTTPS
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              shared-gateway-access: "true"
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: team-a-tls

The Gateway has two listeners. The https listener (port 443) terminates TLS; it has no hostname, so it serves every host whose certificate is listed in its certificateRefs, and Envoy picks the right one per request by SNI. The http listener (port 80) carries plaintext requests: it serves ACME HTTP-01 challenges and is where the HTTP-to-HTTPS redirect below attaches.

allowedRoutes.from: Selector admits routes from any namespace carrying the shared-gateway-access: "true" label; this is the opt-in that makes the Gateway shared. Use from: Same instead if you ever want a Gateway that only serves its own namespace, or from: All to admit every namespace unconditionally (not recommended, since it removes the opt-in).

Apply it: kubectl apply -f gateway.yaml

Configure proxy-protocol with ClientTrafficPolicy

The ClientTrafficPolicy attaches to the Gateway by name and tells Envoy to parse the PROXY-v2 header from the load balancer. It lives in the gateway namespace alongside the Gateway and covers the whole load balancer; app namespaces do not need their own.

Create a file called client-traffic-policy.yaml:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: shared
  namespace: gateway
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: shared
  proxyProtocol:
    optional: false

Note: The policy must live in the same namespace as the Gateway. Envoy Gateway rejects cross-namespace policy targets.

Apply it: kubectl apply -f client-traffic-policy.yaml

Issue a TLS certificate

Terminate TLS on the shared Gateway with a certificate per hostname, issued by cert-manager into the gateway namespace (where TLS terminates). In direct mode clients reach the load balancer directly, so ACME HTTP-01 is the simplest validation: Let’s Encrypt fetches a token over port 80, which the shared Gateway already serves, and no DNS-provider credentials are needed.

Create a file called certificate.yaml:

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-http01
  namespace: gateway
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform@example.com
    privateKeySecretRef:
      name: letsencrypt-http01-account
    solvers:
      - http01:
          gatewayHTTPRoute:
            parentRefs:
              - group: gateway.networking.k8s.io
                kind: Gateway
                name: shared
                namespace: gateway
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: team-a-tls
  namespace: gateway
spec:
  secretName: team-a-tls
  issuerRef:
    name: letsencrypt-http01
    kind: Issuer
  dnsNames:
    - team-a.example.com

Replace the email and hostname with your own, then apply it: kubectl apply -f certificate.yaml

To solve the challenge, cert-manager creates a short-lived HTTPRoute in the gateway namespace, attached to the http listener. It is admitted because you labelled the gateway namespace when you created it. The https listener stays pending until the certificate is issued, then serves it.

Each additional hostname needs its own Certificate and a matching entry in the https listener’s certificateRefs; Envoy then selects the right certificate per request by SNI.

Prefer DNS-01? If you would rather validate over DNS (for example to keep the ACME servers off port 80), use a DNS-01 Issuer instead. DNS-01 needs API credentials for your DNS provider; see the cert-manager DNS-01 docs.

Redirect HTTP to HTTPS

ingress-nginx redirected HTTP to HTTPS for you (the ssl-redirect default). To keep that behaviour, attach one redirect HTTPRoute to the http listener, created once in the gateway namespace (already labelled above). It matches every host on port 80, so app teams do not add their own.

Create https-redirect.yaml:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: https-redirect
  namespace: gateway
spec:
  parentRefs:
    - name: shared
      namespace: gateway
      sectionName: http
  rules:
    - filters:
        - type: RequestRedirect
          requestRedirect:
            scheme: https
            statusCode: 301

Apply it: kubectl apply -f https-redirect.yaml

A request to http://team-a.example.com/ now returns 301 Moved Permanently with Location: https://team-a.example.com/, and the client repeats the request over HTTPS.

Does this break HTTP-01 certificates? No. This redirect matches the path /, while cert-manager’s challenge route matches the longer /.well-known/acme-challenge/ path. Gateway API gives precedence to the longest path match, so ACME challenges are still served over plain HTTP while everything else redirects.

Onboard an application namespace

This is all an app team does; no access to the gateway namespace is needed.

1. Label the namespace so the shared Gateway admits its routes:

kubectl label namespace team-a shared-gateway-access=true

2. Deploy the app. Create app.yaml:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
  namespace: team-a
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
        - name: echo
          image: ealen/echo-server:0.9.2
          ports:
            - containerPort: 80
          env:
            - name: PORT
              value: "80"
---
apiVersion: v1
kind: Service
metadata:
  name: echo
  namespace: team-a
spec:
  selector:
    app: echo
  ports:
    - port: 80
      targetPort: 80

3. Route traffic to it. Create route.yaml. The parentRefs points at the shared Gateway in the gateway namespace, and that cross-namespace reference is what puts this app behind the shared load balancer:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: echo
  namespace: team-a
spec:
  parentRefs:
    - name: shared
      namespace: gateway
      sectionName: https
  hostnames:
    - team-a.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: echo
          port: 80

Apply both: kubectl apply -f app.yaml -f route.yaml

The route’s hostname (team-a.example.com) must have a certificate on the https listener; you issued one above. The backend Service is in the same namespace as the HTTPRoute, so no ReferenceGrant is needed; you only need one if a route points at a Service in a different namespace.

Verify

Check that the shared Gateway got an external address and that traffic flows for the app namespace:

kubectl -n gateway get gateway shared -o jsonpath='{.status.addresses[0].value}'
curl -v https://team-a.example.com/

Confirm the route attached to the shared Gateway:

kubectl -n team-a get httproute echo -o jsonpath='{.status.parents[0].conditions}'

Accepted: True and ResolvedRefs: True mean the cross-namespace attach worked. The backend should see the real client IP in X-Forwarded-For and X-Envoy-External-Address.

Common mistakes

• Namespace not labelled: the HTTPRoute reports Accepted: False with reason NotAllowedByListeners, and traffic never reaches the app. Label the app namespace shared-gateway-access=true.
• Forgetting namespace: in parentRefs: without it the route looks for a Gateway in its own namespace, finds none, and stays unattached. Cross-namespace routes must name the gateway namespace.
• No certificate for the hostname: if the https listener has no certificate matching the route’s hostname, the TLS handshake fails and clients cannot connect. Issue a Certificate for each hostname and add its Secret to the listener’s certificateRefs.
• Forgetting ClientTrafficPolicy: every request returns HTTP 400 Bad Request. The load balancer is prepending a PROXY-v2 binary header; without the policy Envoy treats those bytes as the start of an HTTP request and fails to parse it.
• Putting ClientTrafficPolicy in another namespace: silently ignored. Must be colocated with the Gateway (here, the gateway namespace).
• Setting proxyProtocol.optional: true: opens you up to clients that don’t send the header bypassing client-IP enforcement. Keep it false.
• Testing with curl from outside the load balancer: PROXY-protocol traffic isn’t valid HTTP. Always go through the load balancer’s VIP.

Advanced usage

For more advanced use cases please refer to the documentation provided by each project or contact our support:

• Envoy Gateway overview
• Proxy (X-Forwarded-For) mode
• Gateway API
• Envoy Gateway

1.2 - Proxy (X-Forwarded-For) mode

This guide walks through setting up Envoy Gateway in a cluster where you front the OpenStack load balancer with your own upstream proxy (for example a CDN, WAF, or edge proxy) that terminates the client connection and injects the real client IP into the X-Forwarded-For header. The OpenStack load balancer itself stays in TCP passthrough; the upstream proxy is what carries the client IP for you.

This mode requires your own upstream proxy. It is only correct when a CDN, WAF, or edge proxy that you operate sits in front of the load balancer and injects X-Forwarded-For. Without one, no real client IP ever reaches Envoy, so your backends see only the load balancer’s internal IP. If clients connect straight to Elastx with nothing in front, use direct (PROXY-protocol) mode instead; it carries the client IP for you and needs no proxy.

Note: Your ClientTrafficPolicy must set clientIPDetection.xForwardedFor with numTrustedHops set to the number of trusted proxies in front of Envoy. Without it Envoy will not honour the incoming X-Forwarded-For header and your access logs and rate limiting will see the load balancer’s internal IP.

Note: The upstream proxy must terminate TLS to inject X-Forwarded-For; it can only read and modify headers on decrypted traffic. It then opens a fresh connection to the load balancer (which passes it through untouched) and Envoy terminates TLS again on the Gateway HTTPS listener. A pure TCP/TLS passthrough upstream cannot inject the header.

If you are not sure which variant applies to your cluster, see the Envoy Gateway overview.

The shared-Gateway layout

A cluster runs one shared Gateway in a dedicated namespace that serves routes from all your application namespaces through a single load balancer and IP. Each application namespace opts in with a label and contributes its own HTTPRoutes. This is the standard setup, the same single-entry-point model ingress-nginx gave you, where one controller fronted every host.

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#DAE7EC','primaryBorderColor':'#1E343E','primaryTextColor':'#1E343E','lineColor':'#5A7A8A','clusterBkg':'#EEF3F6','clusterBorder':'#9BB3BF','edgeLabelBackground':'#FFFFFF'}}}%%
flowchart TB
    client(["Clients"]):::client --> proxy["Your upstream proxy<br/>CDN / WAF / edge<br/>injects X-Forwarded-For"]:::proxy
    proxy -->|TCP passthrough| lb["OpenStack load balancer<br/>TCP mode &middot; one LB &middot; one IP"]:::lb
    lb --> gw

    subgraph gwns["gateway namespace"]
        gw["Gateway 'shared'"]:::gw
        ctp["ClientTrafficPolicy<br/>xForwardedFor.numTrustedHops: 1"]:::policy
        cert["TLS certificate<br/>team-a.example.com"]:::policy
    end
    ctp -.->|attaches to| gw
    cert -.->|terminates TLS| gw

    subgraph ta["team-a namespace (labelled)"]
        appa["app + HTTPRoute"]:::app
    end
    subgraph tb["team-b namespace (labelled)"]
        appb["app + HTTPRoute"]:::app
    end
    gw -->|"team-a.example.com"| appa
    gw -->|"team-b.example.com"| appb

    classDef client fill:#FFFFFF,stroke:#1E343E,color:#1E343E;
    classDef proxy fill:#CFE8FF,stroke:#0041C2,color:#1E343E;
    classDef lb fill:#DAE7EC,stroke:#1E343E,color:#1E343E;
    classDef gw fill:#FBBD18,stroke:#1E343E,stroke-width:2px,color:#1E343E;
    classDef policy fill:#F5F8FA,stroke:#1E343E,color:#1E343E;
    classDef app fill:#DAE7EC,stroke:#1E343E,color:#1E343E;

What the shared Gateway gives you:

• One load balancer per cluster: a single LB and IP front all your teams, the way a single ingress controller did before, and your upstream proxy points at a single origin IP.
• One ClientTrafficPolicy to manage, in the gateway namespace, with TLS terminated there (one certificate per hostname you serve).
• Self-service for app teams: an app team only labels its namespace and creates an HTTPRoute; it never touches the shared Gateway.

Each Gateway provisions its own load balancer. Running more than one (a separate Gateway and load balancer for a single namespace, via allowedRoutes.namespaces.from: Same) is a non-standard setup, for the rare case that genuinely needs an isolated IP or blast radius.

Prerequisites

• Envoy Gateway turned on for your cluster in proxy (X-Forwarded-For) mode. Ask our support if it is not.
• A dedicated namespace for the shared Gateway. The examples use gateway.
• One or more application namespaces. The examples use team-a.
• DNS for each public hostname (the examples use team-a.example.com, replace with your own) pointing at your upstream proxy (CDN / WAF / edge proxy), not directly at the load balancer. If it resolves straight to the load balancer, traffic bypasses the proxy and no X-Forwarded-For is injected.
• Your upstream proxy configured with the load balancer’s public IP as its origin / backend.
• cert-manager in the cluster, with a DNS-01 capable Issuer. In proxy mode HTTP-01 cannot reach Envoy (public DNS points at your proxy), so DNS-01 is used; it needs API credentials for your DNS provider. If you are not using our managed cert-manager, install your own.

Create the gateway namespace

kubectl create namespace gateway

Create the shared Gateway

Gateway describes the listeners. Put it in the dedicated gateway namespace and reference the cluster GatewayClass named eg. The allowedRoutes selector is what lets routes in other namespaces attach.

Create a file called gateway.yaml with the following content:

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: shared
  namespace: gateway
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              shared-gateway-access: "true"
    - name: https
      port: 443
      protocol: HTTPS
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              shared-gateway-access: "true"
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: team-a-tls

The Gateway has two listeners. The https listener (port 443) terminates TLS for the re-encrypted hop from your upstream proxy; it has no hostname, so it serves every host whose certificate is listed in its certificateRefs, and Envoy picks the right one per request by SNI. The http listener (port 80) is where an HTTP-to-HTTPS redirect would attach, if you run one here rather than at your proxy (see below).

allowedRoutes.from: Selector admits routes from any namespace carrying the shared-gateway-access: "true" label; this is the opt-in that makes the Gateway shared. Use from: Same instead if you ever want a Gateway that only serves its own namespace, or from: All to admit every namespace unconditionally (not recommended, since it removes the opt-in).

Apply it: kubectl apply -f gateway.yaml

Configure X-Forwarded-For with ClientTrafficPolicy

The ClientTrafficPolicy attaches to the Gateway by name and tells Envoy how many trusted proxies sit in front of it. It lives in the gateway namespace alongside the Gateway and covers the whole load balancer; app namespaces do not need their own.

Create a file called client-traffic-policy.yaml:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: shared
  namespace: gateway
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: shared
  clientIPDetection:
    xForwardedFor:
      numTrustedHops: 1

numTrustedHops tells Envoy how many trusted ingress proxy hops sit in front of it. Set it to the number of upstream proxies that prepend entries to X-Forwarded-For. For a single CDN/WAF/edge proxy in front of the load balancer, 1 is the right value; raise it for chains of multiple proxies.

The setting affects what Envoy itself treats as the client IP, used in access logs and rate limiting (and, in direct mode, the x-envoy-external-address header). Backends always see the full X-Forwarded-For chain that arrived plus the load balancer’s internal IP appended on the right; Envoy does not trim entries before forwarding the request upstream. Backend code that needs the real client IP should parse the chain itself, typically taking the leftmost public IP.

Note: The policy must live in the same namespace as the Gateway. Envoy Gateway rejects cross-namespace policy targets.

Apply it: kubectl apply -f client-traffic-policy.yaml

Alternative: using a custom header

Some load balancer setups forward the client IP in a different header. Use customHeader instead; it is mutually exclusive with xForwardedFor:

  clientIPDetection:
    customHeader:
      name: X-Real-IP

Issue a TLS certificate

Terminate TLS on the shared Gateway with a certificate per hostname, issued by cert-manager into the gateway namespace (where TLS terminates). In proxy mode your public DNS points at the upstream proxy rather than the load balancer, so an HTTP-01 challenge would never reach Envoy. Use DNS-01 validation, which proves control through a DNS record instead; it needs API credentials for your DNS provider.

Create a file called certificate.yaml:

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-dns
  namespace: gateway
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform@example.com
    privateKeySecretRef:
      name: letsencrypt-dns-account
    solvers:
      - dns01:
          # Configure a DNS-01 solver for your DNS provider; see the
          # cert-manager docs: https://cert-manager.io/docs/configuration/acme/dns01/
          {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: team-a-tls
  namespace: gateway
spec:
  secretName: team-a-tls
  issuerRef:
    name: letsencrypt-dns
    kind: Issuer
  dnsNames:
    - team-a.example.com

Replace the email, DNS solver and hostname with your own, then apply it: kubectl apply -f certificate.yaml

Each additional hostname needs its own Certificate and a matching entry in the https listener’s certificateRefs; Envoy then selects the right certificate per request by SNI.

Note: In proxy mode the upstream proxy terminates TLS for the public client; this certificate is for the re-encrypted hop between the upstream proxy and Envoy. Some setups instead let the proxy talk plain HTTP to the load balancer; if so, route via the http listener and you can skip the certificate.

Redirect HTTP to HTTPS

In this mode your upstream proxy (CDN / WAF / edge) usually performs the HTTP-to-HTTPS redirect before traffic ever reaches the load balancer, so you often do not need to configure one here. If you would rather have Envoy do it, attach a redirect HTTPRoute to the http listener exactly as in the direct-mode guide: label the gateway namespace shared-gateway-access: "true" and apply a RequestRedirect route in it.

Onboard an application namespace

This is all an app team does; no access to the gateway namespace is needed.

1. Label the namespace so the shared Gateway admits its routes:

kubectl label namespace team-a shared-gateway-access=true

2. Deploy the app. Create app.yaml:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
  namespace: team-a
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
        - name: echo
          image: ealen/echo-server:0.9.2
          ports:
            - containerPort: 80
          env:
            - name: PORT
              value: "80"
---
apiVersion: v1
kind: Service
metadata:
  name: echo
  namespace: team-a
spec:
  selector:
    app: echo
  ports:
    - port: 80
      targetPort: 80

3. Route traffic to it. Create route.yaml. The parentRefs points at the shared Gateway in the gateway namespace, and that cross-namespace reference is what puts this app behind the shared load balancer:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: echo
  namespace: team-a
spec:
  parentRefs:
    - name: shared
      namespace: gateway
      sectionName: https
  hostnames:
    - team-a.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: echo
          port: 80

Apply both: kubectl apply -f app.yaml -f route.yaml

The route’s hostname (team-a.example.com) must have a certificate on the https listener; you issued one above. The backend Service is in the same namespace as the HTTPRoute, so no ReferenceGrant is needed; you only need one if a route points at a Service in a different namespace.

Verify

Check that the shared Gateway got an external address and that traffic flows for the app namespace:

kubectl -n gateway get gateway shared -o jsonpath='{.status.addresses[0].value}'
curl -v https://team-a.example.com/

Confirm the route attached to the shared Gateway:

kubectl -n team-a get httproute echo -o jsonpath='{.status.parents[0].conditions}'

Accepted: True and ResolvedRefs: True mean the cross-namespace attach worked. The backend should see the real client IP at the left of X-Forwarded-For. Envoy forwards the full chain to the backend (including the load balancer IP it appends on the right) and does not remove entries, so the backend application is responsible for parsing the chain and picking the leftmost public IP.

Common mistakes

• Namespace not labelled: the HTTPRoute reports Accepted: False with reason NotAllowedByListeners, and traffic never reaches the app. Label the app namespace shared-gateway-access=true.
• Forgetting namespace: in parentRefs: without it the route looks for a Gateway in its own namespace, finds none, and stays unattached. Cross-namespace routes must name the gateway namespace.
• No certificate for the hostname: if the https listener has no certificate matching the route’s hostname, the TLS handshake fails and clients cannot connect. Issue a Certificate for each hostname and add its Secret to the listener’s certificateRefs.
• No upstream proxy in front: this variant assumes a CDN, WAF, or other proxy injects X-Forwarded-For before traffic reaches the load balancer. Without one, no real client IP arrives, and your backend will only see the LB’s internal IP. If you have no upstream proxy, use direct (PROXY-protocol) mode instead.
• Forgetting ClientTrafficPolicy: Envoy ignores the incoming X-Forwarded-For and treats the load balancer’s internal IP as the client. Rate limiting and access logs see the LB, not your real client.
• Putting ClientTrafficPolicy in another namespace: silently ignored. Must be colocated with the Gateway (here, the gateway namespace).
• Wrong numTrustedHops: too low and a caller can spoof the client IP by adding their own X-Forwarded-For entry. Too high and Envoy walks too far back into spoofable territory. Count one per trusted upstream proxy.
• Mixing xForwardedFor and customHeader: they are mutually exclusive. Pick one.

Advanced usage

For more advanced use cases please refer to the documentation provided by each project or contact our support:

• Envoy Gateway overview
• Direct (PROXY-protocol) mode
• Gateway API
• Envoy Gateway

1.3 - Migrating from ingress-nginx

This guide helps you move an existing workload from our managed ingress-nginx to Envoy Gateway. The ingress concepts are the same (listeners, routes, TLS, and a choice between direct and proxy mode), but the resources you write are Gateway API objects instead of Ingress objects.

If you are not migrating but setting up fresh, start from the Envoy Gateway overview instead.

What stays the same

• Ingress enters through one load balancer. A single shared Gateway gets one OpenStack load balancer and IP, just as ingress-nginx did. Your public DNS points at that address.
• Direct vs proxy is still a cluster-level mode, set by us: the same choice you made with ingress-nginx (use-proxy-protocol vs use-forwarded-headers), just under new names.

Step 0: Confirm your mode (direct or proxy)

The mode must match how your cluster’s load balancer is provisioned. It is a cluster-level setting that we manage; if you are unsure which one your cluster runs, ask support before you cut over.

• Direct (PROXY-protocol) mode: clients connect straight to our load balancer. The load balancer carries the real client IP with PROXY protocol v2. No CDN or upstream proxy is involved. This is the default and the equivalent of ingress-nginx use-proxy-protocol: "true".
• Proxy (X-Forwarded-For) mode: you operate your own CDN / WAF / edge proxy in front of the load balancer, and it injects X-Forwarded-For. The equivalent of ingress-nginx use-forwarded-headers: "true".

Proxy mode requires your own upstream proxy. It only makes sense if a CDN, WAF, or edge proxy actually sits in front of the load balancer and injects the header. Without one, no real client IP reaches Envoy and your backends see only the load balancer. If clients connect directly to Elastx, use direct (PROXY-protocol) mode; it needs no CDN.

Once you know your mode, the per-resource walkthrough lives in:

• Direct (PROXY-protocol) mode
• Proxy (X-Forwarded-For) mode

Resource mapping

Note: Our Envoy Gateway ships the Gateway API standard channel, which provides only HTTPRoute and GRPCRoute. TCPRoute, TLSRoute and UDPRoute are not installed. If you relied on the nginx tcp-services ConfigMap, contact support before migrating those.

Migration steps

1. Ask support to turn on Envoy Gateway for your cluster, and confirm your mode (direct or proxy; see Step 0). ingress-nginx keeps running alongside it, so you can move one host at a time.
2. Make sure cert-manager can serve Gateway API. If you use our managed cert-manager, we turn on its Gateway API support together with Envoy Gateway. If you run your own, install it (see Install and upgrade cert-manager) and start it with Gateway API support enabled (--enable-gateway-api), or the HTTP-01 gatewayHTTPRoute solver will not work.
3. Recreate your ingress as Gateway API objects following the guide for your mode (direct or proxy). The standard layout is a shared Gateway in a dedicated gateway namespace, with a matching ClientTrafficPolicy and a TLS Certificate per hostname, that your application namespaces attach to. Each app namespace then only carries the shared-gateway-access: "true" label and one HTTPRoute per host. A single ingress-nginx controller served all your hosts through one entry point before; one shared Gateway is its direct equivalent, keeping your cluster on a single load balancer.
4. Translate annotations. Path rewrites, header manipulation and redirects become HTTPRoute filters; retries and circuit breaking become a BackendTrafficPolicy; auth/CORS become a SecurityPolicy.
5. Switch your cert-manager Issuer to a Gateway-aware solver that issues a certificate per hostname: the http01.gatewayHTTPRoute solver in direct mode, or a DNS-01 Issuer in proxy mode (where public DNS points at your upstream proxy). Each mode guide shows the one for that mode.
6. Test before cutover without touching DNS, by resolving your hostname to the new Gateway address locally:

   GW=$(kubectl -n gateway get gateway shared -o jsonpath='{.status.addresses[0].value}')
   curl -v --resolve your.host.example.com:443:"$GW" https://your.host.example.com/
   

7. Cut over DNS to the Gateway’s load-balancer address. Use a low TTL (≈1 minute) beforehand so the change propagates quickly, and a CNAME where possible so only one record needs updating.
8. Clean up once traffic is confirmed on Envoy Gateway: delete your old Ingress objects, and ask support to remove our managed ingress-nginx from your cluster when nothing uses it any more.

Advanced usage

For more advanced use cases please refer to the documentation provided by each project or contact our support:

• Envoy Gateway overview
• Direct (PROXY-protocol) mode
• Proxy (X-Forwarded-For) mode
• Install and upgrade ingress-nginx
• Gateway API
• Envoy Gateway

2 - Cert-manager and Cloudflare demo

In this guide we will use a Cloudflare managed domain and a our own cert-manager to provide LetsEncrypt certificates for a test deployment.

The guide is suitable if you have a domain connected to a single cluster, and would like a to issue/manage certificates from within kubernetes. The setup below becomes Clusterwider, meaning it will deploy certificates to any namespace specifed.

Prerequisites

• DNS managed on Cloudflare
• Cloudflare API token
• Installed cert-manager. See our guide here.
• Installed IngressController. See our guide here.

Setup ClusterIssuer

Create a file to hold the secret of your api token for your Cloudflare DNS. Then create the ClusterIssuer configuration file adapted for Cloudflare.

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token
  namespace: cert-manager
type: Opaque
stringData:
  api-token: "<your api token>"
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: cloudflare-issuer
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: <your email>
    privateKeySecretRef:
      name: cloudflare-issuer-key
    solvers:
    - dns01:
        cloudflare:
          email: <your email>
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token

kubectl apply -f cloudflare-issuer.yml

The clusterIssuer is soon ready. Example output:

kubectl get clusterissuers.cert-manager.io 
NAME                READY   AGE
cloudflare-issuer   True    6d18h

Expose a workload and secure with Let’s encrypt certificate

In this section we will setup a deployment, with it’s accompanying service and ingress object. The ingress object will request a certificate for test2.domain.ltd, and once fully up and running, should provide https://test2.domain.ltd with a valid letsencrypt certificate.

We’ll use the created ClusterIssuer and let cert-manager request new certificates for any added ingress object. This setup requires the “*” record setup in the DNS provider.

This is how the DNS is setup in this particular example: A A record (“domain.ltd”) points to the loadbalancer IP of the cluster. A CNAME record refers to ("*") and points to the A record above.

This example also specifies the namespace “echo2”.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo2-dep
  namespace: echo2
spec:
  selector:
    matchLabels:
      app: echo2
  replicas: 1
  template:
    metadata:
      labels:
        app: echo2
    spec:
      containers:
      - name: echo2
        image: hashicorp/http-echo
        args:
        - "-text=echo2"
        ports:
        - containerPort: 5678
      securityContext:
        runAsUser: 1001
        fsGroup: 1001
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: echo2
  name: echo2-service
  namespace: echo2
spec:
  ports:
    - protocol: TCP
      port: 5678
      targetPort: 5678
  selector:
    app: echo2
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: echo2-ingress
  namespace: echo2
  annotations:
    cert-manager.io/cluster-issuer: cloudflare-issuer
    kubernetes.io/ingress.class: "nginx"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - test2.domain.ltd
    secretName: test2-domain-tls
  rules:
  - host: test2.domain.ltd
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: echo5-service
            port:
              number: 5678

The DNS challenge and certificate issue process takes a couple of minutes. You can follow the progress by watching:

kubectl events -n cert-manager

Once completed, it shall all be accessible at http://test2.domain.ltd

3 - Change PV StorageClass

This guide details all steps to change storage class of a volume. The instruction can be used to migrate from one storage class to another, while retaining data. For example from 8kto v2-4k.

Prerequisites

• Access to the kubernetes cluster
• Access to Openstack kubernetes Project

Preparation steps

1. Populate variables

   Complete with relevant names for your setup. Then copy/paste them into the terminal to set them as environment variables that will be used throughout the guide. PVC is the

   PVC=test1
   NAMESPACE=default
   NEWSTORAGECLASS=v2-1k
   

2. Fetch and populate the PV name by running:

   PV=$(kubectl get pvc -n $NAMESPACE $PVC -o go-template='{{.spec.volumeName}}')
   

3. Create backup of PVC and PV configurations

   Fetch the PVC and PV configurations and store in /tmp/ for later use:

   kubectl get pvc -n $NAMESPACE $PVC -o yaml | tee /tmp/pvc.yaml
   kubectl get pv  $PV -o yaml | tee /tmp/pv.yaml
   

4. Change VolumeReclaimPolicy

   To avoid deletion of the PV when deleting the PVC, the volume needs to have VolumeReclaimPolicy set to Retain.

   Patch:

   kubectl patch pv $PV -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
   

5. Stop pods from accessing the mounted volume (ie kill pods/scale statefulset/etc..).

6. Delete the PVC.

   kubectl delete pvc -n "$NAMESPACE" "$PVC"
   

Login to Openstack

1. Navigate to: Volumes -> Volumes

2. Make a backup of the volume From the drop-down to the right, select backup. The backup is good practice, not used in the following steps.

3. Change the storage type to desired type. The volume should now or shortly have status Available. Dropdown to the right, Edit volume -> Change volume type:

   • Select your desired storage type
   • Select Migration policy=Ondemand

   The window will close, and the volume will be updated and migrated (to the v2 storage platform) if necessary, by the backend. The status becomes “Volume retyping”. Wait until completed.

   We have a complementary guide here.

Back to kubernetes

1. Release the tie between PVC and PV

   The PV is still referencing its old PVC, in the claimRef, found under spec.claimRef.uid. This UID needs to be nullified to release the PV, allowing it to be adopted by a PVC with correct storageClass.

   Patch claimRef to null:

   kubectl patch pv "$PV" -p '{"spec":{"claimRef":{"namespace":"'$NAMESPACE'","name":"'$PVC'","uid":null}}}'
   

2. The PV StorageClass in kubernetes does not match to its counterpart in Openstack.

   We need to patch the storageClassName reference in the PV:

   kubectl patch pv "$PV" -p '{"spec":{"storageClassName":"'$NEWSTORAGECLASS'"}}'
   

3. Prepare a new PVC with the updated storageClass

   We need to modify the saved /tmp/pvc.yaml.

   1. Remove “last-applied-configuration”:

      sed -i '/kubectl.kubernetes.io\/last-applied-configuration: |/ { N; d; }' /tmp/pvc.yaml
      

   2. Update existing storageClassName to the new one:

      sed -i 's/storageClassName: .*/storageClassName: '$NEWSTORAGECLASS'/g' /tmp/pvc.yaml
      

4. Apply the updated /tmp/pvc.yaml

   kubectl apply -f /tmp/pvc.yaml
   

5. Update the PV to bind with the new PVC

   We must allow the new PVC to bind correctly to the old PV. We need to first fetch the new PVC UID, then patch the PV with the PVC UID so kubernetes understands what PVC the PV belongs to.

   1. Retrieve the new PVC UID:

      PVCUID=$(kubectl get -n "$NAMESPACE" pvc "$PVC" -o custom-columns=UID:.metadata.uid --no-headers)
      

   2. Patch the PV with the new UID of the PVC:

      kubectl patch pv "$PV" -p '{"spec":{"claimRef":{"uid":"'$PVCUID'"}}}'
      

6. Reset the Reclaim Policy of the volume to Delete:

   kubectl patch pv $PV -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'
   

7. Completed.

   • Verify the volume works healthily.
   • Update your manifests to reflect the new storageClassName.

4 - Ingress and cert-manager

Follow along demo

In this piece, we show all steps to expose a web service using an Ingress resource. Additionally, we demonstrate how to enable TLS, by using cert-manager to request a Let’s Encrypt certificate.

Prerequisites

1. A DNS record pointing at the public IP address of your worker nodes. In the examples all references to the domain example.ltd must be replaced by the domain you wish to issue certificates for. Configuring DNS is out of scope for this documentation.
2. For clusters created on or after Kubernetes 1.26 you need to ensure there is a Ingress controller and cert-manager installed.
   • Our guide to ingress-nginx can be found here
   • Our guide to installing cert-manager can be found here
   • Upstream documentation: NGINX Ingress Controller
   • Upstream documentation: cert-manager

Create resources

Create a file called ingress.yaml with the following content:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: my-web-service
  name: my-web-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-web-service
  template:
    metadata:
      labels:
        app: my-web-service
    spec:
      securityContext:
        runAsUser: 1001
        fsGroup: 1001
      containers:
      - image: k8s.gcr.io/serve_hostname
        name: servehostname
        ports:
        - containerPort: 9376
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: my-web-service
  name: my-web-service
spec:
  ports:
  - port: 9376
    protocol: TCP
    targetPort: 9376
  selector:
    app: my-web-service
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-web-service-ingress
  annotations:
    cert-manager.io/issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - example.tld
    secretName: example-tld
  rules:
  - host: example.tld
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-web-service
            port:
              number: 9376

Then create the resources in the cluster by running: kubectl apply -f ingress.yaml

Run kubectl get ingress and you should see output similar to this:

NAME                     CLASS   HOSTS         ADDRESS         PORTS     AGE
my-web-service-ingress   nginx   example.tld   91.197.41.241   80, 443   39s

If not, wait a while and try again. Once you see output similar to the above you should be able to reach your service at http://example.tld.

Exposing TCP services

If you wish to expose TCP services note that the tcp-services is located in the default namespace in our clusters.

Enabling TLS

A simple way to enable TLS for your service is by requesting a certificate using the Let’s Encrypt CA. This only requires a few simple steps.

Begin by creating a file called issuer.yaml with the following content:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # Let's Encrypt ACME server for production certificates
    server: https://acme-v02.api.letsencrypt.org/directory
    # This email address will get notifications if failure to renew certificates happens
    email: valid-email@example.tld
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx

Replace the email address with your own. Then create the Issuer in the cluster by running: kubectl apply -f issuer.yaml

Next edit the file called ingress.yaml from the previous example and make sure the Ingress resource matches the example below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-web-service-ingress
  annotations:
    cert-manager.io/issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - example.tld
    secretName: example-tld
  rules:
  - host: example.tld
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-web-service
            port:
              number: 9376

Make sure to replace all references to example.tld by your own domain. Then update the resources by running: kubectl apply -f ingress.yaml

Wait a couple of minutes and your service should be reachable at https://example.tld with a valid certificate.

Network policies

If you are using network policies you will need to add a networkpolicy that allows traffic from the ingress controller to the temporary pod that performs the HTTP challenge. With the default NGINX Ingress Controller provided by us this policy should do the trick.

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: letsencrypt-http-challenge
spec:
  policyTypes:
  - Ingress
  podSelector:
    matchLabels:
      acme.cert-manager.io/http01-solver: "true"
  ingress:
  - ports:
    - port: http
    from:
    - namespaceSelector:
        matchLabels:
          app.kubernetes.io/name: ingress-nginx

Advanced usage

For more advanced use cases please refer to the documentation provided by each project or contact our support:

• Kubernetes Ingress
• NGINX Ingress Controller
• cert-manager
• Let’s Encrypt

5 - Install and expose ArgoCD

Our clusters ship without ArgoCD installed. Running your own is useful if you want to have full control over GitOps delivery into your cluster.

The guide is based on the upstream Argo CD Helm chart (argo/argo-cd).

Prerequisites

Helm needs to be provided with the correct repository:

1. Setup helm repo

   helm repo add argo https://argoproj.github.io/argo-helm
   

2. Make sure to update repo cache

   helm repo update
   

3. Make sure kubectl is configured for the cluster you want to install into.

4. To reach the ArgoCD UI over HTTPS you need something to route external traffic into the cluster and cert-manager to issue the certificate. This guide keeps the ArgoCD side of that independent of which controller you run, and leaves the controller-specific parts to the guide for the controller you picked. You can use our managed add-ons or install your own:

   • Our guide to ingress-nginx can be found here
   • Our guide to installing cert-manager can be found here
   • More on using Ingress with us here

Install

1. Run Helm install, pinning an explicit chart version. --create-namespace creates the argocd namespace for you:

   helm install argocd argo/argo-cd \
     --namespace argocd \
     --create-namespace \
     --version 10.1.4
   

   A full list of available Helm values is on ArgoCD’s ArtifactHub page.

   Note: If you already know you want the UI reachable via Ingress, you can supply a values.yaml up front instead of running helm upgrade afterwards, see the Expose the ArgoCD UI section below.

Verify the installation

1. Check that all pods are running:

   kubectl get pods -n argocd
   

2. Retrieve the initial admin password:

   kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
   

   Upstream recommends deleting this secret once you have changed the admin password.

3. Log in with the ArgoCD CLI over a port-forward:

   kubectl port-forward svc/argocd-server -n argocd 8080:443
   argocd login localhost:8080 --username admin --insecure
   

   Note: On a fresh install the ArgoCD server presents its own self-signed certificate, so the CLI needs --insecure (or an interactive confirmation) to connect. Once you expose the server through the Ingress below, log in against that hostname with --grpc-web instead, and drop --insecure.

Expose the ArgoCD UI

1. Interim access

   Before you have decided on a hostname, reach the UI with a port-forward:

   kubectl port-forward svc/argocd-server -n argocd 8080:443
   

   The UI is then available at https://localhost:8080.

   Note: Both service ports forward to the same container port, so the scheme follows how the server is configured, not the port you forward. Use https://localhost:8080 on a default install, and http://localhost:8080 once you have set server.insecure: true as described below.

2. Decide where TLS terminates

   The ArgoCD server multiplexes gRPC and HTTP on a single port and terminates TLS itself by default. You therefore have to choose one of two arrangements before writing any routing configuration:

   • Terminate TLS at your ingress. Set server.insecure: true so the server serves plain HTTP behind it. This is the simpler option and what the examples below assume.
   • Keep TLS on the backend. Leave the server as it is and configure your ingress to pass TLS through untouched. How that is expressed depends entirely on which ingress you run.

   configs:
     params:
       server.insecure: true
   

   The upstream ArgoCD ingress documentation covers both arrangements in detail.

3. Route traffic to the server

   The chart can create the routing resource for you in either model. Pick the one your controller implements: Gateway API if you run something like Envoy Gateway, Ingress if you run an ingress controller such as ingress-nginx.

   Gateway API. Point an HTTPRoute at an existing Gateway:

   server:
     httproute:
       enabled: true
       parentRefs:
         - name: <your-gateway>
           namespace: <gateway-namespace>
           sectionName: https
       hostnames:
         - argocd.example.tld
   

   Add a GRPCRoute alongside it if you want the CLI to speak plain gRPC rather than tunnelling it over HTTP:

   server:
     grpcroute:
       enabled: true
       parentRefs:
         - name: <your-gateway>
           namespace: <gateway-namespace>
           sectionName: https
       hostnames:
         - argocd.example.tld
   

   Note: With Gateway API the certificate belongs to the Gateway listener, not to the route, so there is no cert-manager annotation to set here. Configure TLS where the Gateway is defined.

   Ingress. If your controller uses Ingress resources instead:

   server:
     ingress:
       enabled: true
       ingressClassName: <your-ingress-class>
       hostname: argocd.example.tld
       annotations:
         cert-manager.io/issuer: letsencrypt-prod
       tls: true
   

   Note: Keep all annotations in one block. A second server: section further down the same values.yaml does not merge with the first one, so earlier annotations would be lost.

   Either way, apply the values with:

   helm upgrade --install argocd argo/argo-cd --namespace argocd --version 10.1.4 --values values.yaml
   

4. Certificates on the Ingress path

   If you routed with an HTTPRoute, skip this. The certificate is attached to the Gateway listener and is configured wherever that Gateway lives.

   cert-manager.io/issuer refers to an Issuer, which is namespaced. The letsencrypt-prod Issuer from our Ingress guide has to exist in the argocd namespace, so apply it with kubectl apply -f issuer.yaml -n argocd. To keep a single issuer for the whole cluster, create it as a ClusterIssuer and use the cert-manager.io/cluster-issuer annotation instead.

   Note: Our managed cert-manager add-on installs the controller and its CRDs only. It does not create any Issuer or ClusterIssuer for you, so one of the above is always required. Without it the certificate stays False and cert-manager reports Referenced "Issuer" not found.

5. Using the ArgoCD CLI against the exposed server

   If you enabled a GRPCRoute, the CLI can speak gRPC directly and needs no extra flag:

   argocd login argocd.example.tld
   

   On the Ingress path there is no equivalent. An Ingress carries one backend protocol and the route above serves the UI over HTTP, so the CLI has to tunnel gRPC over HTTP instead:

   argocd login argocd.example.tld --grpc-web
   

   --grpc-web works through anything that terminates HTTP, so it is the safe choice if you are unsure. Serving plain gRPC through an Ingress means a second Ingress on its own hostname with a controller-specific backend-protocol setting, which is one of the things Gateway API removes the need for.

6. Restricting who can reach it

   ArgoCD holds credentials for every repository it syncs and is a deployment path into your cluster, so restrict access before pointing a public hostname at it.

   The control that does not depend on your ingress controller is the whitelist on the OpenStack load balancer in front of the cluster. It is part of your cluster configuration rather than something inside the cluster, so contact our support with the source ranges that should be allowed and we will apply it. If you run your own ingress controller, the equivalent is loadBalancerSourceRanges on its LoadBalancer service.

   Note: Your ingress controller may also offer a source-range restriction of its own, such as nginx.ingress.kubernetes.io/whitelist-source-range. It matches on the client address as the controller sees it, which is the real client address only when the controller is given it over PROXY protocol. Whether that holds depends on how the ingress in your cluster is set up, so check with our support before relying on the annotation. Treat it as defence in depth behind a load balancer restriction, never as your only control.

Connect a private Git repository

Add a repository with credentials via the CLI:

argocd repo add <repo-url> --username <user> --password <token>

For Azure DevOps, use a Personal Access Token scoped to Code: Read, or an SSH deploy key instead of username/password.

Alternatively, declare the repository as a Secret labelled for ArgoCD:

apiVersion: v1
kind: Secret
metadata:
  name: private-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  url: <repo-url>
  username: <user>
  password: <token>

Note: Avoid committing credentials in plaintext to git. See Managing secrets below for options to keep this manifest encrypted at rest in your repository.

Pull images from a private registry

Create an imagePullSecret in the namespace your application is deployed to:

kubectl create secret docker-registry regcred \
  --namespace <app-namespace> \
  --docker-server=<registry-server> \
  --docker-username=<user> \
  --docker-password=<password>

This works for any registry (Docker Hub, ACR, GCR, Quay, …). Reference the secret from your workload’s ServiceAccount imagePullSecrets, or set it via your Helm chart’s values.

Managing secrets

Since ArgoCD deploys straight from git, secrets should not be committed in plaintext. Two common options, both customer-installed:

• Sealed Secrets encrypts secrets client-side with kubeseal so only the controller in your cluster can decrypt them, safe to commit to git.
• External Secrets syncs secrets from an external secret store (e.g. Vault, cloud provider secret managers) into Kubernetes Secret resources.

Deploy your first application

Create a file called application.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: <repo-url>
    targetRevision: HEAD
    path: <path-to-manifests>
  destination:
    server: https://kubernetes.default.svc
    namespace: <app-namespace>
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Note: ArgoCD does not create the destination namespace on its own, so a sync into a namespace that does not exist fails. CreateNamespace=true lets ArgoCD create it. Drop the option if you prefer to create the namespace yourself.

If you’re deploying a Helm chart instead of plain manifests, use a Helm source:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: <repo-url>
    targetRevision: HEAD
    path: <path-to-chart>
    helm:
      valueFiles:
        - values.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: <app-namespace>
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Then create the resource in the cluster by running:

kubectl apply -f application.yaml

Upgrade

helm upgrade --install argocd argo/argo-cd --namespace argocd --version <new-version>

Uninstall

1. Remove your Application resources first

   kubectl delete applications --all -n argocd
   

   Note: Applications that carry the resources-finalizer.argocd.argoproj.io finalizer need the ArgoCD controller running to be cleaned up. Delete them before uninstalling the release, or the namespace gets stuck in Terminating with nothing left to clear the finalizer. Without that finalizer the workloads ArgoCD deployed are left running, which is usually what you want when you are only replacing ArgoCD itself.

2. Remove the release

   helm uninstall argocd -n argocd
   

3. Remove the namespace if necessary

   kubectl delete namespace argocd
   

4. Remove the CRDs if you are not reinstalling

   kubectl delete crd applications.argoproj.io applicationsets.argoproj.io appprojects.argoproj.io
   

   Note: The chart sets crds.keep: true, so helm uninstall deliberately leaves the CRDs in place to protect your resources during an upgrade. They are cluster-scoped and outlive the namespace. Deleting a CRD deletes every resource of that type in the cluster, so only do this once you are sure ArgoCD is going away for good.

6 - Install and upgrade cert-manager

Starting at Kubernetes version v1.26, our default configured clusters are delivered without cert-manager.

This guide will assist you get a working up to date cert-manager and provide instructions for how to upgrade and delete it. Running your own is useful if you want to have full control.

The guide is based on cert-manager Helm chart, found here. We draw advantage of the option to install CRDs with kubectl, as recommended for a production setup.

Prerequisites

Helm needs to be provided with the correct repository:

1. Setup helm repo

   helm repo add jetstack https://charts.jetstack.io --force-update
   

2. Verify you do not have a namespace named elx-cert-manager as you first need to remove some resources.

   kubectl -n elx-cert-manager delete svc cert-manager cert-manager-webhook
   kubectl -n elx-cert-manager delete deployments.apps cert-manager cert-manager-cainjector cert-manager-webhook
   kubectl delete namespace elx-cert-manager
   

Install

1. Prepare and install CRDs run:

   kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.4/cert-manager.crds.yaml
   

2. Run Helm install:

   helm install \
     cert-manager jetstack/cert-manager \
     --namespace cert-manager \
     --create-namespace \
     --version v1.14.4 \
   

   A full list of available Helm values is on cert-manager’s ArtifactHub page.

3. Verify the installation: Done with cmctl (cert-manager CLI https://cert-manager.io/docs/reference/cmctl/#installation).

   cmctl check api
   

   If everything is working you should get this message The cert-manager API is ready.

Upgrade

The setup used above is referenced in the topic “CRDs managed separately”.

In these examples <version> is “v1.14.4”.

1. Update CRDS:

   kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/<version>/cert-manager.crds.yaml
   

2. Update the Helm chart:

   helm upgrade cert-manager jetstack/cert-manager --namespace cert-manager --version v1.14.4 
   

Uninstall

To uninstall, use the guide here.

7 - Install and upgrade ingress-nginx

This guide will assist you get a working up to date ingress controller and provide instructions for how to upgrade and delete it. Running your own is useful if you want to have full control.

The guide is based on on ingress-nginx Helm chart, found here.

Prerequisites

Helm needs to be provided with the correct repository:

1. Setup helm repo

   helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
   

2. Make sure to update repo cache

   helm repo update
   

Generate values.yaml

We provide settings for two main scenarios of how clients connect to the cluster. The configuration file, values.yaml, must reflect the correct scenario.

• Customer connects directly to the Ingress:

  controller:
    kind: DaemonSet
    metrics:
      enabled: true
    service:
      enabled: true
      annotations:
        loadbalancer.openstack.org/proxy-protocol: "true"
    ingressClassResource:
      default: true
    publishService:
      enabled: false  
    allowSnippetAnnotations: true
    config:
      use-proxy-protocol: "true"
  defaultBackend:
    enabled: true
  

• Customer connects via Proxy:

  controller:
    kind: DaemonSet
    metrics:
      enabled: true
    service:
      enabled: true
      #loadBalancerSourceRanges:
      #  - <Proxy(s)-CIDR>
    ingressClassResource:
      default: true
    publishService:
      enabled: false  
    allowSnippetAnnotations: true
    config:
      use-forwarded-headers: "true"
  defaultBackend:
    enabled: true
  

• Other useful settings:

  For a complete set of options see the upstream documentation here.

    [...]
    service:
      loadBalancerSourceRanges:        # Whitelist source IPs.
        - 133.124.../32
        - 122.123.../24
      annotations:
        loadbalancer.openstack.org/keep-floatingip: "true"  # retain floating IP in floating IP pool.
        loadbalancer.openstack.org/flavor-id: "48ba211c-20f1-4098-9216-d28f3716a305"  # flavor v1-lb-2, by ID.
    [...]
  

Install ingress-nginx

Use the values.yaml generated in the previous step.

helm install ingress-nginx ingress-nginx/ingress-nginx --values values.yaml --namespace ingress-nginx --create-namespace

Example output:

NAME: ingress-nginx
LAST DEPLOYED: Tue Jul 18 11:26:17 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the Load Balancer IP to become available.
You can watch the status by running 'kubectl --namespace default get services -o wide -w ingress-nginx-controller'
[..]

Upgrade ingress-nginx

Use the values.yaml generated in the previous step.

helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx --values values.yaml --namespace ingress-nginx

Example output:

Release "ingress-nginx" has been upgraded. Happy Helming!
NAME: ingress-nginx
LAST DEPLOYED: Tue Jul 18 11:29:41 2023
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the Load Balancer IP to be available.
You can watch the status by running 'kubectl --namespace default get services -o wide -w ingress-nginx-controller'
[..]

Remove ingress-nginx

The best practice is to use the helm template method to remove the ingress. This allows for proper removal of lingering resources, then remove the namespace. Use the values.yaml generated in the previous step.

Note: Avoid running multiple ingress controllers using the same IngressClass.
See more information here.

1. Run the delete command

   helm template ingress-nginx ingress-nginx/ingress-nginx --values values.yaml --namespace ingress-nginx | kubectl delete -f -
   

2. Remove the namespace if necessary

   kubectl delete namespace ingress-nginx
   

8 - Load balancers

Load balancers in our Elastx Kubernetes CaaS service are provided by OpenStack Octavia in collaboration with the Kubernetes Cloud Provider OpenStack. This article will introduce some of the basics of how to use services of service type LoadBalancer to expose service using OpenStack Octavia load balancers. For more advanced use cases you are encouraged to read the official documentation of each project or contacting our support for assistance.

A quick example

Exposing services using a service with type LoadBalancer will give you an unique public IP backed by an OpenStack Octavia load balancer. This example will take you through the steps for creating such a service.

Create the resources

Create a file called lb.yaml with the following content:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: echoserver
  name: echoserver
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: echoserver
  template:
    metadata:
      labels:
        app.kubernetes.io/name: echoserver
    spec:
      containers:
      - image: gcr.io/google-containers/echoserver:1.10
        name: echoserver
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: echoserver
  name: echoserver
  annotations:
    loadbalancer.openstack.org/x-forwarded-for: "true"
    loadbalancer.openstack.org/flavor-id: 48ba211c-20f1-4098-9216-d28f3716a305
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
    name: http
  selector:
    app.kubernetes.io/name: echoserver
  type: LoadBalancer

Then create the resources in the cluster by running: kubectl apply -f lb.yaml

You can watch the load balancer being created by running: kubectl get svc

This should output something like:

NAME         TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
echoserver   LoadBalancer   10.233.32.83   <pending>     80:30838/TCP   6s
kubernetes   ClusterIP      10.233.0.1     <none>        443/TCP        10h

The output in the EXTERNAL-IP column tells us that the load balancer has not yet been completely created.

We can investigate further by running: kubectl describe svc echoserver

Output should look something like this:

Name:                     echoserver
Namespace:                default
Labels:                   app.kubernetes.io/name=echoserver
Annotations:              loadbalancer.openstack.org/x-forwarded-for: true
Selector:                 app.kubernetes.io/name=echoserver
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.233.32.83
IPs:                      10.233.32.83
Port:                     <unset>  80/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  30838/TCP
Endpoints:
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason                Age   From                Message
  ----    ------                ----  ----                -------
  Normal  EnsuringLoadBalancer  115s  service-controller  Ensuring load balancer

Looking at the Events section near the bottom we can see that the Cloud Controller has picked up the order and is provisioning a load balancer.

Running the same command again (kubectl describe svc echoserver) after waiting some time should produce output like:

Name:                     echoserver
Namespace:                default
Labels:                   app.kubernetes.io/name=echoserver
Annotations:              loadbalancer.openstack.org/x-forwarded-for: true
Selector:                 app.kubernetes.io/name=echoserver
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.233.32.83
IPs:                      10.233.32.83
LoadBalancer Ingress:     91.197.41.223
Port:                     <unset>  80/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  30838/TCP
Endpoints:
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason                Age    From                Message
  ----    ------                ----   ----                -------
  Normal  EnsuringLoadBalancer  8m52s  service-controller  Ensuring load balancer
  Normal  EnsuredLoadBalancer   6m43s  service-controller  Ensured load balancer

Again looking at the Events section we can tell that the Cloud Provider has provisioned the load balancer for us (the EnsuredLoadBalancer event). Furthermore we can see the public IP address associated with the service by checking the LoadBalancer Ingress.

Finally to verify that the load balancer and service are operational run: curl http://<IP address from LoadBalancer Ingress>

Your output should look something like:

Hostname: echoserver-84655f4656-sc4k6

Pod Information:
        -no pod information available-

Server values:
        server_version=nginx: 1.13.3 - lua: 10008

Request Information:
        client_address=10.128.0.3
        method=GET
        real path=/
        query=
        request_version=1.1
        request_scheme=http
        request_uri=http://91.197.41.223:8080/

Request Headers:
        accept=*/*
        host=91.197.41.223
        user-agent=curl/7.68.0
        x-forwarded-for=213.179.7.4

Request Body:
        -no body in request-

Things to note:

• You do not need to modify security groups when exposing services using load balancers.
• The client_address is the address of the load balancer and not the client making the request, you can find the real client address in the x-forwarded-for header.
• The x-forwarded-for header is provided by setting the loadbalancer.openstack.org/x-forwarded-for: "true" on the service. Read more about available annotations in the Advanced usage section.

Advanced usage

For more advanced use cases please refer to the documentation provided by each project or contact our support:

• Kubernetes Cloud Provider OpenStack
• OpenStack Octavia

Good to know

Load balancers are billable resources

Adding services of type LoadBalancer will create load balancers in OpenStack, which is a billable resource and you will be charged for them.

Loadbalancer statuses

Load balancers within OpenStack have two distinct statuses, which may cause confusion regarding their meanings:

• Provisioning Status: This status reflects the overall condition of the load balancer itself. If any issues arise with the load balancer, this status will indicate them. Should you encounter any problems with this status, please don’t hesitate to contact Elastx support for assistance.
• Operating Status: This status indicates the health of the configured backends, typically referring to the nodes within your cluster, especially when health checks are enabled (which is the default setting). It’s important to note that an operational status doesn’t necessarily imply a problem, as it depends on your specific configuration. If a service is only exposed on a single node, for instance, this is to be expected since load balancers by default distribute traffic across all cluster nodes.

Provisioning status codes

Operating status codes

High availability properties

OpenStack Octavia load balancers are placed in two of our three availability zones. This is a limitation imposed by the OpenStack Octavia project.

Reconfiguring using annotations

Reconfiguring the load balancers using annotations is not as dynamic and smooth as one would hope. For now, to change the configuration of a load balancer the service needs to be deleted and a new one created.

Loadbalancer protocols

Loadbalancers have support for multiple protocols. In general we would recommend everyone to try avoiding http and https simply because they do not perform as well as other protocols.

Instead use tcp or haproxys proxy protocol and run an ingress controller thats responsible for proxying within clusters and TLS.

Load Balancer Flavors

Load balancers come in multiple flavors. The biggest difference is how much traffic they can handle. If no flavor is deployed, we default to v1-lb-1. However, this flavor can only push around 200 Mbit/s. For customers wanting to push potentially more, we have a couple of flavors to choose from:

To select a flavor for your Load Balancer, add the following to the Kubernetes Service .metadata.annotations:

loadbalancer.openstack.org/flavor-id: <id-of-your-flavor>

Note that this is a destructive operation when modifying an existing Service; it will remove the current Load Balancer and create a new one (with a new public IP).

Full example configuration for a basic LoadBalancer service:

apiVersion: v1
kind: Service
metadata:
  annotations:
    loadbalancer.openstack.org/flavor-id: b4a85cd7-abe0-41aa-9928-d15b69770fd4
  name: my-loadbalancer
spec:
  ports:
  - name: http-80
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app: my-application
  type: LoadBalancer

9 - Migration to Kubernetes CaaS v2

** Please note this document was updated 20240305.

This document will guide through all new changes introduced when migrating to our new kubernetes deployment backend. All customers with a Kubernetes cluster created on Kubernetes 1.25 and earlier are affected.

We have received, and acted upon, customer feedback since our main announcement 2023Q4. We provide two additional paths to reach v1.26:

• We’ve reverted to continue providing Ingress/Certmanager.
• To assist with your transition we can offer you an additional cluster (v1.26 or latest version) up to 30 days at no extra charge.

All customers will receive this information when we upgrade clusters to v1.26, which also includes the migration procedure. Make sure to carefully read through and understand the procedure and changes in order to avoid potential downtime during the upgrade.

Pre-Upgrade Information:

• The following overall steps are crucial for a seamless upgrade process:

  1. Date for the upgrade is agreed upon.
  2. For users of Elastx managed ingress opting to continue with our management services:
     • Elastx integrates a load balancer into the ingress service. The load balancer is assigned an external IP-address that will be used for all DNS records post-transition (do not point DNS to this IP at this point).
     • Date of the traffic transition to the load balancer is agreed upon.

• Important Note Before the Upgrade:

  • Customers are required to carefully read and comprehend all changes outlined in the migration documentation to avoid potential downtime or disruptions.
  • In case of any uncertainties or challenges completing the steps, please contact Elastx support. We are here to assist and can reschedule the upgrade to a more suitable date if needed.

To facilitate a seamless traffic transition, we recommend the following best practices:

• Utilize CNAMEs when configuring domain pointers for the ingress. This approach ensures that only one record needs updating, enhancing efficiency.
• Prior to implementing the change, verify that the CNAME record has a low Time-To-Live (TTL), with a duration of typically 1 minute, to promote rapid propagation.

During the traffic transition:

• All DNS records or proxies need to be updated to point towards the new loadbalancer
  • In order to make this change as seamless as possible. We recommend the customer to make use of CNAMEs when pointing domains towards the ingress. This would ensure only one record needs to be updated. Prior to the change make sure the CNAME record has a low TTL, usually 1 minute is good to ensure rapid propagation

During the traffic transition:

1. Elastx will meticulously update the ingress service configuration to align with your specific setup.
2. The customer is responsible for updating all DNS records or proxies to effectively direct traffic towards the newly implemented load balancer.

During the Upgrade:

• Elastx assumes all necessary pre-upgrade changes have been implemented unless notified otherwise.
• On the scheduled upgrade day, Elastx initiates the upgrade process at the agreed-upon time.
• Note: The Kubernetes API will be temporarily unavailable during the upgrade due to migration to a load balancer.
• Upgrade Procedure:
  • The upgrade involves replacing all nodes in your cluster twice.
  • Migration to the new cluster management backend system will occur during Kubernetes 1.25, followed by the cluster upgrade to Kubernetes 1.26.

After Successful Upgrade:

• Users are advised to download a new kubeconfig from the object store for continued access and management.

Possibility to get a new cluster instead of migrating

To address the growing demand for new clusters rather than upgrades, customers currently running Kubernetes 1.25 (or earlier) can opt for a new Kubernetes cluster instead of migrating their existing one. The new cluster can be of version 1.26 or the latest available (1.29 at the moment). This new cluster is provided free of charge for an initial 30-day period, allowing you the flexibility to migrate your services at your own pace. However, if the migration extends beyond 30 days, please note that you will be billed for both clusters during the extended period. We understand the importance of a smooth transition, and our support team is available to assist you throughout the process.

Ingress

We are updating the way clusters accept incoming traffic by transitioning from accepting traffic on each worker node to utilizing a load balancer. This upgrade, effective from Kubernetes 1.26 onwards, offers automatic addition and removal of worker nodes, providing enhanced fault management and a single IP address for DNS and/or WAF configuration.

Before upgrading to Kubernetes 1.26, a migration to the new Load Balancer is necessary. See below a flowchart of the various configurations. In order to setup the components correctly we need to understand your configuration specifics. Please review your scenario:

Using Your Own Ingress

If you manage your own add-ons, you can continue doing so. Starting from Kubernetes 1.26, clusters will no longer have public IP addresses on all nodes by default. We strongly recommend implementing a load balancer in front of your ingress for improved fault tolerance, especially in handling issues better than web browsers.

Elastx managed ingress

If you are using the Elastx managed ingress, additional details about your setup are required.

Proxy Deployed in Front of the Ingress (CDN, WAF, etc.)

If a proxy is deployed, provide information on the IP addresses used by your proxy. We rely on this information to trust the x-forwarded- headers. By default, connections that do not come from your proxy are blocked directly on the load balancer, enforcing clients to connect through your proxy.

Clients Connect Directly to Your Ingress

If clients connect directly to the ingress, we will redirect them to the new ingress. To maintain client source IPs, we utilize HAProxy proxy protocol in the load balancer. However, during the change, traffic will only be allowed to the load balancer for approximately 1-2 minutes. Please plan accordingly, as some connections may experience downtime during this transition.

Floating IPs

Floating IPs (FIPs) are now available for customers who choose to opt in. As part of the upgrade to Kubernetes 1.26, floating IPs will be removed from nodes by default. Instead, Load Balancers will be employed to efficiently direct traffic to services within the cluster.

Please note that current floating IPs will be lost if customers do not opt in for this feature during the upgrade process.

Should you wish to continue utilizing Floating IPs or enable them in the future, simply inform us, and we’ll ensure to assist you promptly.

A primary use case where Floating IPs prove invaluable is in retaining control over egress IP from the cluster. Without leveraging FIPs, egress traffic will be SNAT’ed via the hypervisor.

Kubernetes API

We are removing floating IPs for all control-plane nodes. Instead, we use a Load Balancer in front of control-planes to ensure the traffic will be sent to an working control-plane node.

Whitelisting of access to the API server is now controlled in the loadbalancer in front of the API. Currently, managing the IP-range whitelist requires a support ticket here. All Elastx IP ranges are always included.

Node local DNS

During the Kubernetes 1.26 upgrade we stop using the nodelocaldns. However to ensure we does not break any existing clusters the service will remain installed.

All nodes being added to the cluster running Kubernetes 1.26 or later will not make use of nodelocaldns and pods being created on upgraded nodes will instead make use of the CoreDNS service located in kube-system.

This may affect customers that make use of network policies. If the policy only allows traffic to nodelocaldns, it is required to update the policy to also allow traffic from the CoreDNS service.

Network policy to allow CoreDNS and NodeLocalDNS Cache

This example allows DNS traffic towards both NodeLocalDNS and CoreDNS. This policy is recommended for customers currently only allowing DNS traffic towards NodeLocalDNS and can be used in a “transition phase” prior to upgrading to Kubernetes 1.26.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-access
spec:
  podSelector: {}
  egress:
    - ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
      to:
        - ipBlock:
            cidr: 169.254.25.10/32
        - podSelector:
            matchLabels:
              k8s-app: kube-dns
  policyTypes:
    - Egress

Network policy to allow CoreDNS

This example shows an example network policy that allows DNS traffic to CoreDNS. This can be used after the upgrade to Kubernetes 1.26

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-access
spec:
  podSelector: {}
  egress:
    - ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
      to:
        - podSelector:
            matchLabels:
              k8s-app: kube-dns
  policyTypes:
    - Egress

10 - Node labels and taints with elx-nodegroup-controller

The elx-nodegroup-controller lets you declaratively manage labels and taints across groups of nodes in your cluster. Instead of manually patching each node after it joins, you define a NodeGroup resource that describes which nodes to target and what labels and taints to apply. The controller keeps the nodes in sync automatically, even after node replacements due to auto-healing or scaling.

The controller is available as a managed add-on through Elastx and can also be deployed from the public GitHub repository if you prefer to manage it yourself.

When is this useful?

A few common scenarios:

• GPU or specialised hardware nodes — taint dedicated nodes so only workloads that explicitly tolerate the taint are scheduled on them.
• Cost or zone affinity — label nodes by nodegroup so workloads can use nodeAffinity to target specific node types.
• Workload isolation — mark nodes reserved for databases, batch jobs, or frontend replicas using a combination of labels and taints.

Concepts

A NodeGroup resource targets nodes in two ways — you can use one or both in the same resource:

When a NodeGroup is deleted the controller removes only the labels and taints it applied. Labels and taints that were on the node before the NodeGroup existed are left untouched.

Applying labels to a nodegroup

The following example labels all nodes whose name contains the segment gpu (e.g. worker-gpu-a, worker-gpu-b, sto1-gpu-1):

apiVersion: k8s.elx.cloud/v1alpha2
kind: NodeGroup
metadata:
  name: gpu-workers
spec:
  nodeGroupNames:
    - gpu
  labels:
    elastx.cloud/node-type: gpu

Apply it:

kubectl apply -f gpu-nodegroup.yaml

Verify the label was applied:

kubectl get nodes -l elastx.cloud/node-type=gpu

Adding a taint to restrict scheduling

Taints prevent workloads from being scheduled on a node unless they explicitly tolerate the taint. This example taints the same gpu nodes with NoSchedule so that only GPU-aware workloads land on them:

apiVersion: k8s.elx.cloud/v1alpha2
kind: NodeGroup
metadata:
  name: gpu-workers
spec:
  nodeGroupNames:
    - gpu
  labels:
    elastx.cloud/node-type: gpu
  taints:
    - key: elastx.cloud/gpu
      value: "true"
      effect: NoSchedule

A pod that should run on these nodes needs a matching toleration:

tolerations:
  - key: elastx.cloud/gpu
    operator: Equal
    value: "true"
    effect: NoSchedule

Targeting nodes by name

If you want to target specific nodes rather than relying on naming patterns, list them explicitly in spec.members:

apiVersion: k8s.elx.cloud/v1alpha2
kind: NodeGroup
metadata:
  name: database-nodes
spec:
  members:
    - worker-sto1-db-1
    - worker-sto2-db-1
    - worker-sto3-db-1
  taints:
    - key: elastx.cloud/role
      value: database
      effect: NoSchedule
  labels:
    elastx.cloud/role: database

You can mix members and nodeGroupNames in the same resource — the sets are merged and deduplicated automatically.

Good to know

• Reserved label prefixes — the controller will reject a NodeGroup that tries to set labels with the prefixes kubernetes.io/, k8s.io/, node.kubernetes.io/, or node-role.kubernetes.io/. These are reserved for Kubernetes itself.
• Taint effects — valid values are NoSchedule, PreferNoSchedule, and NoExecute.
• Limits — a single NodeGroup supports up to 500 explicit members, 50 name segments, 64 labels, and 100 taints.
• Cleanup on delete — deleting a NodeGroup triggers the controller to remove the labels and taints it applied before the resource is fully removed. The controller uses a finalizer to ensure this happens even if the deletion races with a node replacement.
• Node replacements — when auto-healing replaces a node the new node will receive the correct labels and taints on its first reconciliation, no manual intervention needed.

11 - Nodegroups on a shared network

A nodegroup can be attached to a network in another of your OpenStack projects. Your virtual machines in that project and your cluster can then talk privately, without a floating IP per machine. This is available on clusters running Kubernetes 1.35 or later.

In this guide, the VM project is the OpenStack project that owns the network and your virtual machines, and the cluster is your Kubernetes cluster, which runs in its own project.

How it works

• Each node in the nodegroup gets a second address, on a subnet of the shared network, next to its address on the cluster network.
• Only that nodegroup is on the shared network. Other nodegroups are not affected and cannot reach it.
• A nodegroup can use one shared network, and reaches only one subnet on it. If you need several subnets, use one nodegroup per subnet.
• The network stays in the VM project. You keep full control of it, and of the security groups of your virtual machines.

Prepare the network

In the VM project, create the network and subnet, or pick an existing one. The subnet must:

• Not overlap the cluster’s node, pod or service ranges. If it does, we cannot attach it. A new cluster can be created with a Service range of your choice to avoid an overlap.
• Have room for the nodegroup: one address per node at the nodegroup’s maximum size, plus one address per availability zone the nodegroup uses. Nodes that cannot get an address never join the cluster.
• Have the same MTU as the cluster network. With a smaller MTU, pods on those nodes get a smaller MTU too, and large UDP packets to them are dropped.

Ask us to attach it

Contact support and tell us:

• the VM project, and the network (and subnet, if the network has more than one);
• the cluster and the nodegroup that should use it.

We share the network with the cluster’s project and attach it to the nodegroup. You do not need to share it yourself. The nodegroup’s nodes are replaced once, and no other nodes are replaced.

Let your virtual machines accept traffic from the cluster

Traffic from pods to your virtual machines comes from the nodes’ addresses on the shared subnet, not from the pods’ own addresses. Allow the shared subnet’s range in the security groups of the virtual machines that the cluster should reach.

Reach your Services from the virtual machines

Your virtual machines cannot reach pods directly. Expose a Service through an internal load balancer with its address on the shared subnet:

apiVersion: v1
kind: Service
metadata:
  name: my-service
  annotations:
    service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
    loadbalancer.openstack.org/subnet-id: <shared subnet ID>
    loadbalancer.openstack.org/member-subnet-id: <the cluster's own subnet ID>
spec:
  type: LoadBalancer
  selector:
    app: my-app
  ports:
    - port: 80
      targetPort: 8080

All three annotations are needed, and they must be on the Service when you create it. They are read only when the load balancer is created, so adding them later means deleting and recreating the Service.

• Without openstack-internal-load-balancer, the load balancer gets a public IP and is reachable from the internet.
• Without member-subnet-id, the load balancer accepts connections but cannot reach any node, so every request fails.

Ask support for the cluster’s own subnet ID if you do not have it.

If you use our Envoy Gateway addon, put the same three annotations on your Gateway under spec.infrastructure.annotations. A Gateway with them gets an internal load balancer on the shared subnet, and your other Gateways stay public. Your ClientTrafficPolicy stays the same.

Remove it

1. Delete the Services, and Gateways, that have a load balancer on the shared subnet. While one exists, the network cannot be unshared or deleted.
2. Contact support to detach the network from the nodegroup. Its nodes are replaced once more, without the second address.
3. We stop sharing the network. You can then delete it in the VM project.

Deleting the cluster also deletes its load balancers on the shared subnet.

12 - Persistent volumes

Persistent volumes in our Elastx Kubernetes CaaS service are provided by OpenStack Cinder. Volumes are dynamically provisioned by Kubernetes Cloud Provider OpenStack.

Storage classes

8k refers to 8000 IOPS.

See our pricing page under the table Storage to calculate your costs.

Below is the list of storage classes provided in newly created clusters. In case you see other storageclasses in your cluster, consider these legacy and please migrate data away from them. We provide a guide to Change PV StorageClass.

$ kubectl get storageclasses
NAME              PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
v2-128k           cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   27d
v2-16k            cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   27d
v2-1k (default)   cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   27d
v2-32k            cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   27d
v2-4k             cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   27d
v2-64k            cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   27d
v2-8k             cinder.csi.openstack.org   Delete          WaitForFirstConsumer   true                   27d

Example of PersistentVolumeClaim

A quick example of how to create an unused 1Gi persistent volume claim named example:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: example
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
  storageClassName: v2-16k

$ kubectl get persistentvolumeclaim
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
example   Bound    pvc-f8b1dc7f-db84-11e8-bda5-fa163e3803b4   1Gi        RWO            v2-16k            18s

Good to know

Cross mounting of volumes between nodes

Cross mounting of volumes is not supported! That is a volume can only be mounted by a node residing in the same availability zone as the volume. Plan accordingly for ensured high availability!

Limit of volumes and pods per node

In case higher number of volumes or pods are required, consider adding additional worker nodes.

Encryption

All volumes are encrypted at rest in hardware.

Volume type hostPath

A volume of type hostPath is in reality just a local directory on the specific node being mounted in a pod, this means data is stored locally and will be unavailable if the pod is ever rescheduled on another node. This is expected during cluster upgrades or maintenance, however it may also occur because of other reasons, for example if a pod crashes or a node is malfunctioning. Malfunctioning nodes are automatically healed, meaning they are automatically replaced.

You can read more about hostpath here.

If you are looking for a way to store persistent data we recommend to use PVCs. PVCs can move between nodes within one data-center meaning any data stored will be present even if the pod or node is recreated.

Known issues

Resizing encrypted volumes

Legacy: encrypted volumes do not resize properly, please contact our support if you wish to resize such a volume.

13 - Your first deployment

This page will help you getting a deployment up and running and exposed as a load balancer.

Note: This guide is optional and only here to help new Kubernetes users with an example deployment.

You can verify access by running kubectl get nodes and if the output is similar to the example below you are set to go.

❯ kubectl get nodes
NAME                           STATUS   ROLES           AGE     VERSION
hux-lab1-control-plane-c9bmm   Ready    control-plane   2d18h   v1.27.3
hux-lab1-control-plane-j5p42   Ready    control-plane   2d18h   v1.27.3
hux-lab1-control-plane-wlwr8   Ready    control-plane   2d18h   v1.27.3
hux-lab1-worker-447sn          Ready    <none>          2d18h   v1.27.3
hux-lab1-worker-9ltbp          Ready    <none>          2d18h   v1.27.3
hux-lab1-worker-htfbp          Ready    <none>          15h     v1.27.3
hux-lab1-worker-k56hn          Ready    <none>          16h     v1.27.3

Creating an example deployment

To get started we need a deployment to deploy. Below we have a deployment called echoserver we can use for this example.

1. Start off by creating a file called deployment.yaml with the content of the deployment below:

   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app.kubernetes.io/name: echoserver
     name: echoserver
   spec:
     replicas: 3
     selector:
       matchLabels:
         app.kubernetes.io/name: echoserver
     template:
       metadata:
         labels:
           app.kubernetes.io/name: echoserver
       spec:
         containers:
         - image: gcr.io/google-containers/echoserver:1.10
           name: echoserver
   

2. After you have created your file we can apply the deployment by running the following command:

   ❯ kubectl apply -f deployment.yaml
   deployment.apps/echoserver created
   

3. After running the apply command we can verify that 3 pods have been created. This can take a few seconds.

   ❯ kubectl get pod
   NAME                          READY   STATUS    RESTARTS   AGE
   echoserver-545465d8dc-4bqqn   1/1     Running   0          51s
   echoserver-545465d8dc-g5xxr   1/1     Running   0          51s
   echoserver-545465d8dc-ghrj6   1/1     Running   0          51s
   

Exposing our deployment

After your pods are created we need to make sure to expose our deployment. In this example we are creating a service of type loadbalancer. If you run this application in production you would likely install an ingress controller

1. First of we create a file called service.yaml with the content of the service below

   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app.kubernetes.io/name: echoserver
     name: echoserver
     annotations:
       loadbalancer.openstack.org/x-forwarded-for: "true"
   spec:
     ports:
     - port: 80
       protocol: TCP
       targetPort: 8080
       name: http
     selector:
       app.kubernetes.io/name: echoserver
     type: LoadBalancer
   

2. After creating the service.yaml file we apply it using kubectl

   ❯ kubectl apply -f service.yaml
   service/echoserver created
   

3. We should now be able to use our service by running kubectl get service

   ❯ kubectl get service
   NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
   echoserver   LoadBalancer   10.98.121.166   <pending>     80:31701/TCP   54s
   kubernetes   ClusterIP      10.96.0.1       <none>        443/TCP        2d20h
   

   For the echo service we can see that EXTERNAL-IP says <pending> this means that a load balancer is being created but is not yet ready. As soon as the load balancer is up and running we will instead use an IP address here we can use to access our application.

   Loadbalancers usually take around a minute to be created however can sometimes take a little longer.

4. Once the load balancer is up and running the kubectl get service should return something like this:

   ❯ kubectl get service
   NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
   echoserver   LoadBalancer   10.98.121.166   185.24.134.39   80:31701/TCP   2m24s
   kubernetes   ClusterIP      10.96.0.1       <none>          443/TCP        2d20h