Original article
Highly available Kubernetes with batteries for small business - CINAQ

Plan

  • Build this:
    • Highly available (where possible)
    • Ingress with LetsEncrypt
    • NFS central storage
    • Cluster that scales
    • Monitoring with Prometheus, Grafana and Loki

Glossary

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
ingress
    https://kubernetes.io/docs/concepts/services-networking/ingress/

    Exposes HTTP and HTTPS routes from outside
    the cluster to services within the
    cluster.

    Traffic routing is controlled by rules
    defined on the Ingress resource.

        internet
            |
       [ Ingress ]
       --|-----|--
       [ Services ]

    An Ingress may be configured to give
    Services externally-reachable URLs, load
    balance traffic, terminate SSL / TLS, and
    offer name based virtual hosting.

    An Ingress controller is responsible for
    fulfilling the Ingress, usually with a
    load balancer, though it may also
    configure your edge router or additional
    frontends to help handle the traffic.

    An Ingress does not expose arbitrary ports
    or protocols.

    Exposing services other than HTTP and
    HTTPS to the internet typically uses a
    service of type =Service.Type=NodePort= or
    =Service.Type=LoadBalancer=.

Kubernetes' networking model
    Dictates that Pods must be reachable by
    their IP address across Nodes.

    That is, the IP address of a Pod is always
    visible to other Pods in the network, and
    each Pod views its own IP address as the
    same as how other Pods see it.

    https://www.cuelogic.com/blog/kubernetes-networking-model

Kindie

Supporting tools

letsencrypt

https://letsencrypt.org/

1
2
3
4
5
6
7
Internet Security Research Group
ISRG

Let's Encrypt
    A free, automated, and open certificate
    authority brought to you by the nonprofit
    ISRG.

keepalived

Additional articles
Setting up a Linux cluster with Keepalived: Basic configuration | Enable Sysadmin

Create /etc/keepalived/keepalived.conf.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
vrrp_instance VI_1 {
    state MASTER
    interface ens3
    virtual_router_id 101
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass RANDOM_STRING_HERE
    }
    virtual_ipaddress {
        10.0.1.0
    }
}

Replace RANDOM_STRING_HERE with a strong password of your choice if you want (since this is internal network this is not a very big deal).

It is however necessary to set the correct interface name. You can find it with ip a.

After that we can wrap up with:

1
2
systemctl enable keepalived
systemctl start keepalived

We use the same keepalived.conf for all master nodes so that the active master is randomly selected.

Feel free to adjust the priority if desired to influence the preference.

1
agi keepalived
1
2
3
4
5
6
7
8
Kindie
Kubernetes Individual
    An opinionated k8s cluster setup for
    individuals or small business.

    Batteries included so that you can hit the
    ground running and add production workload
    in no time.
1
ansible-search-install keepalived
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
keepalived
    Routing software written in C.

    The main goal of this project is to
    provide simple and robust facilities for
    loadbalancing and high-availability to
    Linux system and Linux based
    infrastructures.

    Loadbalancing framework relies on well-
    known and widely used IPVS kernel module
    providing Layer4 loadbalancing.

    Keepalived implements a set of checkers to
    dynamically and adaptively maintain and
    manage loadbalanced server pool according
    their health.

    On the other hand high-availability is
    achieved by VRRP protocol.

    VRRP is a fundamental brick for router
    failover.

    In addition, Keepalived implements a set
    of hooks to the VRRP finite state machine
    providing low-level and high-speed
    protocol interactions.

    In order to offer fastest network failure
    detection, Keepalived implements BFD
    protocol.

    VRRP state transition can take into
    account BFD hint to drive fast state
    transition.

    Keepalived frameworks can be used
    independently or all together to provide
    resilient infrastructures.

asciinema recording

kubeadm

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Kubeadm
    A tool built to provide kubeadm init and
    kubeadm join as best-practice β€œfast paths”
    for creating k8s clusters.

    kubeadm performs the actions necessary to
    get a minimum viable cluster up and
    running.

    By design, it cares only about
    bootstrapping, not about provisioning
    machines.

    Likewise, installing various nice-to-have
    addons, like the k8s Dashboard, monitoring
    solutions, and cloud-specific addons, is
    not in scope.

    Instead, we expect higher-level and more
    tailored tooling to be built on top of
    kubeadm, and ideally, using kubeadm as the
    basis of all deployments will make it
    easier to create conformant clusters.

Kubernetes Cluster

master nodes

  • 3 nodes running:
    • node1,
    • node2,
    • node3.

Prepare them all with keepalived and kubeadm.

For each node login over SSH to it using the ops username and password you used during installation. After you login switch to root user with sudo su and enter your password again.