learning
/
learn_k8s
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

adding metallb and ingress stuff

Browse Source
master
Drew Bednar 4 years ago
parent 668d313ccf
commit 6131a2e62a
7 changed files with 3895 additions and 0 deletions
Show Stats Download Patch File Download Diff File

146
README.md
Unescape Escape View File

@ -192,3 +192,149 @@ If the cloud environment supports it you should be able to edit the `spec.type`
This builds on top of `NodePort` and your cloud provider create a new load balancerand direct it at This builds on top of `NodePort` and your cloud provider create a new load balancerand direct it at
nodes in your cluster. This should eventually assign an EXTERNAL-IP with a public IP(or hostname) nodes in your cluster. This should eventually assign an EXTERNAL-IP with a public IP(or hostname)
assigned by the cloud vendor. assigned by the cloud vendor.
## Ingress
Users must install thier own Ingress controller. Controllers are "Plugable". You can use a software(Nginx, Envoy), hardware(F5?), or cloud (ELB)
load balancer as the Ingress controller.
So Ingress is split into two sections:
Ingress Spec - The set of objects you write.
Ingress Controler - The component that acts on the spec objects. Generally needs to translate the spec into the expected config for the chosen controller.
This book uses the [Contour(Envoy under the hood) project](https://projectcontour.io/getting-started/).
I have elected to just use the comparable command found in the book. There are other options like an Operator and Helm chart.
```
kubectl apply -f https://projectcontour.io/quickstart/contour.yaml
```
I found the namespace with `kk get namespaces` which turned out to be `projectcontour`.
```
drewbednar@eisenhorn learn_k8s % kk get pods -n projectcontour
NAME READY STATUS RESTARTS AGE
contour-7dd74cc485-sg9tv 1/1 Running 0 3m23s
contour-7dd74cc485-t4j57 1/1 Running 0 3m23s
contour-certgen-v1.16.0-jtnqg 0/1 Completed 0 3m24s
envoy-762hj 2/2 Running 0 3m23s
envoy-v5nt4 1/2 Running 0 3m23s
```
```
drewbednar@eisenhorn learn_k8s % kk get -n projectcontour services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
contour ClusterIP 10.103.217.239 <none> 8001/TCP 4m15s
envoy LoadBalancer 10.96.170.238 <pending> 80:31325/TCP,443:31149/TCP 4m15s
```
### Need a baremetal load balancer
Well fuck looks like since I am on bare metal I don't have a Load Balancer type. [MetalLB](https://metallb.universe.tf/) is what
DR uses so...I guess that's what I am going to implement.
Following the instructions here https://metallb.universe.tf/installation/.
```
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/metallb.yaml
```
This will give us a metalLB system in the namespace metallb-system that is not yet configured.
There was a forum post [Unifi bgp config] that mentioned this but I think I am just going to use L2
config.
Since my network is 10.0.1.1 with subnet mask of 255.255.252.0 and my DHCP range starts at 10.0.1.50
I can set addresses in the block 10.0.1.40-10.0.1.45 like so:
```
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 10.0.1.40-10.0.1.45
```
```
kubectl apply -f metallb-l2-config.yaml
```
and we can check on it
```
kubectl get configmap -n metallb-system config -o yaml
```
Finally we are back in business
```
drewbednar@eisenhorn networking % kk get services -n projectcontour
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
contour ClusterIP 10.103.217.239 <none> 8001/TCP 163m
envoy LoadBalancer 10.96.170.238 10.0.1.40 80:31325/TCP,443:31149/TCP 163m
```
### Configuring DNS (or something close)
Since I do not have DNS in my lab environment I achieve the same kind of routing to LoadBalancer using the /etc/host of my
router on 10.0.1.1
```
10.0.1.40 alpaca.runcible.io bandicoot.runcible.io
```
### Back to ingress
In the networking folder we have `simple-ingress.yaml`
```
kubectl apply -f networking/simple-ingress.yaml
```
verified with
```
kubectl get ingress
```
In detail
```
kubectl describe ingress simple-ingress
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
Name: simple-ingress
Namespace: default
Address: 10.0.1.40
Default backend: alpaca:8080 (192.168.2.88:8080,192.168.9.160:8080,192.168.9.161:8080)
Rules:
Host Path Backends
---- ---- --------
* * alpaca:8080 (192.168.2.88:8080,192.168.9.160:8080,192.168.9.161:8080)
Annotations: <none>
Events: <none>
```
# Appendix
Restart all the pods in a namespace
```
kubectl -n {NAMESPACE} rollout restart deploy
```
https://cloudowski.com/articles/a-recipe-for-on-prem-kubernetes/
https://metallb.universe.tf/
[unifi bgp config]: https://community.ui.com/questions/How-to-configure-BGP-routing-on-USG-Pro-4/ecdfecb5-a8f5-48a5-90da-cc68d054be11

3167
networking/contour.yaml
View File

File diff suppressed because it is too large Load Diff

113
networking/metallb/bgp-example-config.yaml
Unescape Escape View File

@ -0,0 +1,113 @@
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
# The peers section tells MetalLB what BGP routers to connect too. There
# is one entry for each router you want to peer with.
peers:
- # The target IP address for the BGP session.
peer-address: 10.0.0.1
# The BGP AS number that MetalLB expects to see advertised by
# the router.
peer-asn: 64512
# The BGP AS number that MetalLB should speak as.
my-asn: 64512
# (optional) the TCP port to talk to. Defaults to 179, you shouldn't
# need to set this in production.
peer-port: 179
# (optional) The proposed value of the BGP Hold Time timer. Refer to
# BGP reference material to understand what setting this implies.
hold-time: 120s
# (optional) The router ID to use when connecting to this peer. Defaults
# to the node IP address. Generally only useful when you need to peer with
# another BGP router running on the same machine as MetalLB.
router-id: 1.2.3.4
# (optional) Password for TCPMD5 authenticated BGP sessions
# offered by some peers.
password: "yourPassword"
# (optional) The nodes that should connect to this peer. A node
# matches if at least one of the node selectors matches. Within
# one selector, a node matches if all the matchers are
# satisfied. The semantics of each selector are the same as the
# label- and set-based selectors in Kubernetes, documented at
# https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/.
# By default, all nodes are selected.
node-selectors:
- # Match by label=value
match-labels:
kubernetes.io/hostname: prod-01
# Match by 'key OP values' expressions
match-expressions:
- key: beta.kubernetes.io/arch
operator: In
values: [amd64, arm]
# The address-pools section lists the IP addresses that MetalLB is
# allowed to allocate, along with settings for how to advertise
# those addresses over BGP once assigned. You can have as many
# address pools as you want.
address-pools:
- # A name for the address pool. Services can request allocation
# from a specific address pool using this name, by listing this
# name under the 'metallb.universe.tf/address-pool' annotation.
name: my-ip-space
# Protocol can be used to select how the announcement is done.
# Supported values are bgp and layer2.
protocol: bgp
# A list of IP address ranges over which MetalLB has
# authority. You can list multiple ranges in a single pool, they
# will all share the same settings. Each range can be either a
# CIDR prefix, or an explicit start-end range of IPs.
addresses:
- 198.51.100.0/24
- 192.168.0.150-192.168.0.200
# (optional) If true, MetalLB will not allocate any address that
# ends in .0 or .255. Some old, buggy consumer devices
# mistakenly block traffic to such addresses under the guise of
# smurf protection. Such devices have become fairly rare, but
# the option is here if you encounter serving issues.
avoid-buggy-ips: true
# (optional, default true) If false, MetalLB will not automatically
# allocate any address in this pool. Addresses can still explicitly
# be requested via loadBalancerIP or the address-pool annotation.
auto-assign: false
# (optional) A list of BGP advertisements to make, when
# protocol=bgp. Each address that gets assigned out of this pool
# will turn into this many advertisements. For most simple
# setups, you'll probably just want one.
#
# The default value for this field is a single advertisement with
# all parameters set to their respective defaults.
bgp-advertisements:
- # (optional) How much you want to aggregate up the IP address
# before advertising. For example, advertising 1.2.3.4 with
# aggregation-length=24 would end up advertising 1.2.3.0/24.
# For the majority of setups, you'll want to keep this at the
# default of 32, which advertises the entire IP address
# unmodified.
aggregation-length: 32
# (optional) The value of the BGP "local preference" attribute
# for this advertisement. Only used with IBGP peers,
# i.e. peers where peer-asn is the same as my-asn.
localpref: 100
# (optional) BGP communities to attach to this
# advertisement. Communities are given in the standard
# two-part form <asn>:<community number>. You can also use
# alias names (see below).
communities:
- 64512:1
- no-export
# (optional) BGP community aliases. Instead of using hard to
# read BGP community numbers in address pool advertisement
# configurations, you can define alias names here and use those
# elsewhere in the configuration. The "no-export" community used
# above is defined below.
bgp-communities:
# no-export is a well-known BGP community that prevents
# re-advertisement outside of the immediate autonomous system,
# but people don't usually recognize its numerical value. :)
no-export: 65535:65281

12
networking/metallb/metallb-l2-config.yaml
Unescape Escape View File

@ -0,0 +1,12 @@
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 10.0.1.40-10.0.1.45

443
networking/metallb/metallb.yaml
Unescape Escape View File

@ -0,0 +1,443 @@
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
labels:
app: metallb
name: controller
namespace: metallb-system
spec:
allowPrivilegeEscalation: false
allowedCapabilities: []
allowedHostPaths: []
defaultAddCapabilities: []
defaultAllowPrivilegeEscalation: false
fsGroup:
ranges:
- max: 65535
min: 1
rule: MustRunAs
hostIPC: false
hostNetwork: false
hostPID: false
privileged: false
readOnlyRootFilesystem: true
requiredDropCapabilities:
- ALL
runAsUser:
ranges:
- max: 65535
min: 1
rule: MustRunAs
seLinux:
rule: RunAsAny
supplementalGroups:
ranges:
- max: 65535
min: 1
rule: MustRunAs
volumes:
- configMap
- secret
- emptyDir
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
labels:
app: metallb
name: speaker
namespace: metallb-system
spec:
allowPrivilegeEscalation: false
allowedCapabilities:
- NET_RAW
allowedHostPaths: []
defaultAddCapabilities: []
defaultAllowPrivilegeEscalation: false
fsGroup:
rule: RunAsAny
hostIPC: false
hostNetwork: true
hostPID: false
hostPorts:
- max: 7472
min: 7472
- max: 7946
min: 7946
privileged: true
readOnlyRootFilesystem: true
requiredDropCapabilities:
- ALL
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- configMap
- secret
- emptyDir
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app: metallb
name: controller
namespace: metallb-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app: metallb
name: speaker
namespace: metallb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app: metallb
name: metallb-system:controller
rules:
- apiGroups:
- ''
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- services/status
verbs:
- update
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
- apiGroups:
- policy
resourceNames:
- controller
resources:
- podsecuritypolicies
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app: metallb
name: metallb-system:speaker
rules:
- apiGroups:
- ''
resources:
- services
- endpoints
- nodes
verbs:
- get
- list
- watch
- apiGroups: ["discovery.k8s.io"]
resources:
- endpointslices
verbs:
- get
- list
- watch
- apiGroups:
- ''
resources:
- events
verbs:
- create
- patch
- apiGroups:
- policy
resourceNames:
- speaker
resources:
- podsecuritypolicies
verbs:
- use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app: metallb
name: config-watcher
namespace: metallb-system
rules:
- apiGroups:
- ''
resources:
- configmaps
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app: metallb
name: pod-lister
namespace: metallb-system
rules:
- apiGroups:
- ''
resources:
- pods
verbs:
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app: metallb
name: controller
namespace: metallb-system
rules:
- apiGroups:
- ''
resources:
- secrets
verbs:
- create
- apiGroups:
- ''
resources:
- secrets
resourceNames:
- memberlist
verbs:
- list
- apiGroups:
- apps
resources:
- deployments
resourceNames:
- controller
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app: metallb
name: metallb-system:controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: metallb-system:controller
subjects:
- kind: ServiceAccount
name: controller
namespace: metallb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app: metallb
name: metallb-system:speaker
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: metallb-system:speaker
subjects:
- kind: ServiceAccount
name: speaker
namespace: metallb-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app: metallb
name: config-watcher
namespace: metallb-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: config-watcher
subjects:
- kind: ServiceAccount
name: controller
- kind: ServiceAccount
name: speaker
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app: metallb
name: pod-lister
namespace: metallb-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: pod-lister
subjects:
- kind: ServiceAccount
name: speaker
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app: metallb
name: controller
namespace: metallb-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: controller
subjects:
- kind: ServiceAccount
name: controller
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app: metallb
component: speaker
name: speaker
namespace: metallb-system
spec:
selector:
matchLabels:
app: metallb
component: speaker
template:
metadata:
annotations:
prometheus.io/port: '7472'
prometheus.io/scrape: 'true'
labels:
app: metallb
component: speaker
spec:
containers:
- args:
- --port=7472
- --config=config
env:
- name: METALLB_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: METALLB_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: METALLB_ML_BIND_ADDR
valueFrom:
fieldRef:
fieldPath: status.podIP
# needed when another software is also using memberlist / port 7946
# when changing this default you also need to update the container ports definition
# and the PodSecurityPolicy hostPorts definition
#- name: METALLB_ML_BIND_PORT
# value: "7946"
- name: METALLB_ML_LABELS
value: "app=metallb,component=speaker"
- name: METALLB_ML_SECRET_KEY
valueFrom:
secretKeyRef:
name: memberlist
key: secretkey
image: quay.io/metallb/speaker:v0.10.2
name: speaker
ports:
- containerPort: 7472
name: monitoring
- containerPort: 7946
name: memberlist-tcp
- containerPort: 7946
name: memberlist-udp
protocol: UDP
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_RAW
drop:
- ALL
readOnlyRootFilesystem: true
hostNetwork: true
nodeSelector:
kubernetes.io/os: linux
serviceAccountName: speaker
terminationGracePeriodSeconds: 2
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: metallb
component: controller
name: controller
namespace: metallb-system
spec:
revisionHistoryLimit: 3
selector:
matchLabels:
app: metallb
component: controller
template:
metadata:
annotations:
prometheus.io/port: '7472'
prometheus.io/scrape: 'true'
labels:
app: metallb
component: controller
spec:
containers:
- args:
- --port=7472
- --config=config
env:
- name: METALLB_ML_SECRET_NAME
value: memberlist
- name: METALLB_DEPLOYMENT
value: controller
image: quay.io/metallb/controller:v0.10.2
name: controller
ports:
- containerPort: 7472
name: monitoring
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- all
readOnlyRootFilesystem: true
nodeSelector:
kubernetes.io/os: linux
securityContext:
runAsNonRoot: true
runAsUser: 65534
serviceAccountName: controller
terminationGracePeriodSeconds: 0

6
networking/metallb/namespace.yaml
Unescape Escape View File

@ -0,0 +1,6 @@
apiVersion: v1
kind: Namespace
metadata:
name: metallb-system
labels:
app: metallb

8
networking/simple-ingress.yaml
Unescape Escape View File

@ -0,0 +1,8 @@
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: simple-ingress
spec:
backend:
serviceName: alpaca
servicePort: 8080
Write Preview
Loading…
Cancel
Save