Application start delay in k8s containers
Stress/Load testing applicable. When you want to sping up a lot of pods; the application inside ech pod must start simultaneously with others.
The main problem is cluster autoscaling is slow, it means additional nodes will start with delay.
AWS specific case
The idea here: prescale more nodes than required, so cluster autoscaling will not spend time to spin up new nodes
Assume we have EKS with unmanaged Node Groups as ASG. By updating Minimum capacity > 0 you will achieve, more ‘warm’ nodes that can host extra amount of pods.
Generic case
The idea is to use configmap as file mounted to each pod. It acts as a trigger.
After changing config map, all pod will get new file content. It can be used as a condition for some action.
Ensure that all the pods started, so after you can change configmap content.
Typical configmap
1
2
3
4
5
6
7
|
apiVersion: v1
kind: ConfigMap
metadata:
name: my-configmap
data:
file.wait: "false"
---
|
Deployment with 20 pods, the configmap mounted as a volume to each of them
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
|
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-deployment
spec:
selector:
matchLabels:
app: my-deployment
replicas: 50
template:
metadata:
labels:
app: my-deployment
spec:
containers:
- name: my-container
# Change image source
image: my.registry.com
imagePullPolicy: Always
env:
- name: ENV1
value: "false"
- name: ENV2
value: "true"
resources:
requests:
cpu: "1"
memory: "2Gi"
ephemeral-storage: "5Gi"
limits:
cpu: "2"
memory: "2Gi"
volumeMounts:
- name: ephemeral
mountPath: "/tmp"
- name: wait
mountPath: "/tmp/wait"
volumes:
- name: ephemeral
emptyDir: {}
- name: wait
configMap:
name: my-configmap
nodeSelector:
kubernetes.io/arch: amd64
role: my-role
|
Entrypoint script modification
Case 1. checks for a specific file to be present in a directory.
1
2
3
4
5
6
7
8
9
10
11
12
|
file="/tmp/file.wait"
echo "[INFO] Waiting for $file"
while true; do
if [ -f "$file" ]; then
echo "[INFO] $file exists"
break
fi
sleep 1
done
# Main app
/app/myscript.sh
|
Case 2. checks for a specific content to be present in a file.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
file="/tmp/file.wait"
keyword="demo"
if [ -f "$file" ]; then
echo "[INFO] Waiting for $keyword in $file"
while true; do
if [[ "$(cat $file)" == "$keyword" ]]; then
echo "[INFO] $file has $keyword"
break
fi
sleep 1
done
fi
# Main app
/app/myscript.sh
|
Experimental, does not properly work in some cases
1
2
3
|
file="/tmp/file.wait"
keyword="demo"
tail -f -n 1 "$file" | stdbuf -o0 grep -q --line-buffered "$keyword"
|
