You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Browse filesBrowse the repository at this point in the historyBrowse files
authored
Updates to match the Karpenter v1.0.1 release (#16)
* up to graviton deployed and tested
* userdata outstanding
* key intent, intent In [userdata] not in intent In [apps]
* for review
* simplified user data
* disruption budget - to review
* Updates to the Disruption Budgets blueprint
* Updates to the Update Nodes with Drift blueprint
* Updates Karpenter version
* Updates to the Disruption Budgetrs blueprint
---------
Co-authored-by: jakeskyaws <jakesky@amazon.com>
Co-authored-by: EC2 Default User <ec2-user@ip-172-31-37-217.eu-west-1.compute.internal>
Co-authored-by: Christian Melendez <chrismld@amazon.com>
After you have a cluster up and running with Karpenter installed, you can start testing each blueprint. A blueprint might have a NodePool, EC2NodeClass and a workload example. You need to open the blueprint folder and follow the steps to deploy the resources needed to test the blueprint.
123
+
After you have a cluster up and running with Karpenter installed, you can start testing each blueprint. A blueprint might have a `NodePool`, `EC2NodeClass` and a workload example. You need to open the blueprint folder and follow the steps to deploy the resources needed to test the blueprint.
124
124
125
125
Here's the list of blueprints we have so far:
126
126
@@ -146,10 +146,10 @@ The following table describes the list of resources along with the versions wher
Copy file name to clipboardExpand all lines: blueprints/batch-jobs/README.md
+21-19Lines changed: 21 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,4 +1,4 @@
1
-
# Karpenter Blueprint: Protecting batch jobs during the consolidation process
1
+
# Karpenter Blueprint: Protecting batch jobs during the disruption (consolidation) process
2
2
3
3
## Purpose
4
4
Karpenter can actively reduce the cluster cost by identifying when nodes can be removed or replaced because they are empty or there are a cheaper one available after some workload change. This process is called [consolidation](https://karpenter.sh/preview/concepts/disruption/#consolidation), and it implies the disruption of pods that are running in the node, if any, as they need to be rescheduled into another node. In some cases, like when running long batch jobs, you don't want those pods to be disrupted. You want to run them from start to finish without disruption, and replace or delete the node once they finish. To achieve that, you can set the `karpenter.sh/do-not-disrupt: "true"` annotation on the pod (more information [here](https://karpenter.sh/preview/concepts/disruption/#pod-level-controls)). By opting pods out of this disruption, you are telling Karpenter that it should not voluntarily remove a node containing this pod.
@@ -50,15 +50,15 @@ Now, the total number of vCPU required by the running pods are **4 vCPU**:
50
50
- NGINX server - 2 vCPU required
51
51
- 5-minutes job - 2 vCPU required
52
52
53
-
The default behaviour is the one defined in the NodePool: `consolidationPolicy: WhenUnderutilized`. Karpenter identifies the **c6g.4xlarge** (12 vCPU) is underutilized, and performs a consolidation replacement of the node. It launches a cheaper and smaller node: a **c6g.2xlarge** (8 vCPU) instance. You can check these logs by executing the following command in another terminal:
53
+
The default behaviour is the one defined in the NodePool: `consolidationPolicy: WhenEmptyOrUnderutilized`. Karpenter identifies the **c6g.4xlarge** (12 vCPU) is underutilized, and performs a consolidation replacement of the node. It launches a cheaper and smaller node: a **c6g.2xlarge** (8 vCPU) instance. You can check these logs by executing the following command in another terminal:
{"level":"INFO","time":"2024-01-10T15:06:37.063Z","logger":"controller.disruption","message":"disrupting via consolidation replace, terminating 1 candidates ip-10-0-93-19.eu-west-1.compute.internal/c6g.4xlarge/on-demand and replacing with on-demand node from types r6gd.2xlarge, c7i.2xlarge, r5a.2xlarge, m5a.2xlarge, m6a.2xlarge and 37 other(s)","commit":"1072d3b"}
59
+
{"level":"INFO","time":"2024-08-16T10:03:46.529Z","logger":"controller","message":"disrupting nodeclaim(s) via replace, terminating 1 nodes (2 pods) ip-10-0-122-231.eu-west-2.compute.internal/c6g.4xlarge/on-demand and replacing with on-demand node from types c6g.2xlarge, c7g.2xlarge, m6g.2xlarge, c6a.2xlarge, c5a.2xlarge and 32 other(s)","commit":"5bdf9c3","controller":"disruption","namespace":"","name":"","reconcileID":"857f7bb5-a482-48e8-9c52-16a10823e2e4","command-id":"25beb85a-3020-4267-a525-5273e0afc7a7","reason":"underutilized"}
The NGINX server and the 5-min job pods are rescheduled into the new c6g.2xlarge node, so **the job is restarted**, which will cause a disruption the job might not be prepared to handle like doing a checkpoint.
64
64
@@ -106,9 +106,9 @@ You should see something similar to this, where a new node just appeared:
You should see the following events indicating that Karpenter identified the need of a new node, and that it selected an instance type and purchase option:
{"level":"INFO","time":"2024-01-08T10:42:35.190Z","logger":"controller.provisioner","message":"computed new nodeclaim(s) to fit pod(s)","commit":"1072d3b","nodeclaims":1,"pods":3}
140
+
{"level":"INFO","time":"2024-08-16T10:10:47.683Z","logger":"controller","message":"computed new nodeclaim(s) to fit pod(s)","commit":"5bdf9c3","controller":"provisioner","namespace":"","name":"","reconcileID":"d8e8907d-5b93-46bb-893a-63520f3ec12f","nodeclaims":1,"pods":1}
141
141
142
-
{"level":"INFO","time":"2024-01-08T10:42:35.224Z","logger":"controller.provisioner","message":"created nodeclaim","commit":"1072d3b","nodepool":"default","nodeclaim":"default-xzkfq","requests":{"cpu":"11260m","memory":"290Mi","pods":"8"},"instance-types":"c3.4xlarge, c3.8xlarge, c4.4xlarge, c5.4xlarge, c5a.12xlarge and 95 other(s)"}
142
+
{"level":"INFO","time":"2024-08-16T10:10:47.699Z","logger":"controller","message":"created nodeclaim","commit":"5bdf9c3","controller":"provisioner","namespace":"","name":"","reconcileID":"d8e8907d-5b93-46bb-893a-63520f3ec12f","NodePool":{"name":"default"},"NodeClaim":{"name":"default-g4kgp"},"requests":{"cpu":"7260m","memory":"290Mi","pods":"6"},"instance-types":"c4.2xlarge, c5.2xlarge, c5.4xlarge, c5a.2xlarge, c5a.4xlarge and 55 other(s)"}
@@ -172,7 +172,9 @@ Now, the total number of vCPU required by the running pods are **4 vCPU**:
172
172
```
173
173
$> kubectl describe node <node_name>
174
174
...
175
-
Normal DisruptionBlocked 36s karpenter Cannot disrupt Node: Pod "default/5-min-job-9jc4b" has do not evict annotation
175
+
Normal NodeReady 6m7s kubelet Node ip-10-0-97-15.eu-west-1.compute.internal status is now: NodeReady
176
+
Normal DisruptionBlocked 4m12s karpenter Cannot disrupt Node: pod "default/2-min-job-2fssd" has "karpenter.sh/do-not-disrupt" annotation
177
+
Normal DisruptionBlocked 2m12s karpenter Cannot disrupt Node: pod "default/5-min-job-7pqdt" has "karpenter.sh/do-not-disrupt" annotation
176
178
```
177
179
178
180
### Consolidation Replace allowed after last job finishes
@@ -184,21 +186,21 @@ NAME COMPLETIONS DURATION AGE
184
186
```
185
187
Now, **it is possible to replace the node** by a cheaper and smaller instance because the the NGINX server can be disrupted as it does't contain the `karpenter.sh/do-not-disrupt: "true"` annotation. You can check this in the Karpenter logs terminal:
186
188
```
187
-
{"level":"INFO","time":"2024-01-08T10:48:46.480Z","logger":"controller.disruption","message":"disrupting via consolidation replace, terminating 1 candidates ip-10-0-47-60.eu-west-1.compute.internal/c6g.4xlarge/on-demand and replacing with on-demand node from types c5n.2xlarge, r6a.2xlarge, m6a.2xlarge, m6id.xlarge, m7gd.xlarge and 103 other(s)","commit":"1072d3b"}
189
+
{"level":"INFO","time":"2024-08-16T10:17:21.322Z","logger":"controller","message":"created nodeclaim","commit":"5bdf9c3","controller":"disruption","namespace":"","name":"","reconcileID":"1135db0e-45ef-4529-9492-63789a9837c6","NodePool":{"name":"default"},"NodeClaim":{"name":"default-9m4bv"},"requests":{"cpu":"2260m","memory":"290Mi","pods":"6"},"instance-types":"c4.xlarge, c5.xlarge, c5a.xlarge, c5d.xlarge, c5n.xlarge and 32 other(s)"}
Copy file name to clipboardExpand all lines: blueprints/custom-ami/README.md
+15-6Lines changed: 15 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -31,21 +31,30 @@ kubectl apply -f .
31
31
32
32
Here's the important configuration block within the spec of an [`EC2NodeClass`](https://karpenter.sh/preview/concepts/nodeclasses/#specamiselectorterms): **spec.amiSelectorTerms**
33
33
34
-
`spec.amiSelectorTerms` are used to configure custom AMIs for Karpenter to use, where the AMIs are discovered through ids, owners, name, and tags. This field is optional, and Karpenter will use the latest **EKS-optimized AMIs** for the `AMIFamily` if no `amiSelectorTerms` are specified. To select an AMI by name, use the `name` field in the selector term. To select an AMI by id, use the `id` field in the selector term. To ensure that AMIs are owned by the expected owner, use the `owner` field - you can use a combination of account aliases (e.g. self amazon, your-aws-account-name) and account IDs. If this is not set, it defaults to `self,amazon`.
34
+
`amiSelectorTerms` are required and are used to configure AMIs for Karpenter to use. AMIs are discovered through alias, id, owner, name, and [tags](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html).
35
+
36
+
If amiSelectorTerms match more than one AMI, Karpenter will automatically determine which AMI best fits the workloads on the launched worker node under the following constraints:
37
+
38
+
- When launching nodes, Karpenter automatically determines which architecture a custom AMI is compatible with and will use images that match an instanceType’s requirements.
39
+
- Unless using an alias, Karpenter cannot detect requirements other than architecture. If you need to specify different AMIs for different kind of nodes (e.g. accelerated GPU AMIs), you should use a separate EC2NodeClass.
40
+
- If multiple AMIs are found that can be used, Karpenter will choose the latest one.
41
+
- If no AMIs are found that can be used, then no nodes will be provisioned.
42
+
43
+
To select an AMI by name, use the `name` field in the selector term. To select an AMI by id, use the `id` field in the selector term. To ensure that AMIs are owned by the expected owner, use the `owner` field - you can use a combination of account aliases (e.g. self amazon, your-aws-account-name) and account IDs. If this is not set, it defaults to `self,amazon`.
35
44
36
45
> **Tip**
37
46
> AMIs may be specified by any AWS tag, including Name. Selecting by tag
38
47
> or by name using wildcards (*) is supported.
39
48
40
49
```
41
50
amiSelectorTerms:
42
-
- name: "*amazon-eks-node-1.27-*"
51
+
- name: "*amazon-eks-node-1.30-*"
43
52
owner: self
44
-
- name: "*amazon-eks-node-1.27-*"
53
+
- name: "*amazon-eks-node-1.30-*"
45
54
owner: amazon
46
55
```
47
56
48
-
***IMPORTANT NOTE:*** With this configuration, you're saying that you need to use the latest AMI available for an EKS cluster v1.27 which is either owned by you (customized) or Amazon (official image). We're using a regular expression to have the flexibility to use AMIs for either `x86` or `Arm`, workloads that need GPUs, or a nodes with different OS like `Windows`. You're basically letting the workload (pod) to decide which type of node(s) it needs. If you don't have a custom AMI created by you in your account, Karpenter will use the official EKS AMI owned by Amazon.
57
+
***IMPORTANT NOTE:*** With this configuration, you're saying that you need to use the latest AMI available for an EKS cluster v1.30 which is either owned by you (customized) or Amazon (official image). We're using a regular expression to have the flexibility to use AMIs for either `x86` or `Arm`, workloads that need GPUs, or a nodes with different OS like `Windows`. You're basically letting the workload (pod) to decide which type of node(s) it needs. If you don't have a custom AMI created by you in your account, Karpenter will use the official EKS AMI owned by Amazon.
49
58
50
59
## Results
51
60
After waiting for about one minute, you should see a machine ready, and all pods in a `Running` state, like this:
0 commit comments