kubebuilder
weight: 110 title: Kubebuilder date: ‘2022-05-21T00:00:00+08:00’ type: book
动机
目前扩展
工作流程
- 创建一个新的工程目录
- 创建一个或多个资源
API CRD 然后将字段添加到资源 - 在控制器中实现协调循环(reconcile loop
) ,watch 额外的资源 - 在集群中运行测试(自动安装
CRD 并自动启动控制器) - 更新引导集成测试测试新字段和业务逻辑
- 使用用户提供的
Dockerfile 构建和发布容器
设计哲学
- 能使用
go 接口和库,就不使用代码生成 - 能使用代码生成,就不用使用多于一次的存根初始化
- 能使用一次存根,就不
fork 和修改boilerplate - 绝不
fork 和修改boilerplate
示例
下面是一个使用
准备
本文中的示例运行环境及相关软件版本如下:
- Kubernetes MiniKube v1.9.2
- Kubernetes v1.18.0
- Go 1.14
- Kubebuilder 2.3.1
- kustomize 3.6.1
- Docker 19.03.8
使用
Minikube 的DNS 解析问题
如果遇到
使用 minikube ssh
进入/etc/systemd/resolved.conf
文件,将其中的DNS=8.8.8.8
,然后执行 sudo systemctl restart systemd-resolved
即可更改/etc/resolv.conf
文件。
修正
minikube ssh
sudo sed -i 's/#DNS=/DNS=8.8.8.8/g' /etc/systemd/resolved.conf
sudo systemctl restart systemd-resolved
名词解释
在阅读下面的文章前,需要先明确以下两个名词的含义。
- CRD:自定义资源定义,
Kubernetes 中的资源类型。 - CR:Custom Resource,对使用
CRD 创建出来的自定义资源的统称。
安装kubebuilder
到
MacOS
对于/usr/local/kubebuilder
目录下,并将 /usr/local/kubebuilder/bin
添加到您的 $PATH
路径下。
创建项目
我们首先将使用自动配置创建一个项目,该项目在创建
初始化和创建API
创建的项目路径位于 $GOPATH/jimmysong.io/kubebuilder-example
。下文中的操作没有明确说明的话都是在该项目路径下运行。
在项目路径下使用下面的命令初始化项目。
$ kubebuilder init --domain jimmysong.io
在项目根目录下执行下面的命令创建
$ kubebuilder create api --group webapp --version v1 --kind Guestbook
Create Resource under pkg/apis [y/n]?
y
Create Controller under pkg/controller [y/n]?
y
Writing scaffold for you to edit...
api/v1/guestbook_types.go
controllers/guestbook_controller.go
Running make:
$ make
/Users/jimmysong/Workspace/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go: finding github.com/onsi/ginkgo v1.11.0
go: finding github.com/onsi/gomega v1.8.1
go: finding github.com/hpcloud/tail v1.0.0
go: finding gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
go build -o bin/manager main.go
.
├── Dockerfile # 用于构建 Operator 镜像
├── Makefile # 构建时使用
├── PROJECT # 项目配置
├── api
│ └── v1
│ ├── groupversion_info.go
│ ├── guestbook_types.go
│ └── zz_generated.deepcopy.go
├── bin
│ └── manager
├── config
│ ├── certmanager
│ │ ├── certificate.yaml
│ │ ├── kustomization.yaml
│ │ └── kustomizeconfig.yaml
│ ├── crd # 新增 CRD 定义
│ │ ├── kustomization.yaml
│ │ ├── kustomizeconfig.yaml
│ │ └── patches
│ ├── default
│ │ ├── kustomization.yaml
│ │ ├── manager_auth_proxy_patch.yaml
│ │ ├── manager_webhook_patch.yaml
│ │ └── webhookcainjection_patch.yaml
│ ├── manager
│ │ ├── kustomization.yaml
│ │ └── manager.yaml
│ ├── prometheus
│ │ ├── kustomization.yaml
│ │ └── monitor.yaml
│ ├── rbac
│ │ ├── auth_proxy_client_clusterrole.yaml
│ │ ├── auth_proxy_role.yaml
│ │ ├── auth_proxy_role_binding.yaml
│ │ ├── auth_proxy_service.yaml
│ │ ├── guestbook_editor_role.yaml
│ │ ├── guestbook_viewer_role.yaml
│ │ ├── kustomization.yaml
│ │ ├── leader_election_role.yaml
│ │ ├── leader_election_role_binding.yaml
│ │ └── role_binding.yaml
│ ├── samples
│ │ └── webapp_v1_guestbook.yaml # CRD 示例
│ └── webhook
│ ├── kustomization.yaml
│ ├── kustomizeconfig.yaml
│ └── service.yaml
├── controllers # 新增 controller
│ ├── guestbook_controller.go
│ └── suite_test.go
├── go.mod
├── go.sum
├── hack
│ └── boilerplate.go.txt
└── main.go # 新增处理逻辑
15 directories, 40 files
以上就是自动初始化出来的文件。
安装CRD
执行下面的命令安装
$ make install
/Users/jimmysong/Workspace/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/guestbooks.webapp.jimmysong.io created
$ kubectl get crd |grep jimmysong.io
guestbooks.webapp.jimmysong.io 2020-06-06T21:58:17Z
部署controller
在开始部署
修改使用
对于中国大陆用户,可能无法访问config/default/manager_auth_proxy_patch.yaml
文件中的镜像地址,将其中 gcr.io/kube-rbac-proxy:v0.5.0
修改为 jimmysong/kubebuilder-kube-rbac-proxy:v0.5.0
。
有两种方式运行
- 本地运行,用于调试
- 部署到
Kubernetes 上运行,作为生产使用
本地运行
要想在本地运行
make run
你将看到
将
执行下面的命令部署
make docker-build docker-push IMG=jimmysong/kubebuilder-example:latest
make deploy IMG=jimmysong/kubebuilder-example:latest
在初始化项目时,kubebuilder-example-system
,查看
$ kubectl get deployment -n kubebuilder-example-system
NAME READY UP-TO-DATE AVAILABLE AGE
kubebuilder-example-controller-manager 1/1 1 1 3h26m
$ kubectl get pod -n kubebuilder-example-system
NAME READY STATUS RESTARTS AGE
kubebuilder-example-controller-manager-77b4c685f9-2npz8 2/2 Running 0 3h16m
创建CR
kubectl apply -f config/samples/webapp_v1_guestbook.yaml
执行下面的命令查看新创建的
$ kubectl get guestbooks.webapp.jimmysong.io guestbook-sample -o yaml
你将看到类似如下的输出。
apiVersion: webapp.jimmysong.io/v1
kind: Guestbook
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"webapp.jimmysong.io/v1","kind":"Guestbook","metadata":{"annotations":{},"name":"guestbook-sample","namespace":"kubebuilder-example-system"},"spec":{"foo":"bar"}}
creationTimestamp: "2020-06-07T01:04:48Z"
generation: 1
managedFields:
- apiVersion: webapp.jimmysong.io/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:spec:
.: {}
f:foo: {}
manager: kubectl
operation: Update
time: "2020-06-07T01:04:48Z"
name: guestbook-sample
namespace: kubebuilder-example-system
resourceVersion: "1795834"
selfLink: /apis/webapp.jimmysong.io/v1/namespaces/kubebuilder-example-system/guestbooks/guestbook-sample
uid: 051a4266-7f5a-4c57-8180-64222d462bba
spec:
foo: bar
至此一个基本的
增加业务逻辑
下面我们将修改
修改CRD
我们将修改上文中使用FirstName
、LastName
和 Status
字段。
下面是修改后的 api/v1/guestbook_types.go
文件的内容,对应修改的地方已在代码中注释说明。
/*
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package v1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// EDIT THIS FILE! THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required. Any new fields you add must have json tags for the fields to be serialized.
// GuestbookSpec defines the desired state of Guestbook
type GuestbookSpec struct {
// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
// Important: Run "make" to regenerate code after modifying this file
// Foo is an example field of Guestbook. Edit Guestbook_types.go to remove/update
// 添加两个新的字段
FirstName string `json:"firstname"`
LastName string `json:"lastname"`
}
// GuestbookStatus defines the observed state of Guestbook
type GuestbookStatus struct {
// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
// Important: Run "make" to regenerate code after modifying this file
Status string `json:"Status"`
}
// +kubebuilder:object:root=true
// 在这里增加 status 的说明
// +kubebuilder:subresource:status
// Guestbook is the Schema for the guestbooks API
type Guestbook struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec GuestbookSpec `json:"spec,omitempty"`
Status GuestbookStatus `json:"status,omitempty"`
}
// +kubebuilder:object:root=true
// GuestbookList contains a list of Guestbook
type GuestbookList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []Guestbook `json:"items"`
}
func init() {
SchemeBuilder.Register(&Guestbook{}, &GuestbookList{})
}
上面的代码比原先使用
FirstName string `json:"firstname"`
LastName string `json:"lastname"`
Status string `json:"Status"`
// +kubebuilder:subresource:status
修改Reconcile 函数
controllers/guestbook_controller.go
文件的 func (r *GuestbookReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error)
函数中。
// +kubebuilder:rbac:groups=webapp.jimmysong.io,resources=guestbooks,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=webapp.jimmysong.io,resources=guestbooks/status,verbs=get;update;patch
func (r *GuestbookReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
_ = context.Background()
_ = r.Log.WithValues("guestbook", req.NamespacedName)
// your logic here
ctx := context.Background()
_ = r.Log.WithValues("apiexamplea", req.NamespacedName)
// 获取当前的 CR,并打印
obj := &webappv1.Guestbook{}
if err := r.Get(ctx, req.NamespacedName, obj); err != nil {
log.Println(err, "Unable to fetch object")
} else {
log.Println("Geeting from Kubebuilder to", obj.Spec.FirstName, obj.Spec.LastName)
}
// 初始化 CR 的 Status 为 Running
obj.Status.Status = "Running"
if err := r.Status().Update(ctx, obj); err != nil {
log.Println(err, "unable to update status")
}
return ctrl.Result{}, nil
}
这段代码的业务逻辑是当发现有 guestbooks.webapp.jimmysong.io
的
运行测试
修改好
部署
跟上文的做法一样,执行下面的命令部署
make install
运行
跟上文的做法一样,执行下面的命令运行
make run
保持该窗口在前台运行。
部署
修改 config/samples/webapp_v1_guestbook.yaml
文件中的配置。
apiVersion: webapp.jimmysong.io/v1
kind: Guestbook
metadata:
name: guestbook-sample
spec:
# Add fields here
firstname: Jimmy
lastname: Song
将其应用到
kubectl apply -f config/samples/webapp_v1_guestbook.yaml
此时转到上文中运行
go fmt ./...
go vet ./...
/Users/jimmysong/Workspace/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go run ./main.go
2020-06-07T16:48:29.966+0800 INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"}
2020-06-07T16:48:29.967+0800 INFO setup starting manager
2020-06-07T16:48:29.967+0800 INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
2020-06-07T16:48:29.967+0800 INFO controller-runtime.controller Starting EventSource {"controller": "guestbook", "source": "kind source: /, Kind="}
2020-06-07T16:48:30.068+0800 INFO controller-runtime.controller Starting Controller {"controller": "guestbook"}
2020-06-07T16:48:30.068+0800 INFO controller-runtime.controller Starting workers {"controller": "guestbook", "worker count": 1}
2020/06/07 16:48:30 Geeting from Kubebuilder to Jimmy Song
2020-06-07T16:48:30.080+0800 DEBUG controller-runtime.controller Successfully Reconciled {"controller": "guestbook", "request": "kubebuilder-example-system/guestbook-sample"}
从上面的日志中,可以看到这条输出。
2020/06/07 16:48:30 Geeting from Kubebuilder to Jimmy Song
这正是在 Reconcile
函数中的输出。
获取当前的
使用下面的命令获取当前的
# kubectl get guestbooks.webapp.jimmysong.io guestbook-sample -o yaml
将看到如下输出。
apiVersion: webapp.jimmysong.io/v1
kind: Guestbook
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"webapp.jimmysong.io/v1","kind":"Guestbook","metadata":{"annotations":{},"name":"guestbook-sample","namespace":"kubebuilder-example-system"},"spec":{"firstname":"Jimmy","lastname":"Song"}}
creationTimestamp: "2020-06-07T02:54:46Z"
generation: 1
managedFields:
- apiVersion: webapp.jimmysong.io/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:spec:
.: {}
f:firstname: {}
f:lastname: {}
manager: kubectl
operation: Update
time: "2020-06-07T02:54:46Z"
- apiVersion: webapp.jimmysong.io/v1
fieldsType: FieldsV1
fieldsV1:
f:status:
.: {}
f:Status: {}
manager: main
operation: Update
time: "2020-06-07T02:56:38Z"
name: guestbook-sample
namespace: kubebuilder-example-system
resourceVersion: "1813769"
selfLink: /apis/webapp.jimmysong.io/v1/namespaces/kubebuilder-example-system/guestbooks/guestbook-sample
uid: 17da5eae-1020-40d2-821a-9a1f990dd767
spec:
firstname: Jimmy
lastname: Song
status:
Status: Running
我们输出的最后部分:
spec:
firstname: Jimmy
lastname: Song
status:
Status: Running
这正是我们在
删除
使用下面的命令删除
kubectl delete guestbooks.webapp.jimmysong.io guestbook-sample
此时在
2020/06/07 20:09:50 Guestbook.webapp.jimmysong.io "guestbook-sample" not found Unable to fetch object
2020/06/07 20:09:50 resource name may not be empty unable to update status
2020-06-07T20:09:50.380+0800 DEBUG controller-runtime.controller Successfully Reconciled {"controller": "guestbook", "request": "kubebuilder-example-system/guestbook-sample"}
因为该
更多
本示例仅展示了使用
- 初始化项目和
API - 安装
CRD - 部署
Controller - 创建
CR
Reconcile
函数中,请参考 Awesome Cloud Native 中的