视频地址 (opens new window)

编译和启动

1. 编译

cd $GOPATH/src/k8s.io/kubernetes

# 编译单个组建：
sudo make WHAT="cmd/kube-apiserver"

# 编译所有组件：
sudo make all

2. 启动本地单节点集群

1. 启动集群

cd $GOPATH/src/k8s.io/kubernetes
sudo make clean && sudo make all # 可选
sudo ./hack/local-up-cluster.sh

启动的正常输出

make: 进入目录“/home/xing/go/src/k8s.io/kubernetes”
make[1]: 进入目录“/home/xing/go/src/k8s.io/kubernetes”
+++ [1201 17:08:55] Building go targets for linux/amd64
    k8s.io/kubernetes/hack/make-rules/helpers/go2make (non-static)
make[1]: 离开目录“/home/xing/go/src/k8s.io/kubernetes”
+++ [1201 17:09:08] Building go targets for linux/amd64
    k8s.io/kubernetes/cmd/kubectl (static)
    k8s.io/kubernetes/cmd/kube-apiserver (static)
    k8s.io/kubernetes/cmd/kube-controller-manager (static)
    k8s.io/kubernetes/cmd/cloud-controller-manager (non-static)
    k8s.io/kubernetes/cmd/kubelet (non-static)
    k8s.io/kubernetes/cmd/kube-proxy (static)
    k8s.io/kubernetes/cmd/kube-scheduler (static)
make: 离开目录“/home/xing/go/src/k8s.io/kubernetes”
API SERVER secure port is free, proceeding...
Detected host and ready to start services.  Doing some housekeeping first...
Using GO_OUT /home/xing/go/src/k8s.io/kubernetes/_output/local/bin/linux/amd64
Starting services now!
Starting etcd
etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /tmp/tmp.LNJCC3Y7Wu --listen-client-urls http://127.0.0.1:2379 --log-level=warn 2> "/tmp/etcd.log" >/dev/null
Waiting for etcd to come up.
+++ [1201 17:09:45] On try 2, etcd: : {"health":"true","reason":""}
{"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"2","raft_term":"2"}}Generating a RSA private key
.............+++++
.........................+++++
writing new private key to '/var/run/kubernetes/server-ca.key'
-----
Generating a RSA private key
........+++++
................................................+++++
writing new private key to '/var/run/kubernetes/client-ca.key'
-----
Generating a RSA private key
.............................................+++++
.......................................................................................................................+++++
writing new private key to '/var/run/kubernetes/request-header-ca.key'
-----
2022/12/01 17:09:45 [INFO] generate received request
2022/12/01 17:09:45 [INFO] received CSR
2022/12/01 17:09:45 [INFO] generating key: rsa-2048
2022/12/01 17:09:45 [INFO] encoded CSR
2022/12/01 17:09:45 [INFO] signed certificate with serial number 293151017665098086348880715532871957730156180132
2022/12/01 17:09:45 [INFO] generate received request
2022/12/01 17:09:45 [INFO] received CSR
2022/12/01 17:09:45 [INFO] generating key: rsa-2048
2022/12/01 17:09:45 [INFO] encoded CSR
2022/12/01 17:09:45 [INFO] signed certificate with serial number 487348345664182233435909206222659454057143719021
2022/12/01 17:09:45 [INFO] generate received request
2022/12/01 17:09:45 [INFO] received CSR
2022/12/01 17:09:45 [INFO] generating key: rsa-2048
2022/12/01 17:09:46 [INFO] encoded CSR
2022/12/01 17:09:46 [INFO] signed certificate with serial number 195819653141672173053279722256472095709163847153
2022/12/01 17:09:46 [INFO] generate received request
2022/12/01 17:09:46 [INFO] received CSR
2022/12/01 17:09:46 [INFO] generating key: rsa-2048
2022/12/01 17:09:46 [INFO] encoded CSR
2022/12/01 17:09:46 [INFO] signed certificate with serial number 613544450701441397610845302043050839746564066359
2022/12/01 17:09:46 [INFO] generate received request
2022/12/01 17:09:46 [INFO] received CSR
2022/12/01 17:09:46 [INFO] generating key: rsa-2048
2022/12/01 17:09:46 [INFO] encoded CSR
2022/12/01 17:09:46 [INFO] signed certificate with serial number 115061008643950752125067148135424784083397571613
2022/12/01 17:09:46 [INFO] generate received request
2022/12/01 17:09:46 [INFO] received CSR
2022/12/01 17:09:46 [INFO] generating key: rsa-2048
2022/12/01 17:09:46 [INFO] encoded CSR
2022/12/01 17:09:46 [INFO] signed certificate with serial number 584615864316512973410633747924687800725528845655
2022/12/01 17:09:46 [INFO] generate received request
2022/12/01 17:09:46 [INFO] received CSR
2022/12/01 17:09:46 [INFO] generating key: rsa-2048
2022/12/01 17:09:46 [INFO] encoded CSR
2022/12/01 17:09:46 [INFO] signed certificate with serial number 571089137347621389894999036328721562159155378791
2022/12/01 17:09:46 [INFO] generate received request
2022/12/01 17:09:46 [INFO] received CSR
2022/12/01 17:09:46 [INFO] generating key: rsa-2048
2022/12/01 17:09:47 [INFO] encoded CSR
2022/12/01 17:09:47 [INFO] signed certificate with serial number 153203515574633399568421412669629167562151942413
Waiting for apiserver to come up
+++ [1201 17:09:52] On try 5, apiserver: : ok
clusterrolebinding.rbac.authorization.k8s.io/kube-apiserver-kubelet-admin created
clusterrolebinding.rbac.authorization.k8s.io/kubelet-csr created
Cluster "local-up-cluster" set.
use 'kubectl --kubeconfig=/var/run/kubernetes/admin-kube-aggregator.kubeconfig' to use the aggregated API server
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
coredns addon successfully deployed.
Checking CNI Installation at /opt/cni/bin
CNI Installation not found at /opt/cni/bin
CNI_PLUGINS_URL:https://ghps.cc/https://github.com/containernetworking/plugins/releases/download/v1.0.1/cni-plugins-linux-amd64-v1.0.1.tgz
Installing CNI plugin binaries ...
5238fbb2767cbf6aae736ad97a7aa29167525dcd405196dfbc064672a730d3cf  /tmp/cni.amd64.tgz
/tmp/cni.amd64.tgz: 成功
./
./macvlan
./static
./vlan
./portmap
./host-local
./vrf
./bridge
./tuning
./firewall
./host-device
./sbr
./loopback
./dhcp
./ptp
./ipvlan
./bandwidth
Configuring cni
{
  "cniVersion": "0.4.0",
  "name": "containerd-net",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni0",
      "isGateway": true,
      "ipMasq": true,
      "promiscMode": true,
      "ipam": {
        "type": "host-local",
        "ranges": [
          [{
            "subnet": "10.88.0.0/16"
          }],
          [{
            "subnet": "2001:4860:4860::/64"
          }]
        ],
        "routes": [
          { "dst": "0.0.0.0/0" },
          { "dst": "::/0" }
        ]
      }
    },
    {
      "type": "portmap",
      "capabilities": {"portMappings": true}
    }
  ]
}
WARNING : The kubelet is configured to not fail even if swap is enabled; production deployments should disable swap unless testing NodeSwap feature.
2022/12/01 17:10:00 [INFO] generate received request
2022/12/01 17:10:00 [INFO] received CSR
2022/12/01 17:10:00 [INFO] generating key: rsa-2048
2022/12/01 17:10:00 [INFO] encoded CSR
2022/12/01 17:10:00 [INFO] signed certificate with serial number 421978020129055193511090121764118686933107812863
kubelet ( 304932 ) is running.
wait kubelet ready
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
127.0.0.1   NotReady   <none>   2s    v1.24.0-dirty
2022/12/01 17:10:16 [INFO] generate received request
2022/12/01 17:10:16 [INFO] received CSR
2022/12/01 17:10:16 [INFO] generating key: rsa-2048
2022/12/01 17:10:16 [INFO] encoded CSR
2022/12/01 17:10:16 [INFO] signed certificate with serial number 592457083326791792374452832572534131669724027176
Create default storage class for
storageclass.storage.k8s.io/standard created
Local Kubernetes cluster is running. Press Ctrl-C to shut it down.

Logs:
  /tmp/kube-apiserver.log
  /tmp/kube-controller-manager.log

  /tmp/kube-proxy.log
  /tmp/kube-scheduler.log
  /tmp/kubelet.log

To start using your cluster, you can open up another terminal/tab and run:

  export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig
  cluster/kubectl.sh

Alternatively, you can write to the default kubeconfig:

  export KUBERNETES_PROVIDER=local

  cluster/kubectl.sh config set-cluster local --server=https://localhost:6443 --certificate-authority=/var/run/kubernetes/server-ca.crt
  cluster/kubectl.sh config set-credentials myself --client-key=/var/run/kubernetes/client-admin.key --client-certificate=/var/run/kubernetes/client-admin.crt
  cluster/kubectl.sh config set-context local --cluster=local --user=myself
  cluster/kubectl.sh config use-context local
  cluster/kubectl.sh

2. 验证集群

cd ${GOPATH}/src/k8s.io/kubernetes
export KUBECONFIG=/var/run/kubernetes/admin.kubeconfig
cluster/kubectl.sh get nodes

>>> Outpus:
NAME        STATUS   ROLES    AGE    VERSION
127.0.0.1   Ready    <none>   5m3s   v1.24.0-dirty

部署资源

# /home/${USER}/kube/yaml/test.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    name: nginx
spec:
  containers:
    - name: nginx
      image: daocloud.io/library/nginx:1.12.0-alpine
      ports:
        - containerPort: 80

mkdir -p /home/${USER}/kube/yaml
cluster/kubectl.sh create -f /home/${USER}/kube/yaml/test.yaml

>>> Outputs:
pod/nginx created

cluster/kubectl.sh get po
>>> Outputs:
NAME    READY   STATUS              RESTARTS   AGE
nginx   0/1     ContainerCreating   0          70s

3. 启动过程中遇到的问题

1. cni 下载不下来

替换 local-up-cluster.sh 中的变量 CNI_PLUGINS_URL

也可以自己下载下来放到指定目录

CNI_PLUGINS_URL="https://ghps.cc/https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGINS_TARBALL}"
# CNI_PLUGINS_URL="https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGINS_TARBALL}"

调试源码

1. 开启编译选项

cd $GOPATH/src/k8s.io/kubernetes
sudo vim ./hack/lib/golang.sh
# 查找build_binaries()函数 vi语法
:/build_binaries()

gogcflags="all=-trimpath=${trimroot} ${GOGCFLAGS:-}"
if [[ "${DBG:-}" == 1 ]]; then
    # Debugging - disable optimizations and inlining.
    gogcflags="${gogcflags} -N -l"
fi

goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}"
if [[ "${DBG:-}" != 1 ]]; then
    # Not debugging - disable symbols and DWARF.
    goldflags="${goldflags} -s -w"
fi

替换为

gogcflags="all=-trimpath=${trimroot} ${GOGCFLAGS:-}"
# if [[ "${DBG:-}" == 1 ]]; then
#     # Debugging - disable optimizations and inlining.
#     gogcflags="${gogcflags} -N -l"
# fi
gogcflags="${gogcflags} -N -l"
goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}"
# if [[ "${DBG:-}" != 1 ]]; then
#     # Not debugging - disable symbols and DWARF.
#     goldflags="${goldflags} -s -w"
# fi

2. 安装 dlv (opens new window)

git clone https://github.com/go-delve/delve
cd delve
go install github.com/go-delve/delve/cmd/dlv

注

这里我使用了 vscode go Tools 提供的套件

3. 调试（均支持远程调试）（api-server）

0. 调试环境准备

1. 获取启动参数

后面 dlv 需要这些参数

ps -ef | grep  kube-apiserver

/home/xing/go/src/k8s.io/kubernetes/_output/local/bin/linux/amd64/kube-apiserver --authorization-mode=Node,RBAC  --cloud-provider= --cloud-config=   --v=3 --vmodule= --audit-policy-file=/tmp/kube-audit-policy-file --audit-log-path=/tmp/kube-apiserver-audit.log --authorization-webhook-config-file= --authentication-token-webhook-config-file= --cert-dir=/var/run/kubernetes --egress-selector-config-file=/tmp/kube_egress_selector_configuration.yaml --client-ca-file=/var/run/kubernetes/client-ca.crt --kubelet-client-certificate=/var/run/kubernetes/client-kube-apiserver.crt --kubelet-client-key=/var/run/kubernetes/client-kube-apiserver.key --service-account-key-file=/tmp/kube-serviceaccount.key --service-account-lookup=true --service-account-issuer=https://kubernetes.default.svc --service-account-jwks-uri=https://kubernetes.default.svc/openid/v1/jwks --service-account-signing-key-file=/tmp/kube-serviceaccount.key --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,Priority,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction --disable-admission-plugins= --admission-control-config-file= --bind-address=0.0.0.0 --secure-port=6443 --tls-cert-file=/var/run/kubernetes/serving-kube-apiserver.crt --tls-private-key-file=/var/run/kubernetes/serving-kube-apiserver.key --storage-backend=etcd3 --storage-media-type=application/vnd.kubernetes.protobuf --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.0.0.0/24 --feature-gates=AllAlpha=false --external-hostname=localhost --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-client-ca-file=/var/run/kubernetes/request-header-ca.crt --requestheader-allowed-names=system:auth-proxy --proxy-client-cert-file=/var/run/kubernetes/client-auth-proxy.crt --proxy-client-key-file=/var/run/kubernetes/client-auth-proxy.key --cors-allowed-origins="/127.0.0.1(:[0-9]+)?$,/localhost(:[0-9]+)?$"

2. 杀掉 kube-apiserver

kill -9 378253 # 这里替换你的 kube-apiserver pid

# 这时候集群会报警告信息出来
>>> Outputs:
W1201 18:13:25]: API server terminated unexpectedly, see /tmp/kube-apiserver.log

1. 利用 vscode 进行代码调试

1. 以 dlv 方式启动 kube-apiserver

sudo dlv --headless exec /home/xing/go/src/k8s.io/kubernetes/_output/local/bin/linux/amd64/kube-apiserver   --api-version=2 --only-same-user=false  -- --authorization-mode=Node,RBAC  --cloud-provider= --cloud-config=   --v=3 --vmodule= --audit-policy-file=/tmp/kube-audit-policy-file --audit-log-path=/tmp/kube-apiserver-audit.log --authorization-webhook-config-file= --authentication-token-webhook-config-file= --cert-dir=/var/run/kubernetes --egress-selector-config-file=/tmp/kube_egress_selector_configuration.yaml --client-ca-file=/var/run/kubernetes/client-ca.crt --kubelet-client-certificate=/var/run/kubernetes/client-kube-apiserver.crt --kubelet-client-key=/var/run/kubernetes/client-kube-apiserver.key --service-account-key-file=/tmp/kube-serviceaccount.key --service-account-lookup=true --service-account-issuer=https://kubernetes.default.svc --service-account-jwks-uri=https://kubernetes.default.svc/openid/v1/jwks --service-account-signing-key-file=/tmp/kube-serviceaccount.key --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,Priority,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction --disable-admission-plugins= --admission-control-config-file= --bind-address=0.0.0.0 --secure-port=6443 --tls-cert-file=/var/run/kubernetes/serving-kube-apiserver.crt --tls-private-key-file=/var/run/kubernetes/serving-kube-apiserver.key --storage-backend=etcd3 --storage-media-type=application/vnd.kubernetes.protobuf --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.0.0.0/24 --feature-gates=AllAlpha=false --external-hostname=localhost --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-client-ca-file=/var/run/kubernetes/request-header-ca.crt --requestheader-allowed-names=system:auth-proxy --proxy-client-cert-file=/var/run/kubernetes/client-auth-proxy.crt --proxy-client-key-file=/var/run/kubernetes/client-auth-proxy.key --cors-allowed-origins="/127.0.0.1(:[0-9]+)?$,/localhost(:[0-9]+)?$"


# 注：如果指定了 --listen 就是你指定的 如果没有指定则随机生成一个端口 这里我是随机生成
>>> Outputs:
API server listening at: 127.0.0.1:43605

2. 生成配置文件（不会百度或者复制下面即可）

用 vscode 打开源码

// .vscode/launch.json
{
    // 使用 IntelliSense 了解相关属性。
    // 悬停以查看现有属性的描述。
    // 欲了解更多信息，请访问: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Connect to server",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "remotePath": "${workspaceFolder}",
            "port": 2345,
            "host": "127.0.0.1"
        }
    ]
}

3. 开始调试

F5 执行

4. 成功执行调试：

2. 利用 Goland 进行代码调试

3. 利用 命令行 进行代码调试

1. 以 dlv 方式启动 kube-apiserver

sudo dlv --headless exec /home/xing/go/src/k8s.io/kubernetes/_output/local/bin/linux/amd64/kube-apiserver   --api-version=2  -- --authorization-mode=Node,RBAC  --cloud-provider= --cloud-config=   --v=3 --vmodule= --audit-policy-file=/tmp/kube-audit-policy-file --audit-log-path=/tmp/kube-apiserver-audit.log --authorization-webhook-config-file= --authentication-token-webhook-config-file= --cert-dir=/var/run/kubernetes --egress-selector-config-file=/tmp/kube_egress_selector_configuration.yaml --client-ca-file=/var/run/kubernetes/client-ca.crt --kubelet-client-certificate=/var/run/kubernetes/client-kube-apiserver.crt --kubelet-client-key=/var/run/kubernetes/client-kube-apiserver.key --service-account-key-file=/tmp/kube-serviceaccount.key --service-account-lookup=true --service-account-issuer=https://kubernetes.default.svc --service-account-jwks-uri=https://kubernetes.default.svc/openid/v1/jwks --service-account-signing-key-file=/tmp/kube-serviceaccount.key --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,Priority,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction --disable-admission-plugins= --admission-control-config-file= --bind-address=0.0.0.0 --secure-port=6443 --tls-cert-file=/var/run/kubernetes/serving-kube-apiserver.crt --tls-private-key-file=/var/run/kubernetes/serving-kube-apiserver.key --storage-backend=etcd3 --storage-media-type=application/vnd.kubernetes.protobuf --etcd-servers=http://127.0.0.1:2379 --service-cluster-ip-range=10.0.0.0/24 --feature-gates=AllAlpha=false --external-hostname=localhost --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-client-ca-file=/var/run/kubernetes/request-header-ca.crt --requestheader-allowed-names=system:auth-proxy --proxy-client-cert-file=/var/run/kubernetes/client-auth-proxy.crt --proxy-client-key-file=/var/run/kubernetes/client-auth-proxy.key --cors-allowed-origins="/127.0.0.1(:[0-9]+)?$,/localhost(:[0-9]+)?$"


# 注：如果指定了 --listen 就是你指定的 如果没有指定则随机生成一个端口 这里我是随机生成
>>> Outputs:
API server listening at: 127.0.0.1:43605

2. 连接 dlv connect

dlv connect localhost:43605

# 在这里就可以调试了
# 打断点
break cmd/kube-apiserver/apiserver.go:33

>>> Outputs:
Breakpoint 1 set at 0x5233854 for main.main() cmd/kube-apiserver/apiserver.go:33

continue
>>> Outputs:
> main.main() cmd/kube-apiserver/apiserver.go:33 (hits goroutine(1):1 total:1) (PC: 0x5233854)
    28:         _ "k8s.io/component-base/metrics/prometheus/version"  // for version metric registration
    29:         "k8s.io/kubernetes/cmd/kube-apiserver/app"
    30: )
    31:
    32: func main() {
=>  33:         command := app.NewAPIServerCommand()
    34:         code := cli.Run(command)
    35:         os.Exit(code)
    36: }

进行愉快的命令行调试就可以了

4. 调试过程中注意事项

1. dlv

在 dlv 启动参数一定要加 --only-same-user=false 参数选项，否则会出现如下警告，导致远程或本地无法调试

closing connection from different user (127.0.0.1:34980): connections to localhost are only accepted from the same UNIX user for security reasons

大致意思就是仅允许当前启动 dlv app 的用户操作 debug，包括 sudo 指令在内

https://github.com/golang/vscode-go/issues/810 (opens new window)

2. kube-apiserver 参数

--cors-allowed-origins="/127.0.0.1(:[0-9]+)?$,/localhost(:[0-9]+)?$" 参数选项使用了正则表达式，在此处要用 "" 包裹，否则会出现 bash 语法解析错误，无法识别 ( 之类的语法