使用kubernetes部署jenkins

2018-07-18
#DevOps #Kubernetes #Jenkins
DevOps

创建ci空间

给jenkins建立单独的表空间ci，与其他应用隔离
ci.yaml:

1
2
3
4
5
6
apiVersion: v1
kind: Namespace
metadata:
   name: ci
   labels:
     name: ci

创建用户

创建jenkins-admin的用户，进行角色管理jenkins-rbac.yaml：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: jenkins
  name: jenkins-admin
  namespace: ci
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: jenkins-admin
  labels:
    k8s-app: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: jenkins-admin
  namespace: ci
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  labels:
    k8s-app: jenkins
  name: jenkins
rules:
- apiGroups: ["", "extensions", "apps"]
  resources:
    - nodes
    - nodes/proxy
    - endpoints
    - secrets
    - pods
    - deployments
    - services
  verbs: ["get", "list", "watch"]

创建jenkins应用

jenkins需要挂载三个目录，挂载/var/jenkins_home的目录到/home/ap/ci/jenkins/var/jenkins_home，挂载/home/data的目录到/home/ap/ci/jenkins/data，挂载/home/jenkins到/home/ap/ci/jenkins/jenkins

同时jenkins需要安装到master服务器上，所以需要给master打上label，指令：

kubectl label nodes master type=k8s-master

同时在配置文件中设置：

1
2
nodeSelector:
     type: k8s-master

具体配置文件如下
jenkins.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
apiVersion: apps/v1beta2 # for versions before 1.8.0 use apps/v1beta1
kind: Deployment
metadata:
  name: jenkins
  namespace: ci
  labels:
    k8s-app: jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: jenkins
  template:
    metadata:
      labels:
        k8s-app: jenkins
    spec:
	  nodeSelector:
        type: k8s-master
      containers:
      - name: jenkins
        image: jenkins:devops
        volumeMounts:
        - name: var-jenkins-home
          mountPath: /var/jenkins_home
        - name: jenkins-data
          mountPath: /home/data
        - name: home-jenkins
          mountPath: /home/jenkins	  
        ports:
        - containerPort: 8080
        - containerPort: 50000
      volumes:
        - name: var-jenkins-home
          hostPath:
            path: /home/ap/ci/jenkins/var/jenkins_home
        - name: jenkins-data
          hostPath:
            path: /home/ap/ci/jenkins/data
        - name: home-jenkins
          hostPath:
            path: /home/ap/ci/jenkins/jenkins
      serviceAccount: "jenkins-admin"

创建jenkins服务

jenkins-service.yaml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: jenkins
  name: jenkins
  namespace: ci
  annotations:
    prometheus.io/scrape: 'true'
spec:
  ports:
    - port: 8080
      name: jenkins
      targetPort: 8080
      nodePort: 31888
    - port: 50000
      name: jenkins-agent
      targetPort: 50000
  type: NodePort
  selector:
    k8s-app: jenkins

创建完查看集群信息：

1
2
3
[root@master jenkins_k8s]# kubectl get service -n ci
NAME      TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
jenkins   NodePort   10.102.120.26   <none>        8080:31888/TCP,50000:31461/TCP   34m

现在就可以通过http://{IP}:31888访问jenkins了

kubernetes插件配置

下载插件kubernetes，在【系统管理】-【系统设置】-【新增一个云】-【Kubernetes】配置k8s的插件

image

image

image

参考上图的说明，进行配置。要注意的是，这里的Name字段配的名字，后面在配置pipeline的Jenkins任务时，是需要用到的（假设这里使用的名字叫Kubernetes）。然后点【Test Connection】，如果前面的Service Account配置的没问题的话，就会提示“Connection successful”，否则，会有访问apiserver的403权限报错。

用pipeline方式创建一个如下的Jenkins构建任务：

1
2
3
4
5
6
7
8
9
// this guarantees the node will use this template
def label = "mypod-${UUID.randomUUID().toString()}"
podTemplate(label: label) {
    node(label) {
        stage('Run shell') {
            sh 'echo hello world'
        }
    }
}

图2中Jenkins URL请填http://:或者http://jenkins.ci:8080, 尽量不要使用服务器的IP:nodePort，使用服务器IP时，slave节点通过http://IP:50000 端口与jenkins通信，但k8s只能暴露30000-32767端口，所以slave会连接失败

问题与解决

1、 jenkins 插件目录和工作目录的挂载：

1
2
3
4
5
6
7
8
9
10
  volumeMounts:
  - name: jenkins-home
    mountPath: /var/jenkins_home
  ports:
  - containerPort: 8080
  - containerPort: 50000
volumes:
  - name: jenkins-home
    hostPath:
      path: /home/ap/ci/jenkins/test

volumes将容器中的/var/jenkins_home目录挂载到本地的/home/ap/ci/jenkins/test文件夹中

2、 slave连接jenkins

slave连接jenkins时，新建出来的容器报错：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
[root@master jenkins_k8s]# kubectl logs pod/jenkins-slave-vcr29-lg2bz  -n ci
Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior
Jun 14, 2018 8:08:33 AM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: jenkins-slave-vcr29-lg2bz
Jun 14, 2018 8:08:33 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Jun 14, 2018 8:08:33 AM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 3.19
Jun 14, 2018 8:08:33 AM hudson.remoting.Engine startEngine
WARNING: No Working Directory. Using the legacy JAR Cache location: /home/jenkins/.jenkins/cache/jars
Jun 14, 2018 8:08:33 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://10.128.13.24:31888/]
Jun 14, 2018 8:08:33 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, JNLP-connect, Ping, JNLP2-connect]
Jun 14, 2018 8:08:33 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: 10.128.13.24
  Agent port:    50000
  Identity:      2e:b1:d0:06:6f:be:fb:5a:53:86:f6:a5:27:03:49:98
Jun 14, 2018 8:08:33 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Jun 14, 2018 8:08:33 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to 10.128.13.24:50000
Jun 14, 2018 8:08:33 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Jun 14, 2018 8:08:39 AM org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer onRecv
SEVERE: [JNLP4-connect connection to 10.128.13.24/10.128.13.24:50000] 
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
	at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1478)
	at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
	at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
	at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
	at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:392)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669)
	at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.onRecv(AckFilterLayer.java:255)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669)
	at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
	at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$2200(BIONetworkLayer.java:48)
	at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:283)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
	at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
	at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:382)
	... 11 more
Caused by: java.security.cert.CertificateException: Public key of the first certificate in chain (subject: C=US, OU=jenkins.io, O=instances, CN=364eef21b316f8b881b4b55dcc19bb20) is not in the list of trusted keys
	at org.jenkinsci.remoting.protocol.cert.PublicKeyMatchingX509ExtendedTrustManager.checkPublicKey(PublicKeyMatchingX509ExtendedTrustManager.java:217)
	at org.jenkinsci.remoting.protocol.cert.PublicKeyMatchingX509ExtendedTrustManager.checkServerTrusted(PublicKeyMatchingX509ExtendedTrustManager.java:263)
	at org.jenkinsci.remoting.protocol.cert.DelegatingX509ExtendedTrustManager.checkServerTrusted(DelegatingX509ExtendedTrustManager.java:148)
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1501)
	... 18 more

原因：

jenkins-slave连接到http://IP:50000，但k8s只能暴露30000-32767端口，所以slave会连接失败。

三种方法解决：

(1) jenkins的kubernetes配置中，Jenkins URL填http://:或者http://jenkins.ci:8080,jenkins-slave会到:50000请求连接

(2) 使用ingress代替service的port暴露，详细见查考链接

(3) k8s启动时带上解除端口限制的参数，详细见查考链接

参考文档：

RBAC的授权：https://www.kubernetes.org.cn/4062.html

jenkins配置：https://blog.csdn.net/felix_yujing/article/details/78725142

jenkins配置：https://blog.csdn.net/aixiaoyang168/article/details/79767649

ingress代替service的port暴露：https://zhangchenchen.github.io/2017/12/17/achieve-cicd-in-kubernetes-with-jenkins/

k8s启动时带上解除端口限制的参数：https://github.com/kubernetes/kubeadm/issues/122