在 Minikube 上快速安裝 Tanzu Application Platform 1.6

VMware 曾經在多年前發展過一套以容器技術為基礎的PaaS，稱為Cloud Foundry；2013年，與EMC、GE共同投資Pivotal後，VMware將自己的許多資產移交給Pivotal持續發展，除了大家熟悉的 Spring Framework之外，Cloud Foundry也是受到許多金融行庫喜愛的重要的容器平台產品，當時稱為Pivotal Cloud Foundry(簡稱 PCF)。PCF對於開發者的體驗堪稱完美，尤其是Java開發者，因此在VMware併購Pivotal之後，許多客戶希望PCF上便利的操作方式可以複製到K8s或是VMware Tanzu Kubernetes Grid上，而Tanzu Application Platform即是這個因素下，所誕生的產品。這篇文章，我將解釋VMware的Tanzu Application Platform的架構與在Minikube上的安裝過程。

Tanzu Application Platform 架構

說明：

1. TAP是由多個開源組件所構建出來的平台，又為了不同的任務將這些組件進行組合，形成不同的 profile，共計有 full、light、iteration、build、run與view。full跟light字面上很容易理解，iteration做為開發使用，build則負責應用編譯與鏡像生成之用，run可部署在staging 或是production環境，提供GitOps的應用部署模式，最後的view就是用來查看每個叢集的運行狀況。

安裝準備：

1. 硬體：CPU Intel i5 12代，記憶體 64G，SSD 1T，準系統華擎 MiniDesk
2. 安裝Carvel：這是許多容器相關操作的工具集合，我個人推薦使用 Homebrew方式安裝(參考網址https://carvel.dev/)
3. 安裝tanzu cli

brew update
brew install vmware-tanzu/tanzu/tanzu-cli
tanzu plugin install --group vmware-tap/default:v1.6.4

4. 設定 Harbor 憑證

sudo cp microservice.tw.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
sudo systemctl restart docker.service
ls /etc/ssl/certs | awk /microservice.tw/

5. 設定Ubuntu 最大檔案開啟數量

sudo sysctl fs.inotify.max_user_watches=524288
sudo sysctl fs.inotify.max_user_instances=512

Ubuntu 重新開機後，自動設定最大檔案開啟數量，請於 /etc/sysctl.conf中寫入下列兩行設定

fs.inotify.max_user_watches=524288
fs.inotify.max_user_instances=512

6. 啟動並設定 minikube

minikube start -p ite --cpus=16 --memory=32G  --insecure-registry="reg.microservice.tw"
minikube addons enable ingress -p ite                                                   
minikube addons enable ingress-dns -p ite
minikube tunnel -p ite

7. 暫停與繼續使用 minikube

minikube pause -p ite #關機前暫停

minikube unpause -p ite #開機後繼續

8. 下載與準備安裝檔，請於 Harbor 上建立四個 project：tanzu-application-platform、tanzu-cluster-essentials、cicd-tools與k8s-summit-2023，並從 Tanzu Network 下載 Cluster Essentials for VMware Tanzu 設定檔

mkdir $HOME/tanzu-cluster-essentials
tar -xvf cluster-essentials-bundle-1.6.4.tar -C $HOME/tanzu-cluster-essentials

# Set tanzunet as the source registry to copy the Tanzu Application Platform packages from.
export IMGPKG_REGISTRY_HOSTNAME_0=registry.tanzu.vmware.com
export IMGPKG_REGISTRY_USERNAME_0=[USERNAME]
export IMGPKG_REGISTRY_PASSWORD_0=[PASSWORD]

# The user’s registry for copying the Tanzu Application Platform package to.
export IMGPKG_REGISTRY_HOSTNAME_1=reg.microservice.tw
export IMGPKG_REGISTRY_USERNAME_1=[USERNAME]
export IMGPKG_REGISTRY_PASSWORD_1=[PASSWORD]
export TAP_VERSION=1.6.4
export REGISTRY_CA_PATH=$HOME/workspace/tap16/microservice.tw.crt

### Cluster Essentials 部分
# 參考文件：https://docs.vmware.com/en/Cluster-Essentials-for-VMware-Tanzu/1.6/cluster-essentials/deploy.html

#下載 Cluster Essentials 到電腦裡
imgpkg copy \
    -b $IMGPKG_REGISTRY_HOSTNAME_0/tanzu-cluster-essentials/cluster-essentials-bundle@sha256:2f538b69c866023b7d408cce6f0624c5662ee0703d8492e623b7fce10b6f840b \
    --to-tar cluster-essentials-bundle-$TAP_VERSION.tar \
    --include-non-distributable-layers
  
#上傳 Cluster Essentials 到地端的 Harbor  
imgpkg copy \
  --tar cluster-essentials-bundle-$TAP_VERSION.tar \
  --to-repo $IMGPKG_REGISTRY_HOSTNAME_1/tanzu-cluster-essentials/cluster-essentials-bundle \
  --include-non-distributable-layers \
  --registry-ca-cert-path $REGISTRY_CA_PATH

#安裝 Cluster Essentials
kubectl create namespace kapp-controller
kubectl create secret generic kapp-controller-config \
   --namespace kapp-controller \
   --from-file caCerts=$REGISTRY_CA_PATH
   
cd $HOME/tanzu-cluster-essentials
INSTALL_BUNDLE=$IMGPKG_REGISTRY_HOSTNAME_1/tanzu-cluster-essentials/cluster-essentials-bundle@sha256:2f538b69c866023b7d408cce6f0624c5662ee0703d8492e623b7fce10b6f840b \
  INSTALL_REGISTRY_HOSTNAME=$IMGPKG_REGISTRY_HOSTNAME_1 \
  INSTALL_REGISTRY_USERNAME=$IMGPKG_REGISTRY_USERNAME_1 \
  INSTALL_REGISTRY_PASSWORD=$IMGPKG_REGISTRY_PASSWORD_1 \
  ./install.sh

### Tanzu Application Platform 部分
# 參考文件：https://docs.vmware.com/en/VMware-Tanzu-Application-Platform/1.6/tap/install-offline-profile.html

mkdir -p $HOME/workspace/tap16
cd $HOME/workspace/tap16

#下載 Tanzu Application Platform 安裝檔
imgpkg copy \
  -b $IMGPKG_REGISTRY_HOSTNAME_0/tanzu-application-platform/tap-packages:$TAP_VERSION \
  --to-tar tap-packages-$TAP_VERSION.tar \
  --include-non-distributable-layers

#上傳 Tanzu Application Platform 安裝檔到自己的 Harbor
imgpkg copy \
  --tar tap-packages-$TAP_VERSION.tar \
  --to-repo $IMGPKG_REGISTRY_HOSTNAME_1/tanzu-application-platform/tap-packages \
  --include-non-distributable-layers \
  --registry-ca-cert-path $REGISTRY_CA_PATH


kubectl create ns tap-install

tanzu secret registry add tap-registry \
    --server   $IMGPKG_REGISTRY_HOSTNAME_1 \
    --username $IMGPKG_REGISTRY_USERNAME_1 \
    --password $IMGPKG_REGISTRY_PASSWORD_1 \
    --namespace tap-install \
    --export-to-all-namespaces \
    --yes

tanzu secret registry add registry-credentials \
    --server   $IMGPKG_REGISTRY_HOSTNAME_1 \
    --username $IMGPKG_REGISTRY_USERNAME_1 \
    --password $IMGPKG_REGISTRY_PASSWORD_1 \
    --namespace tap-install \
    --export-to-all-namespaces \
    --yes

#加入 TAP Repository
tanzu package repository add tanzu-tap-repository \
  --url $IMGPKG_REGISTRY_HOSTNAME_1/tanzu-application-platform/tap-packages:$TAP_VERSION \
  --namespace tap-install

#確認是否設定完成
tanzu package repository get tanzu-tap-repository --namespace tap-install

tanzu package available list tap.tanzu.vmware.com --namespace tap-install



### Gradle 部分
# 把 gradle 下載下來
imgpkg copy -i gradle:latest --to-tar gradle.tar --include-non-distributable-layers

# 把鏡像塞到內部的 harbor
imgpkg copy --tar gradle.tar  --to-repo reg.microservice.tw/cicd-tools/gradle

9. 準備 Tanzu Application Platform 設定檔

profile: full

shared:
  ingress_domain: "microservice.tw" # eg: tap.corp.tanzu
  image_registry:
    project_path: "reg.microservice.tw/k8s-summit-2023" #"mypvt.azurecr.io/myimages"
    secret:
      name: registry-credentials #registry-credentials
      namespace: tap-install # tap-install
  kubernetes_version: "1.26" # Required when Kubernetes version is 1.25 or later.
  ca_cert_data: |
    -----BEGIN CERTIFICATE-----
    >>>填入 自簽的 Harbor 憑證 <<<
    -----END CERTIFICATE-----

ceip_policy_disclosed: true

local_source_proxy:
  repository: reg.microservice.tw/k8s-summit-2023

  push_secret:
   name: lsp-push-credentials
   namespace: tap-install
   create_export: true

  pull_secret:
   name: lsp-pull-credentials
   namespace: tap-install
   create_export: true

metadata_store:
  app_service_type: ClusterIP 
  ns_for_export_app_cert: "*"
buildservice:
  kp_default_repository: "reg.microservice.tw/k8s-summit-2023/build-service" 
  kp_default_repository_secret:
    name: registry-credentials
    namespace: tap-install 
   

scanning:
  metadataStore:
    url: "" 
grype:
  namespace: default 
  targetImagePullSecret: registry-credentials

supply_chain: basic #basic #testing #testing_scanning

ootb_supply_chain_basic:
  #ootb_supply_chain_testing:
  #ootb_supply_chain_testing_scanning:
  registry:
    server: "reg.microservice.tw"
    repository: "k8s-summit-2023"
  gitops:
    ssh_secret: "git-secret" # git-ssh
  cluster_builder: default
  service_account: default

tap_gui:
  service_type: ClusterIP
  ingressEnabled: "true"
  ingressDomain: "microservice.tw" 
  metadataStoreAutoconfiguration: true 
  app_config:
    app:
      baseUrl: https://tap-gui.microservice.tw 
    catalog:
      locations:
        - type: url
          target: https://github.com/ChunPingWang/tap-gui/blob/main/catalog-info.yaml
    auth:
      environment: development
      providers:
        github: # 參考文件：VMware https://docs.vmware.com/en/VMware-Tanzu-Application-Platform/1.6/tap/tap-gui-auth.html
          development: # 參考文件 : Backstage https://backstage.io/docs/auth/github/provider/
            clientId: [Github ClientID]
            clientSecret: [Github Secret]
    backend:
      baseUrl: https://tap-gui.microservice.tw
      cors:
        origin: https://tap-gui.microservice.tw

cnrs:
  domain_name: "apps.microservice.tw"
  domain_template: "{{.Name}}-{{.Namespace}}.{{.Domain}}"

excluded_packages:
  - policy.apps.tanzu.vmware.com
  - image-policy-webhook.signing.apps.tanzu.vmware.com
  - learningcenter.tanzu.vmware.com
  - workshops.learningcenter.tanzu.vmware.com
  
contour:
  envoy:
    service:
      type: LoadBalancer

10. 開始安裝 Tanzu Application Platform

#開始安裝Tanzu Application Platform
tanzu package install tap -p tap.tanzu.vmware.com \
  -v $TAP_VERSION \
  --values-file tap-values-full.yaml \
  -n tap-install

#確認是否已經全部安裝成功
tanzu package installed list -A

其他操作

# 更新設定
tanzu package installed update tap \
  --values-file tap-values-full.yaml \
  -n tap-install

# 刪除 TAP
tanzu package installed delete tap \
  -n tap-install

11. 後續設定

kubectl get svc -A

查看 tanzu-system-ingress 的 envoy 那行，將 external ip 寫入 /etc/hosts，範例如下，在此為 10.100.130.248 tap-gui-microservice.tw：

12. 設定應用部署 namespace的 Service Account

# 參考文件： https://docs.vmware.com/en/VMware-Tanzu-Application-Platform/1.6/tap/namespace-provisioner-legacy-manual-namespace-setup.html

先在應用部署 namespace 中設定 Harbor 登入資訊

tanzu secret registry add registry-credentials \
  --server $IMGPKG_REGISTRY_HOSTNAME_1 \
  --username $IMGPKG_REGISTRY_USERNAME_1 \
  --password $IMGPKG_REGISTRY_PASSWORD_1 \ 
  --namespace default

再設定 Service Account、Role Binding等

cat <<EOF | kubectl -n YOUR-NAMESPACE apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: tap-registry
  annotations:
    secretgen.carvel.dev/image-pull-secret: ""
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: e30K
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
secrets:
  - name: registry-credentials
imagePullSecrets:
  - name: registry-credentials
  - name: tap-registry
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: default-permit-deliverable
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: deliverable
subjects:
  - kind: ServiceAccount
    name: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: default-permit-workload
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: workload
subjects:
  - kind: ServiceAccount
    name: default
EOF

13. 設定 Tekton 使用下列 yaml 檔

# 參考文件：https://docs.vmware.com/en/VMware-Tanzu-Application-Platform/1.6/tap/getting-started-add-test-and-security.html

kubectl apply -f tekton-pipeline.yaml

yaml 檔內容

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: developer-defined-tekton-pipeline
  namespace: default
  labels:
    apps.tanzu.vmware.com/pipeline: test     # (!) required
spec:
  params:
    - name: source-url                       # (!) required
    - name: source-revision                  # (!) required
  tasks:
    - name: test
      params:
        - name: source-url
          value: $(params.source-url)
        - name: source-revision
          value: $(params.source-revision)
      taskSpec:
        params:
          - name: source-url
          - name: source-revision
        steps:
          - name: test
            image: reg.microservice.tw/cicd-tools/gradle
            script: |-
              cd `mktemp -d`

              wget -qO- $(params.source-url) | tar xvz -m
              ./mvnw test

14. 設定 Scan-Policy

# 參考文件：https://docs.vmware.com/en/VMware-Tanzu-Application-Platform/1.6/tap/getting-started-add-test-and-security.html

kubectl apply -f scan-policy.yaml

scan-policy.yaml 檔案內容如下：

---
apiVersion: scanning.apps.tanzu.vmware.com/v1beta1
kind: ScanPolicy
metadata:
  name: scan-policy
  namespace: default
  labels:
    'app.kubernetes.io/part-of': 'enable-in-gui'
spec:
  regoFile: |
    package main

    # Accepted Values: "Critical", "High", "Medium", "Low", "Negligible", "UnknownSeverity"
    #notAllowedSeverities := ["Critical", "High", "UnknownSeverity"]
    notAllowedSeverities := []
    ignoreCves := []

    contains(array, elem) = true {
      array[_] = elem
    } else = false { true }

    isSafe(match) {
      severities := { e | e := match.ratings.rating.severity } | { e | e := match.ratings.rating[_].severity }
      some i
      fails := contains(notAllowedSeverities, severities[i])
      not fails
    }

    isSafe(match) {
      ignore := contains(ignoreCves, match.id)
      ignore
    }

    deny[msg] {
      comps := { e | e := input.bom.components.component } | { e | e := input.bom.components.component[_] }
      some i
      comp := comps[i]
      vulns := { e | e := comp.vulnerabilities.vulnerability } | { e | e := comp.vulnerabilities.vulnerability[_] }
      some j
      vuln := vulns[j]
      ratings := { e | e := vuln.ratings.rating.severity } | { e | e := vuln.ratings.rating[_].severity }
      not isSafe(vuln)
      msg = sprintf("CVE %s %s %s", [comp.name, vuln.id, ratings])
    }
---
apiVersion: scanning.apps.tanzu.vmware.com/v1beta1
kind: ImageScan
metadata:
  name: sample-public-image-scan
spec:
  registry:
    image: "nginx:1.16"
  scanTemplate: public-image-scan-template
  scanPolicy: scan-policy

15. 部署應用，

先從 Github 下載

mkdir $HOME/workspace/apps

git clone https://github.com/ChunPingWang/tanzu-java-web-app.git

設定tanzu-java-web-app中的 config/workload.yaml

從電腦中的source code 執行

tanzu apps workload create tanzu-java-web-app \
  --file ./config/workload.yaml \
  --local-path . \
  --source-image reg.microservice.tw/k8s-summit-2023/tanzu-java-web-app-source

亦可直接從 Github拉取程式碼

tanzu apps workload create tanzu-java-web-app \
  —git-branch main 
  --git-repo https://github.com/ChunPingWang/tanzu-java-web-app.git \
  —label app.kubernetes.io/part-of=tanzu-java-web-app  \
  —type web \
  —namespace default

查看日誌與詳細資訊

tanzu apps workload tail tanzu-java-web-app --timestamp --since 1h

tanzu apps workload get tanzu-java-web-app

最下面一行URL需要加入到 /etc/hosts，即可以tanzu-java-web-app-default.apps.microservice.tw鏈結使用該應用程式

刪除應用

tanzu apps workload delete tanzu-java-web-app

16. 結論

Tanzu Application Platform 是一個複雜且快速成長的產品，後續我們將持續介紹其他功能與元件。下回見