Open in app

Sign in

Write

Sign in

在 Minikube 上快速安裝 Tanzu Application Platform 1.6

Rex Wang
26 min readOct 25, 2023

VMware 曾經在多年前發展過一套以容器技術為基礎的PaaS，稱為Cloud Foundry；2013年，與EMC、GE共同投資Pivotal後，VMware將自己的許多資產移交給Pivotal持續發展，除了大家熟悉的 Spring Framework之外，Cloud Foundry也是受到許多金融行庫喜愛的重要的容器平台產品，當時稱為Pivotal Cloud Foundry(簡稱 PCF)。PCF對於開發者的體驗堪稱完美，尤其是Java開發者，因此在VMware併購Pivotal之後，許多客戶希望PCF上便利的操作方式可以複製到K8s或是VMware Tanzu Kubernetes Grid上，而Tanzu Application Platform即是這個因素下，所誕生的產品。這篇文章，我將解釋VMware的Tanzu Application Platform的架構與在Minikube上的安裝過程。

Tanzu Application Platform 架構

說明：

  1. TAP是由多個開源組件所構建出來的平台，又為了不同的任務將這些組件進行組合，形成不同的 profile，共計有 full、light、iteration、build、run與view。full跟light字面上很容易理解，iteration做為開發使用，build則負責應用編譯與鏡像生成之用，run可部署在staging 或是production環境，提供GitOps的應用部署模式，最後的view就是用來查看每個叢集的運行狀況。

安裝準備：

  1. 硬體：CPU Intel i5 12代，記憶體 64G，SSD 1T，準系統華擎 MiniDesk
  2. 安裝Carvel：這是許多容器相關操作的工具集合，我個人推薦使用 Homebrew方式安裝(參考網址https://carvel.dev/)
  3. 安裝tanzu cli
brew update
brew install vmware-tanzu/tanzu/tanzu-cli
tanzu plugin install --group vmware-tap/default:v1.6.4

4. 設定 Harbor 憑證

sudo cp microservice.tw.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
sudo systemctl restart docker.service
ls /etc/ssl/certs | awk /microservice.tw/

5. 設定Ubuntu 最大檔案開啟數量

sudo sysctl fs.inotify.max_user_watches=524288
sudo sysctl fs.inotify.max_user_instances=512

Ubuntu 重新開機後，自動設定最大檔案開啟數量，請於 /etc/sysctl.conf中寫入下列兩行設定

fs.inotify.max_user_watches=524288
fs.inotify.max_user_instances=512

6. 啟動並設定 minikube

minikube start -p ite --cpus=16 --memory=32G  --insecure-registry="reg.microservice.tw"
minikube addons enable ingress -p ite                                                   
minikube addons enable ingress-dns -p ite
minikube tunnel -p ite

7. 暫停與繼續使用 minikube

minikube pause -p ite #關機前暫停

minikube unpause -p ite #開機後繼續

8. 下載與準備安裝檔，請於 Harbor 上建立四個 project：tanzu-application-platform、tanzu-cluster-essentials、cicd-tools與k8s-summit-2023，並從 Tanzu Network 下載 Cluster Essentials for VMware Tanzu 設定檔

mkdir $HOME/tanzu-cluster-essentials
tar -xvf cluster-essentials-bundle-1.6.4.tar -C $HOME/tanzu-cluster-essentials

# Set tanzunet as the source registry to copy the Tanzu Application Platform packages from.
export IMGPKG_REGISTRY_HOSTNAME_0=registry.tanzu.vmware.com
export IMGPKG_REGISTRY_USERNAME_0=[USERNAME]
export IMGPKG_REGISTRY_PASSWORD_0=[PASSWORD]

# The user’s registry for copying the Tanzu Application Platform package to.
export IMGPKG_REGISTRY_HOSTNAME_1=reg.microservice.tw
export IMGPKG_REGISTRY_USERNAME_1=[USERNAME]
export IMGPKG_REGISTRY_PASSWORD_1=[PASSWORD]
export TAP_VERSION=1.6.4
export REGISTRY_CA_PATH=$HOME/workspace/tap16/microservice.tw.crt

### Cluster Essentials 部分
# 參考文件：https://docs.vmware.com/en/Cluster-Essentials-for-VMware-Tanzu/1.6/cluster-essentials/deploy.html

#下載 Cluster Essentials 到電腦裡
imgpkg copy \
    -b $IMGPKG_REGISTRY_HOSTNAME_0/tanzu-cluster-essentials/cluster-essentials-bundle@sha256:2f538b69c866023b7d408cce6f0624c5662ee0703d8492e623b7fce10b6f840b \
    --to-tar cluster-essentials-bundle-$TAP_VERSION.tar \
    --include-non-distributable-layers
  
#上傳 Cluster Essentials 到地端的 Harbor  
imgpkg copy \
  --tar cluster-essentials-bundle-$TAP_VERSION.tar \
  --to-repo $IMGPKG_REGISTRY_HOSTNAME_1/tanzu-cluster-essentials/cluster-essentials-bundle \
  --include-non-distributable-layers \
  --registry-ca-cert-path $REGISTRY_CA_PATH

#安裝 Cluster Essentials
kubectl create namespace kapp-controller
kubectl create secret generic kapp-controller-config \
   --namespace kapp-controller \
   --from-file caCerts=$REGISTRY_CA_PATH
   
cd $HOME/tanzu-cluster-essentials
INSTALL_BUNDLE=$IMGPKG_REGISTRY_HOSTNAME_1/tanzu-cluster-essentials/cluster-essentials-bundle@sha256:2f538b69c866023b7d408cce6f0624c5662ee0703d8492e623b7fce10b6f840b \
  INSTALL_REGISTRY_HOSTNAME=$IMGPKG_REGISTRY_HOSTNAME_1 \
  INSTALL_REGISTRY_USERNAME=$IMGPKG_REGISTRY_USERNAME_1 \
  INSTALL_REGISTRY_PASSWORD=$IMGPKG_REGISTRY_PASSWORD_1 \
  ./install.sh

### Tanzu Application Platform 部分
# 參考文件：https://docs.vmware.com/en/VMware-Tanzu-Application-Platform/1.6/tap/install-offline-profile.html

mkdir -p $HOME/workspace/tap16
cd $HOME/workspace/tap16

#下載 Tanzu Application Platform 安裝檔
imgpkg copy \
  -b $IMGPKG_REGISTRY_HOSTNAME_0/tanzu-application-platform/tap-packages:$TAP_VERSION \
  --to-tar tap-packages-$TAP_VERSION.tar \
  --include-non-distributable-layers

#上傳 Tanzu Application Platform 安裝檔到自己的 Harbor
imgpkg copy \
  --tar tap-packages-$TAP_VERSION.tar \
  --to-repo $IMGPKG_REGISTRY_HOSTNAME_1/tanzu-application-platform/tap-packages \
  --include-non-distributable-layers \
  --registry-ca-cert-path $REGISTRY_CA_PATH


kubectl create ns tap-install

tanzu secret registry add tap-registry \
    --server   $IMGPKG_REGISTRY_HOSTNAME_1 \
    --username $IMGPKG_REGISTRY_USERNAME_1 \
    --password $IMGPKG_REGISTRY_PASSWORD_1 \
    --namespace tap-install \
    --export-to-all-namespaces \
    --yes

tanzu secret registry add registry-credentials \
    --server   $IMGPKG_REGISTRY_HOSTNAME_1 \
    --username $IMGPKG_REGISTRY_USERNAME_1 \
    --password $IMGPKG_REGISTRY_PASSWORD_1 \
    --namespace tap-install \
    --export-to-all-namespaces \
    --yes

#加入 TAP Repository
tanzu package repository add tanzu-tap-repository \
  --url $IMGPKG_REGISTRY_HOSTNAME_1/tanzu-application-platform/tap-packages:$TAP_VERSION \
  --namespace tap-install

#確認是否設定完成
tanzu package repository get tanzu-tap-repository --namespace tap-install

tanzu package available list tap.tanzu.vmware.com --namespace tap-install



### Gradle 部分
# 把 gradle 下載下來
imgpkg copy -i gradle:latest --to-tar gradle.tar --include-non-distributable-layers

# 把鏡像塞到內部的 harbor
imgpkg copy --tar gradle.tar  --to-repo reg.microservice.tw/cicd-tools/gradle

9. 準備 Tanzu Application Platform 設定檔

profile: full

shared:
  ingress_domain: "microservice.tw" # eg: tap.corp.tanzu
  image_registry:
    project_path: "reg.microservice.tw/k8s-summit-2023" #"mypvt.azurecr.io/myimages"
    secret:
      name: registry-credentials #registry-credentials
      namespace: tap-install # tap-install
  kubernetes_version: "1.26" # Required when Kubernetes version is 1.25 or later.
  ca_cert_data: |
    -----BEGIN CERTIFICATE-----
    >>>填入 自簽的 Harbor 憑證 <<<
    -----END CERTIFICATE-----

ceip_policy_disclosed: true

local_source_proxy:
  repository: reg.microservice.tw/k8s-summit-2023

  push_secret:
   name: lsp-push-credentials
   namespace: tap-install
   create_export: true

  pull_secret:
   name: lsp-pull-credentials
   namespace: tap-install
   create_export: true

metadata_store:
  app_service_type: ClusterIP 
  ns_for_export_app_cert: "*"
buildservice:
  kp_default_repository: "reg.microservice.tw/k8s-summit-2023/build-service" 
  kp_default_repository_secret:
    name: registry-credentials
    namespace: tap-install 
   

scanning:
  metadataStore:
    url: "" 
grype:
  namespace: default 
  targetImagePullSecret: registry-credentials

supply_chain: basic #basic #testing #testing_scanning

ootb_supply_chain_basic:
  #ootb_supply_chain_testing:
  #ootb_supply_chain_testing_scanning:
  registry:
    server: "reg.microservice.tw"
    repository: "k8s-summit-2023"
  gitops:
    ssh_secret: "git-secret" # git-ssh
  cluster_builder: default
  service_account: default

tap_gui:
  service_type: ClusterIP
  ingressEnabled: "true"
  ingressDomain: "microservice.tw" 
  metadataStoreAutoconfiguration: true 
  app_config:
    app:
      baseUrl: https://tap-gui.microservice.tw 
    catalog:
      locations:
        - type: url
          target: https://github.com/ChunPingWang/tap-gui/blob/main/catalog-info.yaml
    auth:
      environment: development
      providers:
        github: # 參考文件：VMware https://docs.vmware.com/en/VMware-Tanzu-Application-Platform/1.6/tap/tap-gui-auth.html
          development: # 參考文件 : Backstage https://backstage.io/docs/auth/github/provider/
            clientId: [Github ClientID]
            clientSecret: [Github Secret]
    backend:
      baseUrl: https://tap-gui.microservice.tw
      cors:
        origin: https://tap-gui.microservice.tw

cnrs:
  domain_name: "apps.microservice.tw"
  domain_template: "{{.Name}}-{{.Namespace}}.{{.Domain}}"

excluded_packages:
  - policy.apps.tanzu.vmware.com
  - image-policy-webhook.signing.apps.tanzu.vmware.com
  - learningcenter.tanzu.vmware.com
  - workshops.learningcenter.tanzu.vmware.com
  
contour:
  envoy:
    service:
      type: LoadBalancer

10. 開始安裝 Tanzu Application Platform

#開始安裝Tanzu Application Platform
tanzu package install tap -p tap.tanzu.vmware.com \
  -v $TAP_VERSION \
  --values-file tap-values-full.yaml \
  -n tap-install

#確認是否已經全部安裝成功
tanzu package installed list -A

其他操作

# 更新設定
tanzu package installed update tap \
  --values-file tap-values-full.yaml \
  -n tap-install

# 刪除 TAP
tanzu package installed delete tap \
  -n tap-install

11. 後續設定

kubectl get svc -A

查看 tanzu-system-ingress 的 envoy 那行，將 external ip 寫入 /etc/hosts，範例如下，在此為 10.100.130.248 tap-gui-microservice.tw：

12. 設定應用部署 namespace的 Service Account

# 參考文件： https://docs.vmware.com/en/VMware-Tanzu-Application-Platform/1.6/tap/namespace-provisioner-legacy-manual-namespace-setup.html

先在應用部署 namespace 中設定 Harbor 登入資訊

tanzu secret registry add registry-credentials \
  --server $IMGPKG_REGISTRY_HOSTNAME_1 \
  --username $IMGPKG_REGISTRY_USERNAME_1 \
  --password $IMGPKG_REGISTRY_PASSWORD_1 \ 
  --namespace default

再設定 Service Account、Role Binding等

cat <<EOF | kubectl -n YOUR-NAMESPACE apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: tap-registry
  annotations:
    secretgen.carvel.dev/image-pull-secret: ""
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: e30K
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
secrets:
  - name: registry-credentials
imagePullSecrets:
  - name: registry-credentials
  - name: tap-registry
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: default-permit-deliverable
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: deliverable
subjects:
  - kind: ServiceAccount
    name: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: default-permit-workload
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: workload
subjects:
  - kind: ServiceAccount
    name: default
EOF

13. 設定 Tekton 使用下列 yaml 檔

# 參考文件：https://docs.vmware.com/en/VMware-Tanzu-Application-Platform/1.6/tap/getting-started-add-test-and-security.html
kubectl apply -f tekton-pipeline.yaml

yaml 檔內容

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: developer-defined-tekton-pipeline
  namespace: default
  labels:
    apps.tanzu.vmware.com/pipeline: test     # (!) required
spec:
  params:
    - name: source-url                       # (!) required
    - name: source-revision                  # (!) required
  tasks:
    - name: test
      params:
        - name: source-url
          value: $(params.source-url)
        - name: source-revision
          value: $(params.source-revision)
      taskSpec:
        params:
          - name: source-url
          - name: source-revision
        steps:
          - name: test
            image: reg.microservice.tw/cicd-tools/gradle
            script: |-
              cd `mktemp -d`

              wget -qO- $(params.source-url) | tar xvz -m
              ./mvnw test

14. 設定 Scan-Policy

# 參考文件：https://docs.vmware.com/en/VMware-Tanzu-Application-Platform/1.6/tap/getting-started-add-test-and-security.html
kubectl apply -f scan-policy.yaml

scan-policy.yaml 檔案內容如下：

---
apiVersion: scanning.apps.tanzu.vmware.com/v1beta1
kind: ScanPolicy
metadata:
  name: scan-policy
  namespace: default
  labels:
    'app.kubernetes.io/part-of': 'enable-in-gui'
spec:
  regoFile: |
    package main

    # Accepted Values: "Critical", "High", "Medium", "Low", "Negligible", "UnknownSeverity"
    #notAllowedSeverities := ["Critical", "High", "UnknownSeverity"]
    notAllowedSeverities := []
    ignoreCves := []

    contains(array, elem) = true {
      array[_] = elem
    } else = false { true }

    isSafe(match) {
      severities := { e | e := match.ratings.rating.severity } | { e | e := match.ratings.rating[_].severity }
      some i
      fails := contains(notAllowedSeverities, severities[i])
      not fails
    }

    isSafe(match) {
      ignore := contains(ignoreCves, match.id)
      ignore
    }

    deny[msg] {
      comps := { e | e := input.bom.components.component } | { e | e := input.bom.components.component[_] }
      some i
      comp := comps[i]
      vulns := { e | e := comp.vulnerabilities.vulnerability } | { e | e := comp.vulnerabilities.vulnerability[_] }
      some j
      vuln := vulns[j]
      ratings := { e | e := vuln.ratings.rating.severity } | { e | e := vuln.ratings.rating[_].severity }
      not isSafe(vuln)
      msg = sprintf("CVE %s %s %s", [comp.name, vuln.id, ratings])
    }
---
apiVersion: scanning.apps.tanzu.vmware.com/v1beta1
kind: ImageScan
metadata:
  name: sample-public-image-scan
spec:
  registry:
    image: "nginx:1.16"
  scanTemplate: public-image-scan-template
  scanPolicy: scan-policy

15. 部署應用，

先從 Github 下載

mkdir $HOME/workspace/apps

git clone https://github.com/ChunPingWang/tanzu-java-web-app.git

設定tanzu-java-web-app中的 config/workload.yaml

從電腦中的source code 執行

tanzu apps workload create tanzu-java-web-app \
  --file ./config/workload.yaml \
  --local-path . \
  --source-image reg.microservice.tw/k8s-summit-2023/tanzu-java-web-app-source

亦可直接從 Github拉取程式碼

tanzu apps workload create tanzu-java-web-app \
  —git-branch main 
  --git-repo https://github.com/ChunPingWang/tanzu-java-web-app.git \
  —label app.kubernetes.io/part-of=tanzu-java-web-app  \
  —type web \
  —namespace default

查看日誌與詳細資訊

tanzu apps workload tail tanzu-java-web-app --timestamp --since 1h

tanzu apps workload get tanzu-java-web-app

最下面一行URL需要加入到 /etc/hosts，即可以tanzu-java-web-app-default.apps.microservice.tw鏈結使用該應用程式

刪除應用

tanzu apps workload delete tanzu-java-web-app

16. 結論

Tanzu Application Platform 是一個複雜且快速成長的產品，後續我們將持續介紹其他功能與元件。下回見

Sign up to discover human stories that deepen your understanding of the world.

Free

Distraction-free reading. No ads.

Organize your knowledge with lists and highlights.

Tell your story. Find your audience.

Membership

Read member-only stories

Support writers you read most

Earn money for your writing

Listen to audio narrations

Read offline with the Medium app

Rex Wang
Rex Wang

Written by Rex Wang

72 Followers
48 Following

職業生涯歷經不同身份與角色，應用開發是我無法忘情的工作之一；工作中的經驗與個人的學習，逐漸對於應用開發、專案執行有一些想法與感觸，因此想藉此園地留下紀錄

No responses yet

Write a response

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams