카테고리 없음

[AEWS] 10주차 - K8s 시크릿 관리 Update(With Vault)

summary-aws 2025. 4. 11. 21:44

1. vault 란?

2. Kubernetes에 Vault 설치
3. Jenkins + Vault (AppRole) 
4. ArgoCD + Vault Plugin (Kubernetes Auth/AppRole)
5. Vault Secrets Operator (VSO)

 

1. vault 란?

Vault는 주로 토큰(Token)을 기반으로 작동하며, 이 토큰은 클라이언트의 정책(Policy)과 연결되어 있습니다. 각 정책은 경로(path) 기반으로 설정되며, 정책 규칙은 클라이언트가 해당 경로에서 수행할 수 있는 작업과 접근 가능성을 제한합니다.

Vault에서는 토큰을 수동으로 생성해 클라이언트에 할당할 수도 있고, 클라이언트가 로그인하여 토큰을 직접 획득할 수도 있습니다.

 

Vault의 핵심 워크플로우

 


Vault의 주요 기능은 다음과 같습니다:

1. 안전한 비밀 저장 (Secure Secret Storage): → Static 시크릿
Vault는 임의의 key/value 형식의 시크릿을 저장할 수 있으며, 이 시크릿은 영구 저장소에 기록되기 전에 암호화됩니다. 따라서 저장소에 직접 접근하더라도 비밀을 열람할 수 없습니다. 
Vault는 Disk, Consul 등 다양한 저장소를 지원합니다.

2. 동적 비밀 (Dynamic Secrets):
Vault는 AWS나 SQL 데이터베이스와 같은 일부 시스템에 대해 요청 시 비밀을 동적으로 생성할 수 있습니다. 
예를 들어, 애플리케이션이 S3 버킷에 접근해야 할 때 Vault에 자격 증명을 요청하면, Vault는 해당 권한을 가진 AWS 키쌍을 생성해줍니다. 이 동적 시크릿은 일정 시간이 지나면 자동으로 폐기됩니다.

3. 데이터 암호화 (Data Encryption):
Vault는 데이터를 저장하지 않고 암호화 및 복호화를 수행할 수 있습니다. 
이를 통해 보안 팀은 암호화 매개변수를 정의하고, 개발자는 암호화된 데이터를 SQL 데이터베이스 등 외부 저장소에 안전하게 저장할 수 있습니다.

4. 임대 및 갱신 (Leasing and Renewal):
Vault에 저장된 모든 시크릿은 임대 기간(lease)이 설정되어 있으며, 이 기간이 끝나면 해당 비밀은 자동으로 폐기됩니다. 클라이언트는 내장된 갱신 API를 통해 임대를 연장할 수 있습니다.

5. 폐기 (Revocation):
Vault는 비밀 폐기를 기본적으로 지원합니다. 단일 비밀뿐만 아니라 특정 사용자에 의해 읽힌 모든 비밀, 또는 특정 유형의 모든 비밀 등 비밀의 계층 구조 전체를 폐기할 수 있습니다. 이 기능은 키 롤링이나 침입 발생 시 시스템을 신속하게 차단하는 데 유용합니다.

 

이전에 설치된 local 환경의 ArgoCD, Jenkins를 활용한다.

 

Vault 기본 구조의 이해

- 아키텍처

작동 원리

Vault의 기본 워크플로는 다음과 같은 4단계로 구성됩니다:

  1. 인증 (Authentication): 클라이언트가 Vault에 자신을 인증합니다.​
  2. 검증 (Validation): GitHub, LDAP 등 외부 시스템과 연동하여 신뢰성을 확인합니다.​
  3. 권한 확인 (Authorization): 클라이언트의 정책에 따라 어떤 경로에 어떤 작업이 가능한지 확인합니다.​
  4. 액세스 (Access): 정책에 따라 Secret, Key 등에 접근을 허용합니다.

아키텍처 및 운영 절차

1. 초기화 단계 (Initialization Phase)

  • API Endpoint: POST /v1/sys/init
  • 설명: Vault 서버를 처음 시작하면 잠겨 있으며, 이때 마스터 키와 초기 루트 토큰을 생성하는 초기화가 필요합니다.
  • 결과:
    • Unseal Key (여러 개) 발급
    • 초기 루트 토큰 발급

2. 봉인 해제 단계 (Unseal Phase)

  • API Endpoint: POST /v1/sys/unseal
  • 설명: 초기화 후 Vault는 봉인(sealed) 상태이며, 발급된 Unseal 키를 일정 개수 이상 입력하여 봉인을 해제해야 합니다.
  • 작동:
    • 여러 번 요청하여 충분한 수의 키를 입력해야 함 (예: 3 of 5 키)

3. 인증 및 로그인 (Authentication)

  • API Endpoint: POST /v1/auth/{method}/login
  • 예시: POST /v1/auth/userpass/login/myuser, POST /v1/auth/github/login 등
  • 설명: 유저 인증을 위한 단계로, 로그인하면 세션 토큰을 발급받음
  • 결과:
    • 인증 성공 시 client_token 반환

4. 비밀 접근 (Accessing Secrets)

  • 읽기: GET /v1/secret/data/{path}
  • 쓰기: POST /v1/secret/data/{path}
  • 설명:
    • 인증된 토큰으로 Header에 X-Vault-Token: {token} 포함하여 요청
    • 정책(Policy)에 따라 허용된 Secret 경로에 접근 가능

 

 

2. Kubernetes에 Vault 설치

ci/cd 테스트 시 배포된 내용을 활용한다.

Helm을 활용한 Vault 배포

# Create a Kubernetes namespace.
kubectl create namespace vault

# View all resources in a namespace.
kubectl get all --namespace vault

# Setup Helm repo
helm repo add hashicorp https://helm.releases.hashicorp.com

# Check that you have access to the chart.
helm search repo hashicorp/vault
# NAME                                    CHART VERSION   APP VERSION     DESCRIPTION                          
# hashicorp/vault                         0.30.0          1.19.0          Official HashiCorp Vault Chart       
# hashicorp/vault-secrets-gateway         0.0.2           0.1.0           A Helm chart for Kubernetes          
# hashicorp/vault-secrets-operator        0.10.0          0.10.0          Official Vault Secrets Operator Chart

cat <<EOF > override-values.yaml
global:
  enabled: true
  tlsDisable: true  # Disable TLS for demo purposes

server:
  image:
    repository: "hashicorp/vault"
    tag: "1.19.0"

  standalone:
    enabled: true
    replicas: 1  # 단일 노드 실행

    config: |
      ui = true
      disable_mlock = true
      cluster_name = "vault-local"

      listener "tcp" {
        address = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_disable = 1
      }

      storage "raft" { # Raft 구성 권장
        path = "/vault/data"
        node_id = "vault-dev-node-1"
      }
  service:
    enabled: true
    type: NodePort
    port: 8200
    targetPort: 8200
    nodePort: 30000   # Kind에서 열어둔 포트 중 하나 사용

injector:
  enabled: true

ui:
  enabled: true
  serviceType: "NodePort"
EOF

# Helm Install 실행
helm upgrade vault hashicorp/vault -n vault -f override-values.yaml --install

# 네임스페이스 변경 : vault
kubens vault
Context "kind-myk8s" modified.
Active namespace is "vault".

# 배포확인
kubectl get pods,svc,pvc -n vault

NAME                                        READY   STATUS    RESTARTS   AGE
pod/vault-0                                 0/1     Running   0          2m14s
pod/vault-agent-injector-56459c7545-xvss6   1/1     Running   0          2m15s

NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
service/vault                      NodePort    10.96.33.168    <none>        8200:30011/TCP,8201:31814/TCP   46s
service/vault-agent-injector-svc   ClusterIP   10.96.45.133    <none>        443/TCP                         2m15s
service/vault-internal             ClusterIP   None            <none>        8200/TCP,8201/TCP               2m15s
service/vault-ui                   NodePort    10.96.243.200   <none>        8200:32730/TCP                  2m15s

NAME                                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/data-vault-0   Bound    pvc-d183d325-c59b-4cc8-a0cb-2ace85dc2ea3   10Gi       RWO            standard       <unset>                 2m15s

kubectl exec -ti vault-0 -n vault -- vault status 
 - 상태확인
Key                     Value
---                     -----
Seal Type               shamir
Initialized             false
Sealed                  true
Total Shares            0
Threshold               0
Unseal Progress         0/0
Unseal Nonce            n/a
Version                 1.19.0
Build Date              2025-03-04T12:36:40Z
Storage Type            raft
Removed From Cluster    false
HA Enabled              true
command terminated with exit code 2

 

unseal 내용

cat <<EOF > init-unseal.sh
#!/bin/bash

# Vault Pod 이름
VAULT_POD="vault-0"

# Vault 명령 실행
VAULT_CMD="kubectl exec -ti \$VAULT_POD -n vault -- vault"

# 출력 저장 파일
VAULT_KEYS_FILE="./vault-keys.txt"
UNSEAL_KEY_FILE="./vault-unseal-key.txt"
ROOT_TOKEN_FILE="./vault-root-token.txt"

# Vault 초기화 (Unseal Key 1개만 생성되도록 설정)
\$VAULT_CMD operator init -key-shares=1 -key-threshold=1 | sed \$'s/\\x1b\\[[0-9;]*m//g' | tr -d '\r' > "\$VAULT_KEYS_FILE"

# Unseal Key / Root Token 추출
grep 'Unseal Key 1:' "\$VAULT_KEYS_FILE" | awk -F': ' '{print \$2}' > "\$UNSEAL_KEY_FILE"
grep 'Initial Root Token:' "\$VAULT_KEYS_FILE" | awk -F': ' '{print \$2}' > "\$ROOT_TOKEN_FILE"

# Unseal 수행
UNSEAL_KEY=\$(cat "\$UNSEAL_KEY_FILE")
\$VAULT_CMD operator unseal "\$UNSEAL_KEY"

# 결과 출력
echo "[🔓] Vault Unsealed!"
echo "[🔐] Root Token: \$(cat \$ROOT_TOKEN_FILE)"
EOF

# 실행 권한 부여
chmod +x init-unseal.sh

# 실행
./init-unseal.sh


Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            1
Threshold               1
Version                 1.19.0
Build Date              2025-03-04T12:36:40Z
Storage Type            raft
Cluster Name            vault-local
Cluster ID              d0fe99af-1ef7-eebf-f58c-69acf73a3754
Removed From Cluster    false
HA Enabled              true
HA Cluster              n/a
HA Mode                 standby
Active Node Address     <none>
Raft Committed Index    32
Raft Applied Index      32
[🔓] Vault Unsealed!
[🔐] Root Token: hvs.

 

# 1. 바이너리 다운로드
curl -O https://releases.hashicorp.com/vault/1.19.1/vault_1.19.1_linux_amd64.zip

# 2. 압축 해제
unzip vault_1.19.1_linux_amd64.zip

# 3. 실행 파일 이동
sudo mv vault /usr/local/bin/

# 4. 확인
vault version

#서버 등록
export VAULT_ADDR='http://localhost:30000'

 

KV 시크릿 엔진 활성화 및 샘플 구성

# KV v2 형태로 엔진 활성화
vault secrets enable -path=secret kv-v2

# 샘플 시크릿 저장
vault kv put secret/sampleapp/config \
  username="demo" \
  password="p@ssw0rd"
  
# 입력된 데이터 확인
vault kv get secret/sampleapp/config

secret/data/sampleapp/config

======= Metadata =======
Key                Value
---                -----
created_time       2025-04-12T02:47:24.710253183Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
password    p@ssw0rd
username    demo

 

Vault AppRole 방식 인증 구성

# 1. AppRole 인증 방식 활성화
vault auth enable approle || echo "AppRole already enabled"
vault auth list

# 2. 정책 생성
vault policy write sampleapp-policy - <<EOF
path "secret/data/sampleapp/*" {
  capabilities = ["read"]
}
EOF

# 3. AppRole Role 생성
vault write auth/approle/role/sampleapp-role \
  token_policies="sampleapp-policy" \
  secret_id_ttl="1h" \
  token_ttl="1h" \
  token_max_ttl="4h"

# 4. Role ID 및 Secret ID 추출 및 저장
ROLE_ID=$(vault read -field=role_id auth/approle/role/sampleapp-role/role-id)
SECRET_ID=$(vault write -f -field=secret_id auth/approle/role/sampleapp-role/secret-id)

echo "ROLE_ID: $ROLE_ID"
echo "SECRET_ID: $SECRET_ID"

ROLE_ID: 7add8c18-8a99-5629-3c7f-5d22d7eaa396
SECRET_ID: 43b0afcd-62f7-5df4-1871-55ff0ab64b05

# 5. 파일로 저장
mkdir -p approle-creds
echo "$ROLE_ID" > approle-creds/role_id.txt
echo "$SECRET_ID" > approle-creds/secret_id.txt


# 6. (옵션) Kubernetes Secret으로 저장
kubectl create secret generic vault-approle -n vault \
  --from-literal=role_id="${ROLE_ID}" \
  --from-literal=secret_id="${SECRET_ID}" \
  --save-config \
  --dry-run=client -o yaml | kubectl apply -f -

 

3. Jenkins + Vault (AppRole) - CI

 - 실습환경은 기존에 설치해 놓은 Jenkins을 활용 

plugin 설치

 

 ROLE_ID, SECRET_ID 발급

# Role ID 확인 및 Secret ID 신규 발급
ROLE_ID=$(vault read -field=role_id auth/approle/role/sampleapp-role/role-id)
SECRET_ID=$(vault write -f -field=secret_id auth/approle/role/sampleapp-role/secret-id)

echo "ROLE_ID: $ROLE_ID"
echo "SECRET_ID: $SECRET_ID"

 

Vault URL 등록

 

파이프라인 실행

pipeline {
  agent any

  environment {
    VAULT_ADDR = 'http://172.21.19.102:30000' // OS IP  }

  stages {
    stage('Read Vault Secret') {
      steps {
        withVault([
          vaultSecrets: [
            [
              path: 'secret/sampleapp/config',
              engineVersion: 2,
              secretValues: [
                [envVar: 'USERNAME', vaultKey: 'username'],
                [envVar: 'PASSWORD', vaultKey: 'password']
              ]
            ]
          ],
          configuration: [
            vaultUrl: "${VAULT_ADDR}",
            vaultCredentialId: 'vault-approle-creds'
          ]
        ]) {
          sh '''
            echo "Username from Vault: $USERNAME"
            echo "Password from Vault: $PASSWORD"
          '''
          script {
            echo "Username (env): ${env.USERNAME}"
            echo "Password (env): ${env.PASSWORD}"
          }
        }
      }
    }
  }
}

 

 

 

4. ArgoCD + Vault Plugin (Kubernetes Auth/AppRole)

  • ArgoCD Vault Plugin 소개
    • Argo CD에는 다양한 시크릿 관리 도구(HashiCorp Vault, IBM Cloud Secrets Manager, AWS Secrets Manager 등)플러그인을 통해 Kubernetes 리소스에 주입할 수 있도록 지원합니다.
    • 플러그인을 통해 Operator 또는 CRD(Custom Resource Definition)에 의존하지 않고 GitOps와 Argo CD로 시크릿 관리 문제를 해결할 수 있습니다.
    • 특히 Secret 뿐만 아니라, deployment, configMap 또는 기타 Kubernetes 리소스에도 사용할 수 있습니다.

 

ArgoCD Vault Plugin을 위한 Credentials 활성화 - AppRole 인증

Jenkins에서 활용한 Role, Seccret 활용

 

Step 1. ArgoCD Vault Plugin을 위한 Credentials 활성화 - AppRole 인증

kubectl apply -f - <<EOF
kind: Secret
apiVersion: v1
metadata:
  name: argocd-vault-plugin-credentials
  namespace: argocd
type: Opaque
stringData:
  VAULT_ADDR: "http://vault.vault:8200"
  AVP_TYPE: "vault"
  AVP_AUTH_TYPE: "approle"
  AVP_ROLE_ID: 7add8c18-8a99-5629-3c7f-5d22d7eaa396 #Role_ID
  AVP_SECRET_ID: 43b0afcd-62f7-5df4-1871-55ff0ab64b05 #Secret_ID
EOF

 

 

Step 2. ArgoCD Vault Plugin 설치

git clone https://github.com/hyungwook0221/argocd-vault-plugin.git
cd argocd-vault-plugin/manifests/cmp-sidecar

# 예전 문법으로 적용된 부분을 edit fix 명령으로 현행화
# kustomize edit fix

# argocd 네임스페이스 설정
kubens argocd

# 생성될 메니페스트 파일에 대한 확인
kubectl kustomize .

# -k 옵션으로 kusomize 실행
kubectl apply -n argocd -k .

 

Step 3. 샘플 Application 배포하여 Vault와 동기화

GitHub에 저장된 Helm Repo을 배포하며, Helm 메니페스트 내에 변수로 치환된 값(username/password)을 CD 단계에서 Vault 통해서 읽고 렌더링하여 배포

kubectl apply -n argocd -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: argocd
    server: https://kubernetes.default.svc
  project: default
  source:
    path: infra/helm
    repoURL: https://github.com/hyungwook0221/spring-boot-debug-app
    targetRevision: main
    plugin:
      name: argocd-vault-plugin-helm
      env:
        - name: HELM_ARGS
          value: -f new-values.yaml
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
EOF

Application 배포시 참조하는 new-values.yaml 확인

serviceAccount:
  create: true

image:
  repository: luafanti/spring-boot-debug-app
  tag: main
  pullPolicy: IfNotPresent

replicaCount: 1

resources:
  memoryRequest: 256Mi
  memoryLimit: 512Mi
  cpuRequest: 500m
  cpuLimit: 1

probes:
  liveness:
    initialDelaySeconds: 15
    path: /actuator/health/liveness
    failureThreshold: 3
    successThreshold: 1
    timeoutSeconds: 3
    periodSeconds: 5
  readiness:
    initialDelaySeconds: 15
    path: /actuator/health/readiness
    failureThreshold: 3
    successThreshold: 1
    timeoutSeconds: 3
    periodSeconds: 5

ports:
  http:
    name: http
    value: 8080
  management:
    name: management
    value: 8081

envs:
  - name: VAULT_SECRET_USER
    value: <path:secret/data/sampleapp/config#username>
  - name: VAULT_SECRET_PASSWORD
    value: <path:secret/data/sampleapp/config#password>

log:
  level:
    spring: "info"
    service: "info"

 

배포된 설정 값 확인.

 

5. Vault Secrets Operator (VSO)

  •  VSO란?
    • 개요오퍼레이터는 소스 시크릿 데이터를 대상 Kubernetes Secret에 직접 작성하며, 소스에 변경 사항이 발생할 경우 해당 내용을 대상에도 수명 주기 동안 지속적으로 반영합니다. 이렇게 함으로써 애플리케이션은 대상 시크릿에만 접근하면 그 안의 시크릿 데이터를 사용할 수 있습니다.
    • Vault Secrets Operator는 지원하는 Custom Resource Definitions(CRD) 집합의 변경 사항을 감시하여 작동합니다. 각 CRD는 시크릿의 지원되는 소스 중 하나에서 Kubernetes Secret으로 동기화할 수 있도록 필요한 사양을 제공합니다.
    • 기능
      • 여러 시크릿 소스로부터의 동기화 지원
      • 자동 시크릿 일탈 감지 및 수정
      • Deployment, ReplicaSet, StatefulSet Kubernetes 리소스 유형에 대한 자동 시크릿 교체
      • 오퍼레이터 모니터링을 위한 Prometheus 전용 계측 지원
      • Helm 또는 Kustomize를 통한 설치 지원
      • 시크릿 데이터 변환 지원
    • Vault Secrets Operator가 지원하는 주요 기능은 다음과 같습니다:
    • 지원되는 Kubernetes 배포판
      • Amazon Elastic Kubernetes Service (EKS)
      • Google Kubernetes Engine (GKE)
      • Microsoft Azure Kubernetes Service (AKS)
      • Red Hat OpenShift (공식 인증됨)
    • Vault Secrets Operator는 다음과 같은 호스팅된 Kubernetes 환경에서 성공적으로 테스트되었습니다:
  • Vault Secrets Operator(VSO)는 Kubernetes Secrets에서 Vault secrets 및 HCP Vault Secrets Apps를 네이티브하게 사용할 수 있도록 Pods에 제공해줍니다.

 Vault Static Secrets vs Dynamic Secrets 비교

   Static Secrets Dynamic Secrets
정의 Vault에 수동으로 저장된 Key-Value 형태의 정적 비밀 Vault가 외부 시스템(DB, 클라우드 등)과 연동하여 자동 생성되는 일회성 또는 임시 자격 증명
저장 위치 KV (Key-Value) Secret Engine (v1 또는 v2) DB, AWS, GCP, RabbitMQ 등 각종 Dynamic Secret Engine
생성 방식 사용자가 수동으로 생성 (vault kv put) Vault가 요청 시 즉시 생성 (vault read)
수명 (TTL) 기본적으로 만료 없음 (수동으로 삭제해야 함) 자동 만료(lease) 또는 TTL 기반 설정
회전(Rotation) 수동으로 업데이트 필요 자동 회전 및 만료 후 자동 폐기 가능
보안성 비교적 낮음 (누출 시 재사용 가능) 매우 높음 (임시 비밀번호 또는 토큰, 사용 후 폐기)
예시 사용처 API 키, Slack 토큰, 환경설정 값 등 DB 접속 계정, AWS IAM 사용자, 클라우드 토큰 등
접근 방식 vault kv get secret/<path> vault read database/creds/<role>
사용 난이도 쉬움 (단순 저장/조회) 중간~높음 (설정 필요, 연동 대상 구성 필수)

 

Dynamic Secrets 작동확인

 

Step1. VSO 배포를 위한 Chart Values 파일 작성

# vault-operator-values.yaml
defaultVaultConnection:
  enabled: true
  address: "http://vault.vault.svc.cluster.local:8200"
  skipTLSVerify: false
controller:
  manager:
    clientCache:
      persistenceModel: direct-encrypted
      storageEncryption:
        enabled: true
        mount: k8s-auth-mount
        keyName: vso-client-cache
        transitMount: demo-transit
        kubernetes:
          role: auth-role-operator
          serviceAccount: vault-secrets-operator-controller-manager
          tokenAudiences: ["vault"]

helm install vault-secrets-operator hashicorp/vault-secrets-operator \
  -n vault-secrets-operator-system \
  --create-namespace \
  --values vault-operator-values.yaml
  
#
kubectl get-all -n vault-secrets-operator-system

#
kubectl get pod -n vault-secrets-operator-system      
NAME                                                         READY   STATUS    RESTARTS   AGE
vault-secrets-operator-controller-manager-7f67cd89fd-pjk5f   2/2     Running   0          34s

kubectl describe pod -n vault-secrets-operator-system                                       
...
Service Account:  vault-secrets-operator-controller-manager
...
Containers:
  kube-rbac-proxy:
    Container ID:  containerd://db3eae7b836fb4f1b4236c494c8fa96ada94769a6c602e1a150c75293a6a4162
    Image:         quay.io/brancz/kube-rbac-proxy:v0.18.1
  ...
  manager:
    Container ID:  containerd://1ab1545fb4bd86ac52d6c7609a3e962cd2d1a81daa9bbd9c82f79d9a0d8b6466
    Image:         hashicorp/vault-secrets-operator:0.10.0
...

#
kubectl rbac-tool lookup vault-secrets-operator-controller-manager
  SUBJECT                                   | SUBJECT TYPE   | SCOPE       | NAMESPACE                     | ROLE                                        | BINDING                                             
--------------------------------------------+----------------+-------------+-------------------------------+---------------------------------------------+-----------------------------------------------------
  vault-secrets-operator-controller-manager | ServiceAccount | ClusterRole |                               | vault-secrets-operator-proxy-role           | vault-secrets-operator-proxy-rolebinding            
  vault-secrets-operator-controller-manager | ServiceAccount | ClusterRole |                               | vault-secrets-operator-manager-role         | vault-secrets-operator-manager-rolebinding          
  vault-secrets-operator-controller-manager | ServiceAccount | Role        | vault-secrets-operator-system | vault-secrets-operator-leader-election-role | vault-secrets-operator-leader-election-rolebinding  

#
kubectl rolesum vault-secrets-operator-controller-manager -n vault-secrets-operator-system

#
kubectl get vaultauth -n vault-secrets-operator-system vault-secrets-operator-default-transit-auth -o jsonpath='{.spec}' | jq
{
  "kubernetes": {
    "audiences": [
      "vault"
    ],
    "role": "auth-role-operator",
    "serviceAccount": "vault-secrets-operator-controller-manager",
    "tokenExpirationSeconds": 600
  },
  "method": "kubernetes",
  "mount": "k8s-auth-mount",
  "storageEncryption": {
    "keyName": "vso-client-cache",
    "mount": "demo-transit"
  },
  "vaultConnectionRef": "default"
}

kubectl get vaultconnection -n vault-secrets-operator-system default -o jsonpath='{.spec}' | jq
{
  "address": "http://vault.vault.svc.cluster.local:8200",
  "skipTLSVerify": false
}

 

Step3. PostgreSQL 설치 (Bitnami Helm Chart)

kubectl create ns postgres

helm repo add bitnami https://charts.bitnami.com/bitnami

helm upgrade --install postgres bitnami/postgresql \
  --namespace postgres \
  --set auth.audit.logConnections=true \
  --set auth.postgresPassword=secret-pass
  
# psql 로그인 확인
kubectl exec -it -n postgres postgres-postgresql-0 -- sh -c "PGPASSWORD=secret-pass psql -U postgres -h localhost -c '\l'"

 

Step4. Kubernetes Auth Method 설정

#
kubectl exec -it vault-0 -n vault -- cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
-----BEGIN CERTIFICATE-----
MIIDBTCCAe2gAwIBAgIIfA5oV86RmOswDQYJKoZIhvcNAQELBQAwFTETMBEGA1UE
AxMKa3ViZXJuZXRlczAeFw0yNTA0MTAwMDUzMTdaFw0zNTA0MDgwMDU4MTdaMBUx
EzARBgNVBAMTCmt1YmVybmV0ZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQDSUCHyBXVhBb91uZKdqwFAussfSxqPadYhrj659fp/XuJQrRrOMLZAh5eP
6MCkeYbHeMY2stwBzuwzSzyaxkJdKiR5yKhtIHT7vpYn4XT0bNdyWWKAHkjyqjsk
DYkAbwQ1xksL3DwArRfUSdZPCVsDLhu+G3wbGKiW64q0Rb1pEk/gK2AhxnSSqoeR
YNzh10FA/CxS2wiLFfsm8NAn4KHG134AKY0Vxb/EkXnL9b2jRK+BfSh/sSpuEzN3
16Xy1fC5UX/xCdTvwx+57xOoFhymaF0aVISSavzrkaYd8kaPwAKtoAlxvDmDrVhv
YKRfCTerHUCOL3IPQqChIZQIUvoVAgMBAAGjWTBXMA4GA1UdDwEB/wQEAwICpDAP
BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSDK0FYEL+FNdgIpMhCnj9g2CkAIjAV
BgNVHREEDjAMggprdWJlcm5ldGVzMA0GCSqGSIb3DQEBCwUAA4IBAQC5/EdWTRD6
SoghjPe/bKKfQzFxPmEZVTb0JI0CWVUZorGzFcHCMCXMOf/XuECkK5tqn7QR/Nw8
yy3XvmT6DY+J2GshrusJXfMEOJ9RQIwDBJKHoZqbcyI/w294Ecp1Mpivb8WeI2FW
wMsaDoswQ3H/0zrEBwhhpUAWcmjm6w4/7Tsid9t1DsQrp9xH37wU/uP07HyEIKPS
TCQ2tUueHSVihNhCg7RejjbiGy8EUZV6aGIrtFgss9Mzk+5m+1KXm8o+Lj/YNYFx
JFeII2h3ZptO4I9vHrohXKVw0esHNzdHQPeGEXCWHTd+K38jIJPfggA8Yu1htbUa
MKEIbcT53UHH
-----END CERTIFICATE-----

# 해당 ca.crt 는 k8s 의 루트인증서 정보
kubectl exec -it vault-0 -n vault -- cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt > vso-ca.crt
openssl x509 -in vso-ca.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 8939197036714105067 (0x7c0e6857ce9198eb)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
...

# vault 서비스 어카운트의 token 확인
kubectl exec -it vault-0 -n vault -- cat /var/run/secrets/kubernetes.io/serviceaccount/token
eyJhbGciOiJSUzI1NiIsImtpZCI6InJBT1dvdDVZME1wMjZ4NmJuZkVvNk9nc1BGY1FIVDgyaUllUy1VN05oejgifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzc1ODAzNjMyLCJpYXQiOjE3NDQyNjc2MzIsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwianRpIjoiZTJkOTdiYTItNTMzMi00ZDFlLWJjYWMtYzg3NDdmNGQ2MWIwIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJ2YXVsdCIsIm5vZGUiOnsibmFtZSI6Im15azhzLWNvbnRyb2wtcGxhbmUiLCJ1aWQiOiI5ZGZkYTg5YS1hYzZkLTRmNTYtYjlhOC03MTg2YTI3MWVjZjMifSwicG9kIjp7Im5hbWUiOiJ2YXVsdC0wIiwidWlkIjoiNGFlMTVhZTctOTI0NS00OTZkLWE0ZDItYWI3ZTk4NDA1Yjg1In0sInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJ2YXVsdCIsInVpZCI6ImNjYjgwNjJkLTQ4OTctNDI1MC1hYzg3LTYxMTAxYWQ0YjI4NyJ9LCJ3YXJuYWZ0ZXIiOjE3NDQyNzEyMzl9LCJuYmYiOjE3NDQyNjc2MzIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDp2YXVsdDp2YXVsdCJ9.dekUKWseyWNgYO4uUi1MHI16BvHOAmeO61gxO-LIEK7m5ahG72wLwUnl5G22eRlA8T9X2DnvB8tIPyGDawzkCJWve7PTq-_76QOXZO8a3-KUY9kJ3gI-GymN9jwV9eUxXoFGqp73WxZCK-derP6aj5WZirSCHb1YQnioKmxQ4frhQVYi-GDLLz6A2OH-MnZIM68LvFPxK06Id2McMlqOFnV3JCDSzxvyEtnANQgdNWXIit-o6FFzDvKhlfwN6Yrjs9AO7JEyFeQy7M-C2_mQq_Xqht-cO28b8tA4kEJLIdWSIfu1LAWP06YZYeEbQM-8qZ9HoXR1L-QFtonMLkJ1pg

# https://jwt.io/ 에서 token 디코드 확인 시 PAYLOAD 부분
{
  "aud": [
    "https://kubernetes.default.svc.cluster.local"
  ],
  "exp": 1775803632,
  "iat": 1744267632,
  "iss": "https://kubernetes.default.svc.cluster.local",
  "jti": "e2d97ba2-5332-4d1e-bcac-c8747f4d61b0",
  "kubernetes.io": {
    "namespace": "vault",
    "node": {
      "name": "myk8s-control-plane",
      "uid": "9dfda89a-ac6d-4f56-b9a8-7186a271ecf3"
    },
    "pod": {
      "name": "vault-0",
      "uid": "4ae15ae7-9245-496d-a4d2-ab7e98405b85"
    },
    "serviceaccount": {
      "name": "vault",
      "uid": "ccb8062d-4897-4250-ac87-61101ad4b287"
    },
    "warnafter": 1744271239
  },
  "nbf": 1744267632,
  "sub": "system:serviceaccount:vault:vault"
}

# vault SA 토큰 만료 기간 : 대략 1시간(3600초)+7초
k get pod -n vault vault-0 -o yaml
...
 - name: kube-api-access-9rqgv
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt



kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/sh
----------------------------------------------------------------
# 최초 설치시 획득한 vault root token 사용. 
# 예시) hvs.hnlFKWjE10FOyrwLMeK9PrCC
vault login

# Kubernetes 인증 메서드 활성화
vault auth enable -path k8s-auth-mount kubernetes

#
vault auth list
Path               Type          Accessor                    Description                Version
----               ----          --------                    -----------                -------
approle/           approle       auth_approle_03723e90       n/a                        n/a
k8s-auth-mount/    kubernetes    auth_kubernetes_a2ee4150    n/a                        n/a
token/             token         auth_token_de4f4e43         token based credentials    n/a


# Kubernetes 클러스터 정보 구성
vault write auth/k8s-auth-mount/config \
  kubernetes_host="https://kubernetes.default.svc:443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"

# 설정 확인  
vault read auth/k8s-auth-mount/config
Key                                  Value
---                                  -----
disable_iss_validation               true
disable_local_ca_jwt                 false
issuer                               n/a
kubernetes_ca_cert                   -----BEGIN CERTIFICATE-----
MIIDBTCCAe2gAwIBAgIIa+jKQHykinUwDQYJKoZIhvcNAQELBQAwFTETMBEGA1UE
AxMKa3ViZXJuZXRlczAeFw0yNTAzMjUxMjAzMTNaFw0zNTAzMjMxMjA4MTNaMBUx
EzARBgNVBAMTCmt1YmVybmV0ZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC150wPhEtvw5vx0pzS0YJO7tybw907Klqq/5wtLHXQUwMcyFDdjKjTvSo7
taCtZ7W7c3ywsrOB3snnmSPUs727NlKYEN9TerDv3Iz05xentD6QiP4YhDMbYKkt
njNRCh5rt5kyVzi5Cz7cdmQjFUuXQ4kBoI1swxUyRZOqUlnSfJVexyAIAkXHaKgN
j0zyaUN68RX5zVh4UpK00fX8TQCvdvJqNP5WQtyDZhRjQDtxC39qRZEeTfcSMsr6
7J989/oQbAbLIRQiYFVZH5Sll/f8Mmr4eLIq41Yaetff/PvAgMIBmbkIdwKn0DKg
001IwlbFa3q671SUlCw1LUCYoA7hAgMBAAGjWTBXMA4GA1UdDwEB/wQEAwICpDAP
BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBQJuB++DVqQPWjs3bzJ4llu66rbnzAV
BgNVHREEDjAMggprdWJlcm5ldGVzMA0GCSqGSIb3DQEBCwUAA4IBAQAdneEakTPE
2OIb7TCUuLlM2KaD5Thv6SGX6sIZGfdFczqt4wHIjpVNCDTZAAexw4kgwOxdX6d+
it/X+4l0viJSn1TKounISNAZbDxF+9Kq0noJjURW1AtMpCN763lTAvatzWfBypdc
BgUj6PtkTRIF0kh62IWkjPCQbPX8mo6lnZvvgzWBT06Q+qnw9EmaHzJWHFYr8GJC
jXn35XMEJWAHnsl+a8x2UbgexfiD07+kfk9F7SmKEq1ThX/E2FF67RZHAtG7Cyk/
HTtVti1wH8TDiZZtph4j3kFcZFOMfvHZ8REIkn7BW4aY7UJwIrC4nAdmXnU6Fpa4
Ig2axnfQP0m8
-----END CERTIFICATE-----
kubernetes_host                      https://kubernetes.default.svc:443
pem_keys                             []
token_reviewer_jwt_set               true
use_annotations_as_alias_metadata    false

----------------------------------------------------------------

 

Step5. Vault Kubernetes Auth Role 설정

vault write auth/k8s-auth-mount/role/auth-role \
   bound_service_account_names=demo-dynamic-app \
   bound_service_account_namespaces=demo-ns \
   token_ttl=0 \
   token_period=120 \
   token_policies=demo-auth-policy-db \
   audience=vault

 

Step6. Vault Database Secret Engine 설정

# demo-db라는 경로로 Database Secret Engine을 활성화
vault secrets enable -path=demo-db database

# PostgreSQL 연결 정보 등록
# 해당 과정은 postgres가 정상적으로 동작 시 적용 가능
# allowed_roles: 이후 설정할 Role 이름 지정
vault write demo-db/config/demo-db \
   plugin_name=postgresql-database-plugin \
   allowed_roles="dev-postgres" \
   connection_url="postgresql://{{username}}:{{password}}@postgres-postgresql.postgres.svc.cluster.local:5432/postgres?sslmode=disable" \
   username="postgres" \
   password="secret-pass"
  
# DB 사용자 동적 생성 Role 등록
# 해당 Role 사용 시 Vault가 동적으로 사용자 계정과 비밀번호를 생성 가능
# TTL은 생성된 자격증명의 유효 시간 (30초~10분)
## creation_statements=... : Vault가 계정을 자동 생성할 때 실행할 SQL 문. {{name}}, {{password}}, {{expiration}}은 Vault가 자동으로 채워줌.
## revocation_statements=... : Vault가 동적 사용자 계정을 폐기(revoke)할 때 실행할 SQL 문. 권한을 모두 회수.
## default_ttl="1m" / max_ttl="1m" : 기본 1분만 유효합니다. 갱신을 안 하면 1분 후 자동 만료 (계정도 자동 삭제). 최대 TTL도 1분이므로 연장도 최대 1분까지만.
vault write demo-db/roles/dev-postgres \
   db_name=demo-db \
   creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
      GRANT ALL PRIVILEGES ON DATABASE postgres TO \"{{name}}\";" \
   revocation_statements="REVOKE ALL ON DATABASE postgres FROM  \"{{name}}\";" \
   backend=demo-db \
   name=dev-postgres \
   default_ttl="1m" \
   max_ttl="1m"

# 정책 설정: DB 자격증명 읽기 권한
# demo-db/creds/dev-postgres 경로에 대한 read 권한 부여
# 추후 Kubernetes 서비스 어카운트(demo-dynamic-app)에 이 정책을 연결해서 자격증명 요청 가능
vault policy write demo-auth-policy-db - <<EOF
path "demo-db/creds/dev-postgres" {
   capabilities = ["read"]
}
EOF

 

Step7. Transit Secret Engine 설정 + VSO 연동 Role 구성

kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/sh
----------------------------------------------------------------
# Transit Secret Engine 활성화
# transit 엔진을 demo-transit 경로로 활성화.
# 데이터를 저장하지 않고 암복호화 기능만 제공하는 Vault의 기능
# 클라이언트 캐시는 리더십 변경 시에도 Vault 토큰 및 동적 비밀 임대를 계속 추적하고 갱신할 수 있으므로 원활한 업그레이드를 지원합니다
## Vault 서버에 클라이언트 캐시를 저장하고 암호화할 수 있습니다.
## Vault 동적 비밀을 사용하는 경우 클라이언트 캐시를 영구적으로 저장하고 암호화하는 것이 좋습니다.
## 이렇게 하면 재시작 및 업그레이드를 통해 동적 비밀 임대가 유지됩니다.
vault secrets enable -path=demo-transit transit
vault secrets list -detailed

# vso-client-cache라는 키를 생성
# 이 키는 VSO가 암복호화 시 사용할 암호화 키 역할
vault write -force demo-transit/keys/vso-client-cache

# vso-client-cache 키에 대해 암호화(encrypt), 복호화(decrypt)를 허용하는 정책 생성
vault policy write demo-auth-policy-operator - <<EOF
path "demo-transit/encrypt/vso-client-cache" {
   capabilities = ["create", "update"]
}
path "demo-transit/decrypt/vso-client-cache" {
   capabilities = ["create", "update"]
}
EOF

# Vault Secrets Operator가 사용하는 ServiceAccount에 위 정책을 바인딩
# vso가 Vault에 로그인할 때 사용할 수 있는 JWT 기반 Role 설정
# 해당 Role을 통해 Operator는 Transit 엔진을 이용한 암복호화 API 호출 가능
vault write auth/k8s-auth-mount/role/auth-role-operator \
   bound_service_account_names=vault-secrets-operator-controller-manager \
   bound_service_account_namespaces=vault-secrets-operator-system \
   token_ttl=0 \
   token_period=120 \
   token_policies=demo-auth-policy-operator \
   audience=vault
   
vault read auth/k8s-auth-mount/role/auth-role-operator

/ $ vault read auth/k8s-auth-mount/role/auth-role-operator
Key                                         Value
---                                         -----
alias_name_source                           serviceaccount_uid
audience                                    vault
bound_service_account_names                 [vault-secrets-operator-controller-manager]
bound_service_account_namespace_selector    n/a
bound_service_account_namespaces            [vault-secrets-operator-system]
token_bound_cidrs                           []
token_explicit_max_ttl                      0s
token_max_ttl                               0s
token_no_default_policy                     false
token_num_uses                              0
token_period                                2m
token_policies                              [demo-auth-policy-operator]
token_ttl                                   0s
token_type                                  default

 

 

Step8. 샘플 애플리케이션 YAML 작성 및 배포

kubectl create ns demo-ns

kubectl apply -f demo.yaml

---
# vault-auth-dynamic.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: demo-ns
  name: demo-dynamic-app
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: dynamic-auth
  namespace: demo-ns
spec:
  method: kubernetes
  mount: k8s-auth-mount
  kubernetes:
    role: auth-role
    serviceAccount: demo-dynamic-app
    audiences:
      - vault
---
# app-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: vso-db-demo
  namespace: demo-ns
---
# vault-dynamic-secret.yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: vso-db-demo
  namespace: demo-ns
spec:
  refreshAfter: 25s
  mount: demo-db
  path: creds/dev-postgres
  destination:
    name: vso-db-demo
    create: true
    overwrite: true
  vaultAuthRef: dynamic-auth
  rolloutRestartTargets:
  - kind: Deployment
    name: vaultdemo
---
# app-spring-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vaultdemo
  namespace: demo-ns
  labels:
    app: vaultdemo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vaultdemo
  template:
    metadata:
      labels:
        app: vaultdemo
    spec:
      volumes:
        - name: secrets
          secret:
            secretName: "vso-db-demo"
      containers:
        - name: vaultdemo
          image: hyungwookhub/vso-spring-demo:v5
          imagePullPolicy: IfNotPresent
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: "vso-db-demo"
                  key: password
            - name: DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: "vso-db-demo"
                  key: username
            - name: DB_HOST
              value: "postgres-postgresql.postgres.svc.cluster.local"
            - name: DB_PORT
              value: "5432"
            - name: DB_NAME
              value: "postgres"
          ports:
            - containerPort: 8088
          volumeMounts:
            - name: secrets
              mountPath: /etc/secrets
              readOnly: true
---
apiVersion: v1
kind: Service
metadata:
  name: vaultdemo
  namespace: demo-ns
spec:
  ports:
    - name: vaultdemo
      port: 8088         
      targetPort: 8088 
      nodePort: 30003
  selector:
    app: vaultdemo
  type: NodePort

 

작동확인

동작확인(CLI)

# 스크릿이 변경되고, 적용을 위해서 파드가 42~45초 사이로 재생성됨
while true; do kubectl get pod -n demo-ns ; echo; kubectl view-secret -n demo-ns vso-db-demo --all; date; sleep 30 ; echo ; done 
NAME                         READY   STATUS    RESTARTS   AGE
vaultdemo-7779bcc8c6-tn74g   1/1     Running   0          27s

_raw='{"password":"igZwq-mC04usPQ7QOVDM","username":"v-k8s-auth-dev-post-5mYscZjhGgMhqcWIRpjo-1744273869"}'
password='igZwq-mC04usPQ7QOVDM'
username='v-k8s-auth-dev-post-5mYscZjhGgMhqcWIRpjo-1744273869'
Thu Apr 10 17:31:36 KST 2025


# 실제 postgresql 에 사용자 정보 확인 : 계속 추가되고 있음..
kubectl exec -it -n postgres postgres-postgresql-0 -- sh -c "PGPASSWORD=secret-pass psql -U postgres -h localhost -c '\du'"


# 로그 확인
kubectl stern -n demo-ns -l app=vaultdemo
...

kubectl stern -n vault vault-0
...
vault-0 vault 2025-04-10T08:32:09.484Z [INFO]  expiration: revoked lease: lease_id=demo-db/creds/dev-postgres/Ph1sg8efnqssOU5FVIuAqhMW
vault-0 vault 2025-04-10T08:32:54.229Z [INFO]  expiration: revoked lease: lease_id=demo-db/creds/dev-postgres/XKkHZ65ZYLeILvo8GqWfE7F3
vault-0 vault 2025-04-10T08:33:36.804Z [INFO]  expiration: revoked lease: lease_id=demo-db/creds/dev-postgres/rk1KUFsV7SjZPNFsCqCHGZVx
...


# vault-secrets-operator 가 clientCachePersistenceModel 설정 적용 정보 확인 : 참고로 해당 파드가 재시작되지는 않음.
kubectl stern -n vault-secrets-operator-system -l app.kubernetes.io/name=vault-secrets-operator
...

# 암호화된 캐시 저장소 확인 : vso-cc-<auth method>
kubectl get secret -n vault-secrets-operator-system -l app.kubernetes.io/component=client-cache-storage 
NAME                                       TYPE     DATA   AGE
vso-cc-kubernetes-5530fb1481fb1695773196   Opaque   2      5m21s

kubectl get secret -n vault-secrets-operator-system -l app.kubernetes.io/component=client-cache-storage -o yaml

 

'카테고리 없음'의 다른글

  • 현재글[AEWS] 10주차 - K8s 시크릿 관리 Update(With Vault)

티스토리툴바