What is Vault?
HashiCorp Vault vs Key Management Service
HashiCorp Vault
HashiCorp Vault is designed to be a full secrets managements solution. It is capable of storing encrypted data, generating dynamic credentials like database passwords or IAM credentials, and encrypted data as a pass-through. It aims to be a single source for secrets in a cloud-agnostic way. HashiCorp Vault runs as a service. As a tradeoff for these more advanced features, you’ll need to run Vault yourself, manage HA, setup on-call, etc. You pay a flat rate (VM core hours) regardless of the number of secrets or requests.
We’re using vault because we’re running stuff outside of the AWS eco-system on bare VMs and bare servers.
Key Management Service
Google Cloud KMS (and other KMS services) on the other hand, is a managed encryption service. You post plaintext data, KMS encrypts it (managing the keys and rotation), and returns the ciphertext to you. It’s your responsibility to store that data. When you want the plaintext back, you post the ciphertext to KMS and it decrypts and returns the plaintext. It’s worth noting that Google Cloud KMS is globally distributed and offered as a managed service. You pay per request and don’t have to manage any services.
KMS is not a secret store. It encrypts data, but it’s your job to store the encrypted result. KMS manages the encryption keys, not the underlying data.
AWS KMS is pretty darn expensive. Of course, that does not take into account the cost of getting the vault server up and maintaining it.
HashiCorp Vault vs k8s Secret
k8s Secret only base64-encoded the data without REAL encryption.
You can use third-party Secrets store providers to keep your confidential data outside your cluster and then configure Pods to access that information.
Configure access to external Secrets providers:
Prerequisites
Install Minikube
1
|
sudo pacman -S minikube libvirt qemu-desktop dnsmasq iptables-nft
|
1
2
3
4
|
sudo usermod -aG libvirt $(whoami)
sudo systemctl start libvirtd.service
sudo systemctl enable libvirtd.service
sudo systemctl status libvirtd.service
|
Install kubectl CLI
Install helm CLI
Start Minikube
set the kvm2 provider in minikube:
1
|
docker context use default
|
1
|
minikube config set driver kvm2
|
1
2
|
export http_proxy=socks5://127.0.0.1:20170
export https_proxy=socks5://127.0.0.1:20170
|
1
|
minikube start --profile tutorial-cluster --memory=16384 --cpus=4 --kubernetes-version=v1.25.2 --docker-env http_proxy=$http_proxy --docker-env https_proxy=$https_proxy
|
Check that kubectl is configured to talk to your cluster, by running the kubectl version command.
Install the Vault Helm chart
Add the HashiCorp Helm repository.
1
|
helm repo add hashicorp https://helm.releases.hashicorp.com
|
1
|
helm search repo hashicorp/vault
|
Install the latest version of the Vault Helm chart with Integrated Storage.
Create a file named helm-vault-raft-values.yml with the following contents:
1
2
3
4
5
6
7
8
|
cat > helm-vault-raft-values.yml <<EOF
server:
affinity: ""
ha:
enabled: true
raft:
enabled: true
EOF
|
1
|
helm install vault hashicorp/vault --values helm-vault-raft-values.yml
|
Display all the pods within the default namespace, and wait until running status of all pods.
Initialize vault-0 with one key share and one key threshold.
1
2
3
4
|
kubectl exec vault-0 -- vault operator init \
-key-shares=1 \
-key-threshold=1 \
-format=json > cluster-keys.json
|
The operator init command generates a root key that it disassembles into key shares -key-shares=1 and then sets the number of key shares required to unseal Vault -key-threshold=1. These key shares are written to the output as unseal keys in JSON format -format=json. Here the output is redirected to a file named cluster-keys.json.
Do not run an unsealed Vault in production with a single key share and a single key threshold. This approach is only used here to simplify the unsealing process for this demonstration.
1
|
jq -r ".unseal_keys_b64[]" cluster-keys.json
|
Create a variable named VAULT_UNSEAL_KEY to capture the Vault unseal key.
1
|
VAULT_UNSEAL_KEY=$(jq -r ".unseal_keys_b64[]" cluster-keys.json)
|
Unseal Vault running on the vault-0 pod.
1
|
kubectl exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY
|
The operator unseal command reports that Vault is initialized and unsealed.
Join the vault-1 pod to the Raft cluster.
1
|
kubectl exec -ti vault-1 -- vault operator raft join http://vault-0.vault-internal:8200
|
Join the vault-2 pod to the Raft cluster.
1
|
kubectl exec -ti vault-2 -- vault operator raft join http://vault-0.vault-internal:8200
|
Use the unseal key from above to unseal vault-1.
1
|
kubectl exec -ti vault-1 -- vault operator unseal $VAULT_UNSEAL_KEY
|
1
|
kubectl exec -ti vault-2 -- vault operator unseal $VAULT_UNSEAL_KEY
|
Set a secret in Vault
Vault generated an initial root token when it was initialized.
Display the root token found in cluster-keys.json.
1
|
jq -r ".root_token" cluster-keys.json
|
Start an interactive shell session on the vault-0 pod.
1
|
kubectl exec --stdin=true --tty=true vault-0 -- /bin/sh
|
Login with the root token when prompted.
Enable an instance of the kv-v2 secrets engine at the path secret.
1
|
vault secrets enable -path=secret kv-v2
|
Create a secret at path secret/webapp/config with a username and password.
1
|
vault kv put secret/webapp/config username="static-user" password="static-password"
|
Verify that the secret is defined at the path secret/webapp/config.
1
|
vault kv get secret/webapp/config
|
You successfully created the secret for the web application.
Exit the vault-0 pod.
Start an interactive shell session on the vault-0 pod.
1
|
kubectl exec --stdin=true --tty=true vault-0 -- /bin/sh
|
Enable the Kubernetes authentication method.
1
|
vault auth enable kubernetes
|
Configure the Kubernetes authentication method to use the location of the Kubernetes API.
1
2
|
vault write auth/kubernetes/config \
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
|
Write out the policy named webapp that enables the read capability for secrets at path secret/data/webapp/config.
1
2
3
4
5
|
vault policy write webapp - <<EOF
path "secret/data/webapp/config" {
capabilities = ["read"]
}
EOF
|
Create a Kubernetes authentication role, named webapp, that connects the Kubernetes service account name and webapp policy.
1
2
3
4
5
|
vault write auth/kubernetes/role/webapp \
bound_service_account_names=vault \
bound_service_account_namespaces=default \
policies=webapp \
ttl=24h
|
The role connects the Kubernetes service account, vault, and namespace, default, with the Vault policy, webapp. The tokens returned after authentication are valid for 24 hours.
Exit the vault-0 pod.
Created a web application
Create a simple web application
The example web application performs the single function of listening for HTTP requests. During a request it reads the Kubernetes service token, logs into Vault, and then requests the secret.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
|
package main
import (
"encoding/json"
"fmt"
"log"
"net/http"
"os"
)
func main() {
vaultToken := "root"
port := os.Getenv("SERVICE_PORT")
if port == "" {
port = "8080"
log.Println("PORT environment variable not set, defaulting to", port)
}
vaultUrl := os.Getenv("VAULT_ADDR")
if vaultUrl == "" {
vaultUrl = "http://vault:8200"
}
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
log.Println("Received Request - Port forwarding is working.")
// If the JWT path is set up then get the new token from Vault using the k8s Auth
jwtPath := os.Getenv("JWT_PATH")
if jwtPath != "" {
jwtFile, err := os.ReadFile(jwtPath)
if err != nil {
fmt.Println("Error reading JWT file at", jwtPath, ": ", err)
return
}
jwt := string(jwtFile)
fmt.Println("Read JWT:", jwt)
authPath := "auth/kubernetes/login"
// Create the payload for Vault authentication
pl := VaultJWTPayload { Role: "webapp", JWT: jwt }
jwtPayload, err := json.Marshal(pl)
if err != nil {
fmt.Println("Error encoding Vault request JSON:", err)
return
}
// Send a request to Vault to retrieve a token
vaultLoginResponse := &VaultLoginResponse{}
err = SendRequest(vaultUrl + "/v1/" + authPath, "", "POST", jwtPayload, vaultLoginResponse)
if err != nil {
fmt.Println("Error getting response from Vault k8s login:", err)
return
}
vaultToken = vaultLoginResponse.Auth.ClientToken
fmt.Println("Retrieved token: ", vaultToken)
}
secretsPath := "secret/data/webapp/config"
// Send a request to Vault using the token to retrieve the secret
vaultSecretResponse := &VaultSecretResponse{}
err := SendRequest(vaultUrl + "/v1/" + secretsPath, vaultToken, "GET", nil, &vaultSecretResponse)
if err != nil {
fmt.Println("Error getting secret from Vault:", err)
return
}
secretResponseData, ok := vaultSecretResponse.Data.Data.(map[string]interface{})
if ok {
for key, value := range secretResponseData {
fmt.Fprintf(w, "%s:%s ", key, value)
}
} else {
fmt.Println("Error getting the secret from Vault, cannot convert Data to map[string]interface{}")
}
})
log.Println("Listening on port", port)
if err := http.ListenAndServe(":" + port, nil); err != nil {
log.Fatalf("Failed to start server: %s", err)
}
}
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
|
package main
import (
"bytes"
"encoding/json"
"fmt"
"net/http"
"time"
)
func SendRequest(url string, token string, requestType string, payload []byte, target interface{}) error {
req, err := http.NewRequest(requestType, url, bytes.NewBuffer(payload))
if err != nil {
fmt.Println("Error creating request:", err)
return err
}
req.Header.Set("Content-Type", "application/json")
if token != "" {
req.Header.Set("X-Vault-Token", token)
}
client := &http.Client{Timeout: 10 * time.Second}
res, err := client.Do(req)
if err != nil {
fmt.Println("Error sending request to Vault:", err)
return err
}
defer res.Body.Close()
return json.NewDecoder(res.Body).Decode(target)
}
|
1
2
3
4
5
6
|
package main
type VaultJWTPayload struct {
Role string `json:"role"`
JWT string `json:"jwt"`
}
|
1
|
vim vault_login_response.go
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
|
package main
type VaultLoginResponse struct {
RequestID string `json:"request_id"`
LeaseID string `json:"lease_id"`
Renewable bool `json:"renewable"`
LeaseDuration int `json:"lease_duration"`
Data interface{} `json:"data"`
WrapInfo interface{} `json:"wrap_info"`
Warnings interface{} `json:"warnings"`
Auth struct {
ClientToken string `json:"client_token"`
Accessor string `json:"accessor"`
Policies []string `json:"policies"`
TokenPolicies []string `json:"token_policies"`
Metadata struct {
Role string `json:"role"`
ServiceAccountName string `json:"service_account_name"`
ServiceAccountNamespace string `json:"service_account_namespace"`
ServiceAccountSecretName string `json:"service_account_secret_name"`
ServiceAccountUID string `json:"service_account_uid"`
} `json:"metadata"`
LeaseDuration int `json:"lease_duration"`
Renewable bool `json:"renewable"`
EntityID string `json:"entity_id"`
TokenType string `json:"token_type"`
Orphan bool `json:"orphan"`
MfaRequirement interface{} `json:"mfa_requirement"`
NumUses int `json:"num_uses"`
} `json:"auth"`
}
|
1
|
vim vault_secret_response.go
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
|
package main
type VaultSecretResponse struct {
RequestID string `json:"request_id"`
LeaseID string `json:"lease_id"`
Renewable bool `json:"renewable"`
LeaseDuration int `json:"lease_duration"`
Data struct {
Data interface{} `json:"data"`
} `json:"data"`
Warnings interface{} `json:"warnings"`
Auth interface{} `json:"auth"`
}package main
type VaultSecretResponse struct {
RequestID string `json:"request_id"`
LeaseID string `json:"lease_id"`
Renewable bool `json:"renewable"`
LeaseDuration int `json:"lease_duration"`
Data struct {
Data interface{} `json:"data"`
} `json:"data"`
Warnings interface{} `json:"warnings"`
Auth interface{} `json:"auth"`
}
|
Create a Dockerfile for the application
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
# Use an official Golang image as the base image
FROM golang:latest
# Set the working directory in the container to /app
WORKDIR /app
# Copy the current directory contents into the container at /app
COPY . .
# Build the Go app
RUN go build -o main .
# Expose port 8080 to the host
EXPOSE 8080
# Specify the command to run the app
CMD ["./main"]
|
Build Docker image
You need to use eval $(minikube docker-env) in your current terminal window. This will use the minikube docker-env for the current session.
1
|
eval $(minikube docker-env)
|
The image needs to be on the minikube virtual machine. So now you need to build your image again.
1
|
docker build --tag simple-vault-client:latest .
|
Then, use imagePullPolicy: Never in the manifest file to use the local image registry.
Launch a web application
deployment-01-webapp.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
|
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: webapp
labels:
app: webapp
spec:
replicas: 1
selector:
matchLabels:
app: webapp
template:
metadata:
labels:
app: webapp
spec:
serviceAccountName: vault
containers:
- name: app
image: simple-vault-client:latest
imagePullPolicy: Never
env:
- name: VAULT_ADDR
value: 'http://vault:8200'
- name: JWT_PATH
value: '/var/run/secrets/kubernetes.io/serviceaccount/token'
- name: SERVICE_PORT
value: '8080'
|
- JWT_PATH sets the path of the JSON web token (JWT) issued by Kubernetes. This token is used by the web application to authenticate with Vault.
- VAULT_ADDR sets the address of the Vault service. The Helm chart defined a Kubernetes service named
vault that forwards requests to its endpoints (i.e. The pods named vault-0, vault-1, and vault-2).
- SERVICE_PORT sets the port that the service listens for incoming HTTP requests.
Deploy the webapp in Kubernetes by applying the file deployment-01-webapp.yml.
1
|
kubectl apply --filename deployment-01-webapp.yml
|
Get all the pods within the default namespace, Wait till all pods are in Running status.
In another terminal, port forward all requests made to http://localhost:8080 to the webapp pod on port 8080.
1
2
3
|
kubectl port-forward \
$(kubectl get pod -l app=webapp -o jsonpath="{.items[0].metadata.name}") \
8080:8080
|
In the original terminal, perform a curl request at http://localhost:8080.
1
|
curl http://localhost:8080
|
Clean up
First, stop the running local Kubernetes cluster.
This deactivates minikube, and all pods still exist at this point.
Delete the local Kubernetes cluster. Be aware that minikube delete removes the minikube deployment including all pods.
Reference
Vault installation to minikube via Helm with Integrated Storage