Menyajikan LLM seperti DeepSeek-R1 671B atau Llama 3.1 405B di GKE

Membuat cluster GKE

Anda dapat menyajikan model menggunakan vLLM di beberapa node GPU dalam cluster GKE Autopilot atau Standard. Sebaiknya gunakan cluster Autopilot untuk pengalaman Kubernetes yang terkelola sepenuhnya. Untuk memilih mode operasi GKE yang paling sesuai untuk workload Anda, lihat Memilih mode operasi GKE.

  gcloud container clusters create-auto ${CLUSTER_NAME} \     --project=${PROJECT_ID} \     --location=${REGION} \     --cluster-version=${CLUSTER_VERSION} 

Konfigurasi kubectl untuk berkomunikasi dengan cluster Anda

Konfigurasi kubectl untuk berkomunikasi dengan cluster Anda menggunakan perintah berikut:

gcloud container clusters get-credentials CLUSTER_NAME --location=REGION 

Buat Secret Kubernetes untuk kredensial Hugging Face

Buat Secret Kubernetes yang berisi token Hugging Face menggunakan perintah berikut:

kubectl create secret generic hf-secret \   --from-literal=hf_api_token=${HF_TOKEN} \   --dry-run=client -o yaml | kubectl apply -f - 

Instal LeaderWorkerSet

Untuk menginstal LWS, jalankan perintah berikut:

kubectl apply --server-side -f https://github.com/kubernetes-sigs/lws/releases/latest/download/manifests.yaml 

Validasi bahwa pengontrol LeaderWorkerSet berjalan di namespace lws-system menggunakan perintah berikut:

kubectl get pod -n lws-system 

Outputnya mirip dengan hal berikut ini:

NAME                                     READY   STATUS    RESTARTS   AGE lws-controller-manager-546585777-crkpt   1/1     Running   0          4d21h lws-controller-manager-546585777-zbt2l   1/1     Running   0          4d21h 

Men-deploy Server Model vLLM

Untuk men-deploy server model vLLM, ikuti langkah-langkah berikut:

1. Terapkan manifes, bergantung pada LLM yang ingin Anda deploy.

   DeepSeek-R1

   1. Periksa manifes vllm-deepseek-r1-A3.yaml.

       apiVersion: leaderworkerset.x-k8s.io/v1 kind: LeaderWorkerSet metadata:   name: vllm spec:   replicas: 1   leaderWorkerTemplate:     size: 2     restartPolicy: RecreateGroupOnPodRestart     leaderTemplate:       metadata:         labels:           role: leader       spec:         nodeSelector:           cloud.google.com/gke-accelerator: nvidia-h100-80gb         containers:           - name: vllm-leader             image: vllm/vllm-openai:v0.8.5             env:               - name: HUGGING_FACE_HUB_TOKEN                 valueFrom:                   secretKeyRef:                     name: hf-secret                     key: hf_api_token             command:               - sh               - -c               - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh leader --ray_cluster_size=$(LWS_GROUP_SIZE);                 python3 -m vllm.entrypoints.openai.api_server --port 8080 --model deepseek-ai/DeepSeek-R1 --tensor-parallel-size 8 --pipeline-parallel-size 2 --trust-remote-code --max-model-len 4096"             resources:               limits:                 nvidia.com/gpu: "8"             ports:               - containerPort: 8080             readinessProbe:               tcpSocket:                 port: 8080               initialDelaySeconds: 15               periodSeconds: 10             volumeMounts:               - mountPath: /dev/shm                 name: dshm         volumes:         - name: dshm           emptyDir:             medium: Memory             sizeLimit: 15Gi     workerTemplate:       spec:         nodeSelector:           cloud.google.com/gke-accelerator: nvidia-h100-80gb         containers:           - name: vllm-worker             image: vllm/vllm-openai:v0.8.5             command:               - sh               - -c               - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh worker --ray_address=$(LWS_LEADER_ADDRESS)"             resources:               limits:                 nvidia.com/gpu: "8"             env:               - name: HUGGING_FACE_HUB_TOKEN                 valueFrom:                   secretKeyRef:                     name: hf-secret                     key: hf_api_token             volumeMounts:               - mountPath: /dev/shm                 name: dshm            volumes:         - name: dshm           emptyDir:             medium: Memory             sizeLimit: 15Gi --- apiVersion: v1 kind: Service metadata:   name: vllm-leader spec:   ports:     - name: http       port: 8080       protocol: TCP       targetPort: 8080   selector:     leaderworkerset.sigs.k8s.io/name: vllm     role: leader   type: ClusterIP 

   2. Terapkan manifes dengan menjalankan perintah berikut:

      kubectl apply -f vllm-deepseek-r1-A3.yaml 

   Llama 3.1 405B

   1. Periksa manifes vllm-llama3-405b-A3.yaml.

       apiVersion: leaderworkerset.x-k8s.io/v1 kind: LeaderWorkerSet metadata:   name: vllm spec:   replicas: 1   leaderWorkerTemplate:     size: 2     restartPolicy: RecreateGroupOnPodRestart     leaderTemplate:       metadata:         labels:           role: leader       spec:         nodeSelector:           cloud.google.com/gke-accelerator: nvidia-h100-80gb         containers:           - name: vllm-leader             image: vllm/vllm-openai:v0.8.5             env:               - name: HUGGING_FACE_HUB_TOKEN                 valueFrom:                   secretKeyRef:                     name: hf-secret                     key: hf_api_token             command:               - sh               - -c               - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh leader --ray_cluster_size=$(LWS_GROUP_SIZE);                 python3 -m vllm.entrypoints.openai.api_server --port 8080 --model meta-llama/Meta-Llama-3.1-405B-Instruct --tensor-parallel-size 8 --pipeline-parallel-size 2"             resources:               limits:                 nvidia.com/gpu: "8"             ports:               - containerPort: 8080             readinessProbe:               tcpSocket:                 port: 8080               initialDelaySeconds: 15               periodSeconds: 10             volumeMounts:               - mountPath: /dev/shm                 name: dshm         volumes:         - name: dshm           emptyDir:             medium: Memory             sizeLimit: 15Gi     workerTemplate:       spec:         nodeSelector:           cloud.google.com/gke-accelerator: nvidia-h100-80gb         containers:           - name: vllm-worker             image: vllm/vllm-openai:v0.8.5             command:               - sh               - -c               - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh worker --ray_address=$(LWS_LEADER_ADDRESS)"             resources:               limits:                 nvidia.com/gpu: "8"             env:               - name: HUGGING_FACE_HUB_TOKEN                 valueFrom:                   secretKeyRef:                     name: hf-secret                     key: hf_api_token             volumeMounts:               - mountPath: /dev/shm                 name: dshm            volumes:         - name: dshm           emptyDir:             medium: Memory             sizeLimit: 15Gi --- apiVersion: v1 kind: Service metadata:   name: vllm-leader spec:   ports:     - name: http       port: 8080       protocol: TCP       targetPort: 8080   selector:     leaderworkerset.sigs.k8s.io/name: vllm     role: leader   type: ClusterIP 

   2. Terapkan manifes dengan menjalankan perintah berikut:

      kubectl apply -f vllm-llama3-405b-A3.yaml 

2. Tunggu hingga checkpoint model selesai didownload. Penyelesaian operasi ini mungkin memerlukan waktu beberapa menit.

3. Lihat log dari server model yang sedang berjalan dengan perintah berikut:

   kubectl logs vllm-0 -c vllm-leader 

   Output-nya akan terlihat seperti berikut:

   INFO 08-09 21:01:34 api_server.py:297] Route: /detokenize, Methods: POST INFO 08-09 21:01:34 api_server.py:297] Route: /v1/models, Methods: GET INFO 08-09 21:01:34 api_server.py:297] Route: /version, Methods: GET INFO 08-09 21:01:34 api_server.py:297] Route: /v1/chat/completions, Methods: POST INFO 08-09 21:01:34 api_server.py:297] Route: /v1/completions, Methods: POST INFO 08-09 21:01:34 api_server.py:297] Route: /v1/embeddings, Methods: POST INFO:     Started server process [7428] INFO:     Waiting for application startup. INFO:     Application startup complete. INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit) 

Menyajikan model

Siapkan penerusan port ke model dengan menjalankan perintah berikut:

kubectl port-forward svc/vllm-leader 8080:8080 

Berinteraksi dengan model menggunakan curl

Untuk berinteraksi dengan model menggunakan curl, ikuti petunjuk berikut:

curl http://localhost:8080/v1/completions \ -H "Content-Type: application/json" \ -d '{     "model": "deepseek-ai/DeepSeek-R1",     "prompt": "I have four boxes. I put the red box on the bottom and put the blue box on top. Then I put the yellow box on top the blue. Then I take the blue box out and put it on top. And finally I put the green box on the top. Give me the final order of the boxes from bottom to top. Show your reasoning but be brief",     "max_tokens": 1024,     "temperature": 0 }' 

{ "id": "cmpl-f2222b5589d947419f59f6e9fe24c5bd", "object": "text_completion", "created": 1738269669, "model": "deepseek-ai/DeepSeek-R1", "choices": [   {     "index": 0,     "text": ".\n\nOkay, let's see. The user has four boxes and is moving them around. Let me try to visualize each step. \n\nFirst, the red box is placed on the bottom. So the stack starts with red. Then the blue box is put on top of red. Now the order is red (bottom), blue. Next, the yellow box is added on top of blue. So now it's red, blue, yellow. \n\nThen the user takes the blue box out. Wait, blue is in the middle. If they remove blue, the stack would be red and yellow. But where do they put the blue box? The instruction says to put it on top. So after removing blue, the stack is red, yellow. Then blue is placed on top, making it red, yellow, blue. \n\nFinally, the green box is added on the top. So the final order should be red (bottom), yellow, blue, green. Let me double-check each step to make sure I didn't mix up any steps. Starting with red, then blue, then yellow. Remove blue from the middle, so yellow is now on top of red. Then place blue on top of that, so red, yellow, blue. Then green on top. Yes, that seems right. The key step is removing the blue box from the middle, which leaves yellow on red, then blue goes back on top, followed by green. So the final order from bottom to top is red, yellow, blue, green.\n\n**Final Answer**\nThe final order from bottom to top is \\boxed{red}, \\boxed{yellow}, \\boxed{blue}, \\boxed{green}.\n</think>\n\n1. Start with the red box at the bottom.\n2. Place the blue box on top of the red box. Order: red (bottom), blue.\n3. Place the yellow box on top of the blue box. Order: red, blue, yellow.\n4. Remove the blue box (from the middle) and place it on top. Order: red, yellow, blue.\n5. Place the green box on top. Final order: red, yellow, blue, green.\n\n\\boxed{red}, \\boxed{yellow}, \\boxed{blue}, \\boxed{green}",     "logprobs": null,     "finish_reason": "stop",     "stop_reason": null,     "prompt_logprobs": null   } ], "usage": {   "prompt_tokens": 76,   "total_tokens": 544,   "completion_tokens": 468,   "prompt_tokens_details": null } } 

curl http://localhost:8080/v1/completions \ -H "Content-Type: application/json" \ -d '{     "model": "meta-llama/Meta-Llama-3.1-405B-Instruct",     "prompt": "San Francisco is a",     "max_tokens": 7,     "temperature": 0 }' 

{"id":"cmpl-0a2310f30ac3454aa7f2c5bb6a292e6c", "object":"text_completion","created":1723238375,"model":"meta-llama/Llama-3.1-405B-Instruct","choices":[{"index":0,"text":" top destination for foodies, with","logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":5,"total_tokens":12,"completion_tokens":7}} 

Menyiapkan autoscaler kustom

Di bagian ini, Anda akan menyiapkan penskalaan otomatis Pod horizontal untuk menggunakan metrik Prometheus kustom. Anda menggunakan metrik Google Cloud Managed Service for Prometheus dari server vLLM.

Untuk mempelajari lebih lanjut, lihat Google Cloud Managed Service for Prometheus. Fitur ini harus diaktifkan secara default di cluster GKE.

1. Siapkan Adaptor Stackdriver Metrik Kustom di cluster Anda:

   kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml 

2. Tambahkan peran Monitoring Viewer ke akun layanan yang digunakan oleh Custom Metrics Stackdriver Adapter:

   gcloud projects add-iam-policy-binding projects/PROJECT_ID \     --role roles/monitoring.viewer \     --member=principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/custom-metrics/sa/custom-metrics-stackdriver-adapter 

3. Simpan manifes berikut sebagai vllm_pod_monitor.yaml:

    apiVersion: monitoring.googleapis.com/v1 kind: PodMonitoring metadata:  name: vllm-pod-monitoring spec:  selector:    matchLabels:     leaderworkerset.sigs.k8s.io/name: vllm     role: leader  endpoints:  - path: /metrics    port: 8080    interval: 15s 

4. Terapkan manifes ke cluster:

   kubectl apply -f vllm_pod_monitor.yaml 

Membuat beban pada endpoint vLLM

Buat beban di server vLLM untuk menguji cara GKE melakukan penskalaan otomatis dengan metrik vLLM kustom.

1. Siapkan penerusan port ke model:

   kubectl port-forward svc/vllm-leader 8080:8080 

2. Jalankan skrip bash (load.sh) untuk mengirimkan N jumlah permintaan paralel ke endpoint vLLM:

   #!/bin/bash # Set the number of parallel processes to run. N=PARALLEL_PROCESSES # Get the external IP address of the vLLM load balancer service. export vllm_service=$(kubectl get service vllm-service -o jsonpath='{.status.loadBalancer.ingress[0].ip}') # Loop from 1 to N to start the parallel processes. for i in $(seq 1 $N); do   # Start an infinite loop to continuously send requests.   while true; do     # Use curl to send a completion request to the vLLM service.     curl http://$vllm_service:8000/v1/completions -H "Content-Type: application/json" -d '{"model": "meta-llama/Llama-3.1-70B", "prompt": "Write a story about san francisco", "max_tokens": 100, "temperature": 0}'   done &  # Run in the background done # Keep the script running until it is manually stopped. wait 

   Ganti PARALLEL_PROCESSES dengan jumlah proses paralel yang ingin Anda jalankan.

3. Jalankan skrip bash:

   nohup ./load.sh & 

Pastikan bahwa Google Cloud Managed Service for Prometheus menyerap metrik

Setelah Google Cloud Managed Service for Prometheus meng-scrape metrik dan Anda menambahkan beban ke endpoint vLLM, Anda dapat melihat metrik di Cloud Monitoring.

1. Di konsol Google Cloud , buka halaman Metrics explorer.

   Buka Metrics explorer

2. Klik < > PromQL.

3. Masukkan kueri berikut untuk mengamati metrik traffic:

   vllm:gpu_cache_usage_perc{cluster='CLUSTER_NAME'} 

Gambar berikut adalah contoh grafik setelah eksekusi skrip pemuatan. Grafik ini menunjukkan bahwa Google Cloud Managed Service for Prometheus menyerap metrik traffic sebagai respons terhadap beban yang ditambahkan ke endpoint vLLM:

Men-deploy konfigurasi Horizontal Pod Autoscaler

Saat Anda memutuskan metrik mana yang akan digunakan untuk penskalaan otomatis, sebaiknya gunakan metrik berikut untuk vLLM:

• num_requests_waiting: metrik ini terkait dengan jumlah permintaan yang menunggu dalam antrean server model. Jumlah ini mulai bertambah secara signifikan saat cache kv penuh.

• gpu_cache_usage_perc: metrik ini terkait dengan pemanfaatan cache KV, yang secara langsung berkorelasi dengan jumlah permintaan yang diproses untuk siklus inferensi tertentu di server model.

Sebaiknya gunakan num_requests_waiting saat Anda mengoptimalkan throughput dan biaya, serta saat target latensi dapat dicapai dengan throughput maksimum server model Anda.

Sebaiknya gunakan gpu_cache_usage_perc jika Anda memiliki workload yang sensitif terhadap latensi dan penskalaan berbasis antrean tidak cukup cepat untuk memenuhi persyaratan Anda.

Untuk penjelasan lebih lanjut, lihat Praktik terbaik untuk menskalakan otomatis workload inferensi model bahasa besar (LLM) dengan GPU.

Saat memilih target averageValue untuk konfigurasi HPA, Anda perlu menentukan metrik mana yang akan digunakan untuk penskalaan otomatis secara eksperimental. Untuk mendapatkan ide tambahan tentang cara mengoptimalkan eksperimen, lihat postingan blog Hemat GPU: Penskalaan otomatis yang lebih cerdas untuk workload inferensi GKE Anda. profile-generator yang digunakan dalam postingan blog ini juga berfungsi untuk vLLM.

Untuk men-deploy konfigurasi Horizontal Pod Autoscaler menggunakan num_requests_waiting, ikuti langkah-langkah berikut:

1. Simpan manifes berikut sebagai vllm-hpa.yaml:

    apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata:   name: lws-hpa spec:   minReplicas: 1   maxReplicas: 2   metrics:   - type: Pods     pods:       metric:         name: prometheus.googleapis.com|vllm:num_requests_waiting|gauge       target:         type: AverageValue         averageValue: 5   scaleTargetRef:     apiVersion: leaderworkerset.x-k8s.io/v1     kind: LeaderWorkerSet     name: vllm 

   Metrik vLLM di Google Cloud Managed Service for Prometheus mengikuti format vllm:metric_name.

   Praktik terbaik:

   Gunakan num_requests_waiting untuk menskalakan throughput. Gunakan gpu_cache_usage_perc untuk kasus penggunaan GPU yang sensitif terhadap latensi.

2. Deploy konfigurasi Horizontal Pod Autoscaler:

   kubectl apply -f vllm-hpa.yaml 

   GKE menjadwalkan Pod lain untuk di-deploy, yang memicu autoscaler node pool untuk menambahkan node kedua sebelum men-deploy replika vLLM kedua.

3. Lihat progres penskalaan otomatis Pod:

   kubectl get hpa --watch 

   Outputnya mirip dengan hal berikut ini:

   NAME      REFERENCE              TARGETS   MINPODS   MAXPODS   REPLICAS   AGE lws-hpa   LeaderWorkerSet/vllm   0/1       1         2         1          6d1h lws-hpa   LeaderWorkerSet/vllm   1/1       1         2         1          6d1h lws-hpa   LeaderWorkerSet/vllm   0/1       1         2         1          6d1h lws-hpa   LeaderWorkerSet/vllm   4/1       1         2         1          6d1h lws-hpa   LeaderWorkerSet/vllm   0/1       1         2         2          6d1h 

Mempercepat waktu pemuatan model dengan Google Cloud Hyperdisk ML

Dengan jenis LLM ini, vLLM dapat memerlukan waktu yang cukup lama untuk didownload, dimuat, dan di-warm up pada setiap replika baru. Misalnya, proses tersebut dapat memakan waktu sekitar 90 menit dengan Llama 3.1 405B. Anda dapat mengurangi waktu ini (menjadi 20 menit dengan Llama 3.1 405B) dengan mendownload model langsung ke volume Hyperdisk ML dan memasang volume tersebut ke setiap Pod. Untuk menyelesaikan operasi ini, tutorial ini menggunakan volume ML Hyperdisk dan Tugas Kubernetes. Pengontrol Tugas di Kubernetes membuat satu atau beberapa Pod dan memastikan bahwa Pod tersebut berhasil menjalankan tugas tertentu.

Untuk mempercepat waktu pemuatan model, lakukan langkah-langkah berikut:

1. Simpan contoh manifes berikut sebagai producer-pvc.yaml:

   kind: PersistentVolumeClaim apiVersion: v1 metadata:   name: producer-pvc spec:   # Specifies the StorageClass to use. Hyperdisk ML is optimized for ML workloads.   storageClassName: hyperdisk-ml   accessModes:   - ReadWriteOnce   resources:     requests:       storage: 800Gi 

2. Simpan contoh manifes berikut sebagai producer-job.yaml:

   DeepSeek-R1

    apiVersion: batch/v1 kind: Job metadata:   name: producer-job spec:   template:  # Template for the Pods the Job will create     spec:       affinity:         nodeAffinity:           requiredDuringSchedulingIgnoredDuringExecution:             nodeSelectorTerms:             - matchExpressions:               - key: cloud.google.com/machine-family                 operator: In                 values:                 - "c3"             - matchExpressions:               - key: topology.kubernetes.io/zone                 operator: In                 values:                 - "ZONE"       containers:       - name: copy         resources:           requests:             cpu: "32"           limits:             cpu: "32"         image: python:3.11-alpine         command:         - sh         - -c         - "pip install 'huggingface_hub==0.24.6' && \           huggingface-cli download deepseek-ai/DeepSeek-R1 --local-dir-use-symlinks=False --local-dir=/data/DeepSeek-R1 --include *.safetensors *.json *.py"         env:         - name: HUGGING_FACE_HUB_TOKEN           valueFrom:             secretKeyRef:               name: hf-secret               key: hf_api_token         volumeMounts:           - mountPath: "/data"             name: volume       restartPolicy: Never       volumes:         - name: volume           persistentVolumeClaim:             claimName: producer-pvc   parallelism: 1         # Run 1 Pods concurrently   completions: 1         # Once 1 Pods complete successfully, the Job is done   backoffLimit: 4        # Max retries on failure  

   Llama 3.1 405B

    apiVersion: batch/v1 kind: Job metadata:   name: producer-job spec:   template:  # Template for the Pods the Job will create     spec:       affinity:         nodeAffinity:           requiredDuringSchedulingIgnoredDuringExecution:             nodeSelectorTerms:             - matchExpressions:               - key: cloud.google.com/machine-family                 operator: In                 values:                 - "c3"             - matchExpressions:               - key: topology.kubernetes.io/zone                 operator: In                 values:                 - "ZONE"       containers:       - name: copy         resources:           requests:             cpu: "32"           limits:             cpu: "32"         image: python:3.11-alpine         command:         - sh         - -c         - "pip install 'huggingface_hub==0.24.6' && \           huggingface-cli download meta-llama/Meta-Llama-3.1-405B-Instruct --local-dir-use-symlinks=False --local-dir=/data/Meta-Llama-3.1-405B-Instruct --include *.safetensors *.json"         env:         - name: HUGGING_FACE_HUB_TOKEN           valueFrom:             secretKeyRef:               name: hf-secret               key: hf_api_token         volumeMounts:           - mountPath: "/data"             name: volume       restartPolicy: Never       volumes:         - name: volume           persistentVolumeClaim:             claimName: producer-pvc   parallelism: 1         # Run 1 Pods concurrently   completions: 1         # Once 1 Pods complete successfully, the Job is done   backoffLimit: 4        # Max retries on failure  

3. Ikuti petunjuk di Mempercepat pemuatan data AI/ML dengan Hyperdisk ML, menggunakan dua file yang Anda buat pada langkah sebelumnya.

   Setelah langkah ini, Anda telah membuat dan mengisi volume ML Hyperdisk dengan data model.

4. Deploy deployment server GPU multi-node vLLM, yang akan menggunakan volume Hyperdisk ML yang baru dibuat untuk data model.

   DeepSeek-R1

     apiVersion: leaderworkerset.x-k8s.io/v1 kind: LeaderWorkerSet metadata:   name: vllm spec:   replicas: 1   leaderWorkerTemplate:     size: 2     restartPolicy: RecreateGroupOnPodRestart     leaderTemplate:       metadata:         labels:           role: leader       spec:         containers:           - name: vllm-leader             image: vllm/vllm-openai:v0.8.5             env:               - name: HUGGING_FACE_HUB_TOKEN                 valueFrom:                   secretKeyRef:                     name: hf-secret                     key: hf_api_token             command:               - sh               - -c               - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh leader --ray_cluster_size=$(LWS_GROUP_SIZE);                 python3 -m vllm.entrypoints.openai.api_server --port 8080 --model /models/DeepSeek-R1 --tensor-parallel-size 8 --pipeline-parallel-size 2 --trust-remote-code --max-model-len 4096"             resources:               limits:                 nvidia.com/gpu: "8"             ports:               - containerPort: 8080             readinessProbe:               tcpSocket:                 port: 8080               initialDelaySeconds: 15               periodSeconds: 10             volumeMounts:               - mountPath: /dev/shm                 name: dshm               - mountPath: /models                 name: deepseek-r1         volumes:         - name: dshm           emptyDir:             medium: Memory         - name: deepseek-r1           persistentVolumeClaim:             claimName: hdml-static-pvc     workerTemplate:       spec:         containers:           - name: vllm-worker             image: vllm/vllm-openai:v0.8.5             command:               - sh               - -c               - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh worker --ray_address=$(LWS_LEADER_ADDRESS)"             resources:               limits:                 nvidia.com/gpu: "8"             env:               - name: HUGGING_FACE_HUB_TOKEN                 valueFrom:                   secretKeyRef:                     name: hf-secret                     key: hf_api_token             volumeMounts:               - mountPath: /dev/shm                 name: dshm               - mountPath: /models                 name: deepseek-r1         volumes:         - name: dshm           emptyDir:             medium: Memory         - name: deepseek-r1           persistentVolumeClaim:             claimName: hdml-static-pvc --- apiVersion: v1 kind: Service metadata:   name: vllm-leader spec:   ports:     - name: http       port: 8080       protocol: TCP       targetPort: 8080   selector:     leaderworkerset.sigs.k8s.io/name: vllm     role: leader   type: ClusterIP 

   Llama 3.1 405B

     apiVersion: leaderworkerset.x-k8s.io/v1 kind: LeaderWorkerSet metadata:   name: vllm spec:   replicas: 1   leaderWorkerTemplate:     size: 2     restartPolicy: RecreateGroupOnPodRestart     leaderTemplate:       metadata:         labels:           role: leader       spec:         containers:           - name: vllm-leader             image: vllm/vllm-openai:v0.8.5             env:               - name: HUGGING_FACE_HUB_TOKEN                 valueFrom:                   secretKeyRef:                     name: hf-secret                     key: hf_api_token             command:               - sh               - -c               - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh leader --ray_cluster_size=$(LWS_GROUP_SIZE);                 python3 -m vllm.entrypoints.openai.api_server --port 8080 --model /models/Meta-Llama-3.1-405B-Instruct --tensor-parallel-size 8 --pipeline-parallel-size 2"             resources:               limits:                 nvidia.com/gpu: "8"             ports:               - containerPort: 8080             readinessProbe:               tcpSocket:                 port: 8080               initialDelaySeconds: 15               periodSeconds: 10             volumeMounts:               - mountPath: /dev/shm                 name: dshm               - mountPath: /models                 name: llama3-405b         volumes:         - name: dshm           emptyDir:             medium: Memory         - name: llama3-405b           persistentVolumeClaim:             claimName: hdml-static-pvc     workerTemplate:       spec:         containers:           - name: vllm-worker             image: vllm/vllm-openai:v0.8.5             command:               - sh               - -c               - "bash /vllm-workspace/examples/online_serving/multi-node-serving.sh worker --ray_address=$(LWS_LEADER_ADDRESS)"             resources:               limits:                 nvidia.com/gpu: "8"             env:               - name: HUGGING_FACE_HUB_TOKEN                 valueFrom:                   secretKeyRef:                     name: hf-secret                     key: hf_api_token             volumeMounts:               - mountPath: /dev/shm                 name: dshm               - mountPath: /models                 name: llama3-405b         volumes:         - name: dshm           emptyDir:             medium: Memory         - name: llama3-405b           persistentVolumeClaim:             claimName: hdml-static-pvc --- apiVersion: v1 kind: Service metadata:   name: vllm-leader spec:   ports:     - name: http       port: 8080       protocol: TCP       targetPort: 8080   selector:     leaderworkerset.sigs.k8s.io/name: vllm     role: leader   type: ClusterIP