Procedure

1. Download the OpenShift Container Platform stable client from the mirror site.

2. Extract the tar file in your system with the following command:

   tar xvzf <file-name>

3. Place the oc binary in a directory on your $PATH.

4. Create an install-config.yml file with the following configurations. Replace BASE.DOMAIN, CLUSTER_NAME, PULL_SECRET, and SSL_PUBLIC_KEY with applicable values.

   additionalTrustBundlePolicy: Proxyonly
   apiVersion: v1
   baseDomain: BASE.DOMAIN
   compute:
   - architecture: amd64
     hyperthreading: Enabled
     name: worker
     platform:
       aws:
         type: m6i.2xlarge
     replicas: 3
   controlPlane:
     architecture: amd64
     hyperthreading: Enabled
     name: master
     platform:
       aws:
         type: m6i.2xlarge
     replicas: 3
   metadata:
     creationTimestamp: null
     name: CLUSTER_NAME
   networking:
     clusterNetwork:
     - cidr: 10.128.0.0/14
       hostPrefix: 23
     machineNetwork:
     - cidr: 10.0.0.0/16
     networkType: OVNKubernetes
     serviceNetwork:
     - 172.30.0.0/16
   platform:
     aws:
       region: REGION
   publish: External
   pullSecret: 'PULL_SECRET'
   sshKey: |
     SSH_PUBLIC_KEY

   + Run ./openshift-install create cluster from the oc binary path to install the OpenShift Container Platform cluster.

Procedure

1. Log in to the OCP cluster web interface with cluster-admin privileges.
2. Select Node Feature Discovery Operator on the Operators page.
3. In the NodeFeatureDiscovery tab, click Create a NodeFeatureDiscovery to create an instance with default values.
4. Select Nvidia GPU Operator on the Operators page.
5. In the ClusterPolicy tab, click Create a ClusterPolicy to create an instance with default values.

Procedure

1. Log in to the OpenShift Container Platform CLI and type the following command to get CLUSTER_NAME, CLUSTER_ID, AVALABILITY_ZONE, and REGION to list currently available machineset in your cluster:

   $ oc get machineset -n openshift-machine-api

2. Enter the following command to view values of a specific compute machine set custom resource (CR):

   $ oc get machineset <machineset_name> \
     -n openshift-machine-api -o yaml

3. Enter the values for CLUSTER_NAME, CLUSTER_ID, AVALABILITY_ZONE, REGION, instanceType, cluster-api/accelerator in the sample machineset YAML configuration.

4. Type the following command to create the machineset resource for nodes in your OpenShift Container Platform cluster.

   oc create -f machineset.yml

5. Enter the following command to get the status of the machineset, machine, and node CRs.

   watch 'oc get machineset -n openshift-machine-api && oc get machines -n openshift-machine-api && oc get nodes'

   Sample machineset yaml file

   cat << EOF > machineset.yml
   apiVersion: machine.openshift.io/v1beta1
   kind: MachineSet
   metadata:
     annotations:
       capacity.cluster-autoscaler.kubernetes.io/labels: kubernetes.io/arch=amd64
       machine.openshift.io/GPU: "8"
       machine.openshift.io/memoryMb: "786432"
       machine.openshift.io/vCPU: "192"
     labels:
       machine.openshift.io/cluster-api-cluster: CLUSTER_NAME-CLUSTER_ID
     name: CLUSTER_NAME-CLUSTER_ID-gpu-AVAILABILITY_ZONE
     namespace: openshift-machine-api
   spec:
     selector:
       matchLabels:
         machine.openshift.io/cluster-api-cluster: CLUSTER_NAME-CLUSTER_ID
         machine.openshift.io/cluster-api-machineset: CLUSTER_NAME-CLUSTER_ID-gpu-AVAILABILITY_ZONE
     template:
       metadata:
         labels:
           machine.openshift.io/cluster-api-cluster: CLUSTER_NAME-CLUSTER_ID
           machine.openshift.io/cluster-api-machine-role: gpu
           machine.openshift.io/cluster-api-machine-type: gpu
           machine.openshift.io/cluster-api-machineset: CLUSTER_NAME-CLUSTER_ID-gpu-AVAILABILITY_ZONE
       spec:
         replicas: 0
         lifecycleHooks: {}
         metadata:
           labels:
             cluster-api/accelerator: A10G
             node-role.kubernetes.io/gpu: ""
         providerSpec:
           value:
             ami:
               id: ami-0c65d71e89d43aa90 1
             apiVersion: awsproviderconfig.openshift.io/v1beta1
             blockDevices:
             - ebs:
                 iops: 0
                 kmsKey: {}
                 volumeSize: 240
                 volumeType: gp2
             credentialsSecret:
               name: aws-cloud-credentials
             deviceIndex: 0
             iamInstanceProfile:
               id: CLUSTER_NAME-CLUSTER_ID-worker-profile
             instanceType: g5.48xlarge
             kind: AWSMachineProviderConfig
             metadata:
               creationTimestamp: null
             metadataServiceOptions: {}
             placement:
               availabilityZone: AVAILABILITY_ZONE
               region: REGION
             securityGroups:
             - filters:
               - name: tag:Name
                 values:
                 - CLUSTER_NAME-CLUSTER_ID-node
             - filters:
               - name: tag:Name
                 values:
                 - CLUSTER_NAME-CLUSTER_ID-lb
             subnet:
               filters:
               - name: tag:Name
                 values:
                 - CLUSTER_NAME-CLUSTER_ID-subnet-private-AVAILABILITY_ZONE
             tags:
             - name: kubernetes.io/cluster/CLUSTER_NAME-CLUSTER_ID
               value: owned
             userDataSecret:
               name: worker-user-data
   EOF

   1

   Specify a valid Red Hat Enterprise Linux CoreOS (RHCOS) Amazon Machine Image (AMI) for your AWS zone for your OpenShift Container Platform nodes. If you want to use an AWS Marketplace image, you must complete the OpenShift Container Platform subscription from the AWS Marketplace to obtain an AMI ID for your region.

Procedure

1. Modify the parameters for the ClusterAutoscaler custom resource (CR) by using the sample resource definition file.

   cat << EOF > <filename>.yml
   apiVersion: autoscaling.openshift.io/v1
   kind: ClusterAutoscaler
   metadata:
     generation: 1
     name: default
   spec:
     logVerbosity: 4
     maxNodeProvisionTime: 15m
     podPriorityThreshold: -10
     resourceLimits:
       gpus:
       - max: 16
         min: 0
         type: A10G
     scaleDown:
       delayAfterAdd: 20m
       delayAfterDelete: 5m
       delayAfterFailure: 30s
       enabled: true
       unneededTime: 5m
   EOF

Procedure

1. Modify the parameters for the MachineAutoscaler custom resource (CR) by using the sample resource definition file.

   cat << EOF > <filename>.yml
   apiVersion: autoscaling.openshift.io/v1beta1
   kind: MachineAutoscaler
   metadata:
     name: CLUSTER_NAME-CLUSTER_ID-gpu-AVAILABILITY_ZONE
     namespace: "openshift-machine-api"
   spec:
     minReplicas: 0
     maxReplicas: 2
     scaleTargetRef:
       apiVersion: machine.openshift.io/v1beta1
       kind: MachineSet
       name: CLUSTER_NAME-CLUSTER_ID-gpu-AVAILABILITY_ZONE
   EOF

Procedure

1. Install the OCP AI operator on the Red Hat OpenShift Container Platform Service on AWS web console.
2. From the Data Science Cluster tab of the operator, click Create DataScienceCluster to create an instance with default values.
3. After you create the Data Science Cluster instance, select Red Hat OpenShift AI from the application launcher icon at the top to launch the OCP AI web console.

Procedure

1. On the OpenShift Container Platform AI dashboard, click Settings > Serving runtimes. The Serving runtimes page lists the vLLM ServingRuntime for KServe custom resource (CR). KServe orchestrates model serving for all types of models and includes model-serving runtimes that implement the loading of given types of model servers. KServe also handles the lifecycle of the deployment object, storage access, and networking setup.
2. Click on the kebab menu for vLLM ServingRuntime for KServe and select Duplicate serving runtime.
3. Enter a different display name for the serving runtime and increase the value for serving.knative.dev/progress-deadline to 60m.

4. To support multiple GPU nodes and scaling, add --distributed-executor-backend and --tensor-parallel-size to containers.args as follows:

   spec:
     containers:
       - args:
           - --port=8080
           - --model=/mnt/models
           - --served-model-name={{.Name}}
           - --distributed-executor-backend=mp
           - --tensor-parallel-size=8

   Next, you must create an accelerator profile if you want to run a GPU node for the first time.

Procedure

1. On the OpenShift Container Platform AI dashboard, click Settings > Accelerator profiles.
2. Click Create accelerator profile.
3. On the Create accelerator profile dialog, type NVIDIA GPU as the Name and nvidia.com/gpu as the Identifier.
4. To enable or disable the accelerator profile immediately after creation, click the toggle in the Enable column.
5. Click Add toleration to open the Add toleration dialog. Toleration schedules pods with matching taints.
6. From the Operator list, select Exists. The key/effect parameters in the taint must match the same parameters configured under toleration in the pod. You must leave a blank value parameter, which matches any value.
7. Enter nvidia.com/gpu as the key.
8. From the Effect list, select NoSchedule. New pods that do not match the taint are not scheduled onto that node. Existing pods on the node remain.
9. Click Add to add the toleration configuration for the node.
10. Click Create accelerator profile to complete the accelerator configuration.

Procedure

1. In the OpenShift Container Platform AI web console, select Data science projects. The Data science projects page shows a list of projects that you can access. For each user-requested project in the list, the Name column shows the project display name, the user who requested the project, and the project description.
2. Click Create project. In the Create project dialog, update the Name field to enter a unique display name for your project.
3. Optional: In the Description field, provide a project description.
4. Click Create. Your project is listed on the Data science projects page.
5. Click the name of your project, select the Connections tab, and click Create connection.
6. In the Connection type drop down, select S3 compatible object storage - v1.
7. In the Connection details section, enter the connection name, the access key, the secret key, endpoint to your storage bucket, and the region.
8. Click Create.

Procedure

1. In the OpenShift Container Platform AI dashboard, navigate to the project details page and click the Models tab.
2. In the Single-model serving platform tile, click Select single-model to open the Deploy model dialog.

3. Complete the following configurations:

   1. Model name: enter the model name in RFC compliant format.
   2. Serving runtime: select the serving runtime you configured.
   3. Model framework: select vLLM.
   4. Model server size: select Large.
   5. Accelerator: select the accelerator you configured.
   6. Number of accelerators: enter 8
   7. Make deployed models available through an external route: enable the option.
   8. Require token authentication: enable the option.
   9. Existing connection: enable existing connection under Source model location.
   10. Connection: select the data connection name for the data connection you created
   11. Path: Enter the path to your model. For example, Llama-3.1-8B-Instruct.
4. Click Deploy. On the Models tab, an endpoint and token will be provided for your model after it is provisioned.

5. Export variables to access the model after the model status becomes Ready. Replace <values> in the command with applicable values for the variables.

   $ export SERVING_NAME=<serving-name>
   $ export NAMESPACE=<project-name>
   $ export TOKEN=$(oc get secret -n $NAMESPACE default-name-$SERVING_NAME-sa -o go-template='{{ .data.token }}' | base64 -d)
   $ export ENDPOINT=https://$(oc get route -n istio-system ${SERVING_NAME}-${NAMESPACE} -o go-template='{{ .spec.host }}')
   $ curl -k -w %{certs} $ENDPOINT > ca-cert.pem
   $ export SSL_CERT_FILE=ca-cert.pem
   $ export REQUESTS_CA_BUNDLE=ca-cert.pem
   $ export OPENAI_API_BASE="$ENDPOINT/v1"

   Note

   Note the OPENAI_API_BASE endpoint URL.

   Verification

   You can verify the successful deployment of the model by using the following curl command to write a short Python programme by using the model. Replace PROVIDED_ENDPOINT and PROVIDED_TOKEN with the values on the Models tab. The following command has an example model name which must be replaced with the model that you deployed.

Procedure

1. Enter the following command to generate the OPENAI_API_KEY:

   $ export OPENAI_API_KEY=$(oc create token --duration=87600h -n ${NAMESPACE} ${SERVING_NAME}-sa)

2. Next, you must enter the following configuration in the provider-settings.yaml in the MTA with Developer Lightspeed Visual Studio (VS) Code extension.

   openshift-kai-test-generation: &active
       environment:
         SSL_CERT_FILE: "<name-of-SSL_CERT_FILE>"
         REQUESTS_CA_BUNDLE: "<name-of-REQUESTS_CA_BUNDLE>"
         OPENAI_API_KEY: "<OPENAI_API_KEY>"
   
       provider: "ChatOpenAI"
       args:
         model: "<serving-name>"
         base_url:  "https://<serving-name>-<data-science-project-name>.apps.konveyor-ai.migration.redhat.com/v1"