Skip to main content

AI Training - Monitor your cloud resources

Stephanie R.
Updated

Description

AI-Training provides users with access to several types of resources. Resource usage can be fetched through a dedicated UI on Grafana.

 

Requirements

  • a running job on AI-Training

 

UI Access

The URL to access the monitoring is built like this: https://monitoring.<REGION>.ai.cloud.ovh.us/d/gpu?var-job=<JOB-ID>

It can be fetched in the CLI using this command:

ovhai job get <JOB-ID>

---
Name:       ai-training-transformers-heuristic
ID:         xxx
Created At: 2021-04-14 09:30:02.709449 UTC
Updated At: 2021-04-14 09:30:09.328392 UTC
User:       user-xxx
Spec:       
  Image:        ovhcom/ai-training-transformers:3.1.0
  Command:      
  Env:          
  Default Port: 8080
  Unsecure:     false
  Resources:    
    GPU:       1
    GPU Model: Tesla-V100S
    CPU:       13
  Volumes:      
  Timeout:      0
  Labels:       
  SSH Keys:     
Status:     
  State:          RUNNING
  Queued At:      14-04-21 09:30
  Started At:     14-04-21 09:30
  Stopped At:     
  IP:             10.42.165.21
  Duration:       2s
  Job Url:        https://xxx.job.console.preprod.ai.cloud.ovh.us
  SSH Url:        
  Monitoring Url: https://monitoring.console.preprod.ai.cloud.ovh.us/d/gpu?var-job=xxx&from=1618392608000
  Infos:          
  Data Sync:      
  Sync Progress:  
  History:        
    State         Date
    -----         ----
    QUEUED        2021-04-14 09:30:02.708117 UTC
    INITIALIZING  2021-04-14 09:30:02.802995 UTC
    PENDING       2021-04-14 09:30:02.824278 UTC
    RUNNING       2021-04-14 09:30:09.327003 UTC

It is displayed in the Monitoring Url line, which is accessed through any private browser.

 

Panel Details

We will now go through each panel to give a short description.

PaaS_AI_Training_monitoring02.png

 

GPU Usage

NOTE: This panel will only be present on GPU jobs.

This panel displays the usage of each GPU allocated to your job.

 

GPU Memory

NOTE: This panel will only be present on GPU jobs.

This panel displays the usage and limit of memory for each GPU allocated to your Job

 

CPU Usage

This panel displays the overall CPU usage of your job.

 

Memory Usage

This panel displays the usage and limit of Memory allocated to your job.

 

Network Usage

This panel displays input and output traffic on your job.

 

Ephemeral storage usage

This panel shows the usage and limit of ephemeral storage allocated to your job. Jobs can use ephemeral storage for data not within a synchronized container.

If your usage goes beyond the limit of the ephemeral storage, your job will be rejected.

 

Go further

For more information and tutorials, please see our other AI & Machine Learning support guides or explore the guides for other OVHcloud products and services.

If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.

Related articles