GPUs Monitor

gpus_monitor is a Pythonic NVIDIA ® GPUs activities monitoring tool designed to report (stdout, email, etc.) new or recently died computing processes.

Use case

Basically, when you have just run a new stable training on the machine where gpus_monitor is listening to, you will received within few seconds an email notification about this brand new script launch. This email will contains several informations regarding this process.

You will also receive an email if a compute process died (either because of EXIT_STATUS = 0 or not).

Mail content gpus_monitor will send you :

1. New training detected :

From: gpusstatus@mydomain.com Subject : 1 processes running on <MACHINE_NAME> (LOCAL_IP_OF_THE_MACHINE)

New events (triggered on the 08/11/2020 11:45:52):

        ---------------------------------------------------------------------------------------------------------------
        A new process (PID : 12350) has been launched on GPU 0 (Quadro RTX 4000) by <owner_of_the_process> since 08/11/2020 11:43:48
        His owner (<owner_of_the_process>) has executed the following command :
            python3 test_torch.py
        From :
            <absolute_path_to_the_script_of_the_new_launched_process>
        
        CPU Status (currently):
            For this process : 19/40 logic cores (47.5%)
        
        GPU Status (currently):
            - Used memory (for this process): 879 / 7979.1875 MiB (11.02 % used)
            - Used memory (for all processes running on this GPU) 7935.3125 / 7979.1875 MiB (99.45 % used)
            - Temperature : 83 Celsius
            - Driver version : 435.21
        ---------------------------------------------------------------------------------------------------------------

This message has been automatically send by a robot. Please don't answer to this mail Please, feel free to open a merge request on github.com/araison12/gpus_monitor if you have encountered a bug or to share your ideas to improve this tool

2. Training died (either finished well or not)

From: gpusstatus@mydomain.com Subject : 1 processes running on <MACHINE_NAME> (LOCAL_IP_OF_THE_MACHINE)

New events (triggered on the 08/11/2020 11:47:29):

    ---------------------------------------------------------------------------------------------------------------
    The process (PID : 12350) launched by araison since 08/11/2020 11:43:48 has ended.
    His owner <owner_of_the_process> had executed the following command :
        python3 test_torch.py
    From :
        <absolute_path_to_the_script_of_the_died_process>
    
    The process took 0:03:41 to finish.
    --------------------------------------------------------------------------------------------------------------

This message has been automatically send by a robot. Please don't answer to this mail Please, feel free to open a merge request on github.com/araison12/gpus_monitor if you have encountered a bug or to share your ideas to improve this tool

How to

1. Cloning this repository :

git clone https://gitea.zaclys.com/araison/gpus_monitor.git

2. Installing dependencies :

pip3 install -r gpus_monitor/requirements.txt

or

python3 gpus_monitor/setup.py install --user

3. Add peoples mail to the list of the persons_to_inform.yaml file :

Example:

list:  
	- <email 1>
	- <other_person_to_inform@hisdomain.com>

Note : You can hot-add/remove mails in this file without the need of killing the scanning process !

4. Add SMTP Server parameters (server adress, credentials, port number, etc..)

You can manage these stuff in the gpus_monitor/src/gpus_monitor/config.py file : To adjust these varibales you have to edit the gpus_monitor/src/gpus_monitor/config.py file.

cd gpus_monitor/src/gpus_monitor/
vim config.py

For privacy purposes, login of my dedicated SMTP account are stored in a machine in 2 environment variables.

USER = os.environ.get(
    "GPUSMONITOR_MAIL_USER"
)  

PASSWORD = os.environ.get(
    "GPUSMONITOR_MAIL_PASSWORD"
) 
PORT = 465
SMTP_SERVER = "smtp.example.com"

See https://askubuntu.com/a/58828 to handle efficiently (permanent adding) environment variables.

5. Adjust the scanning rate of gpus_monitor and the processes age that he has to take in account.

The WAITING_TIME variable adjusts the scan timing rate of gpus_monitor.

WAITING_TIME = 0.5  # min

The PROCESS_AGE variable adjusts the processes age that gpus_monitor has to track down.

PROCESS_AGE = 2  # min (gpus_monitor only consider >=2min aged processes)

6. Executing gpus_monitor when machine starts up.

crontab -e

Add the following line to the brandnew opened file :

@reboot python3 /path/to/gpu_monitor/src/gpus_monitor/main.py

Ideas to enhance the project :

If you have any ideas to improve this project, don't hesitate to make a merge request ! :)

To test gpus_monitor by your own:

I've implemented the tiny non linear XOR problem in PyTorch. You can test gpus_monitor by your own while running :

python3 gpus_monitor/test_torch.py