gpus_monitor/README.rst

# GPUs Monitor



`gpus_monitor` is a Python GPUs activities monitoring tool designed to report by email new and recently died compute processes over the machine where it has been run on.
Basically, when you have just run a new stable training on the machine where `gpus_monitor` listen to, you will received in a few seconds an email notification. This email will contains several informations about the process you've launched.
You received also an email if a compute process died (with EXIT_STATUS = 0 or not).


### Kind of mail gpus_monitor is going to send you :

1. New training detected :

>> From: <gpusstatus@mydomain.com>
>> Subject : 1 processes running on <MACHINE_NAME> (LOCAL_IP_OF_THE_MACHINE)
>> 
>> New events (triggered on the 08/11/2020 11:45:52):
>> 
>>             ---------------------------------------------------------------------------------------------------------------
>>             A new process (PID : 12350) has been launched on GPU 0 (Quadro RTX 4000) by <owner_of_the_process> since 08/11/2020 11:43:48
>>             His owner (<owner_of_the_process>) has executed the following command :
>>                 python3 test_torch.py
>>             From :
>>                 <absolute_path_to_the_script_of_the_new_launched_process>
>>             
>>             CPU Status (currently):
>>                 For this process : 19/40 logic cores (47.5%)
>>             
>>             GPU Status (currently):
>>                 - Used memory (for this process): 879 / 7979.1875 MiB (11.02 % used)
>>                 - Used memory (for all processes running on this GPU) 7935.3125 / 7979.1875 MiB (99.45 % used)
>>                 - Temperature : 83 Celsius
>>                 - Driver version : 435.21
>>             ---------------------------------------------------------------------------------------------------------------
>>             
>>             
>>         
>> This message has been automatically send by a robot. Please don't answer to this mail
>> Please, feel free to open a merge request on github.com/araison12/gpus_monitor if you have encountered a bug or to share your ideas to improve this tool


2. Training died (either finished well or not)
>> From: <gpusstatus@mydomain.com>
>> Subject : 1 processes running on <MACHINE_NAME> (LOCAL_IP_OF_THE_MACHINE)
>> 
>> New events (triggered on the 08/11/2020 11:47:29):
>> 
>>         ---------------------------------------------------------------------------------------------------------------
>>         The process (PID : 12350) launched by araison since 08/11/2020 11:43:48 has ended.
>>         His owner araison had executed the following command :
>>             python3 test_torch.py
>>         From :
>>             <absolute_path_to_the_script_of_the_died_process>
>>         
>>         The process took 0:03:41 to finish.
>>         --------------------------------------------------------------------------------------------------------------
>>     
>> This message has been automatically send by a robot. Please don't answer to this mail
>> Please, feel free to open a merge request on github.com/araison12/gpus_monitor if you have encountered a bug or to share your ideas to improve this tool      


## Instructions to use gpus_monitor :


1. Cloning this repository :

`git clone https://github.com/araison12/gpus_monitor.git`

2. Installing dependencies :

`pip3 install -r gpus_monitor/requirements.txt`

or

`python3 setup.py install --user`

3. Add peoples mail to the list of the `persons_to_inform.yaml` file :

Example:

```yaml
list:  
	- adrien.raison@univ-poitiers.fr
	- other_person_to_inform@hisdomain.com
```
	
	

Note : You can hot-add/remove mails in this file without the need of killing the scanning process !

4. Add SMTP Server parameters (server adress, credentials, port number, etc..)

You can manage these stuff in the `gpus_monitor/src/gpus_monitor/config.py` file :
To adjust these varibales you have to edit the `gpus_monitor/src/gpus_monitor/config.py` file.

```bash
cd gpus_monitor/src/gpus_monitor/
vim config.py
```


For privacy purposes, login of my dedicated SMTP account are stored in 2 machine environment variables. I've set up a brandnew Gmail account for my `gpus_monitor` instance. I can share with you my credentials in order to use a single SMTP account for `gpus_monitor` instance on several machines, feel free to send me an email !
Otherwise, fill in with your own SMTP server configuration.


```python
USER = os.environ.get(
    "GPUSMONITOR_MAIL_USER"
)  

PASSWORD = os.environ.get(
    "GPUSMONITOR_MAIL_PASSWORD"
) 
PORT = 465
SMTP_SERVER = "smtp.gmail.com"
```

See https://askubuntu.com/a/58828 to handle efficiently (permanent adding) environment variables.

5. Adjust the scanning rate of `gpus_monitor` and the processes age that he has to take in account.


The `WAITING_TIME` variable adjusts the scan timing rate of gpus_monitor.

```python
WAITING_TIME = 0.5  # min
```

The `PROCESS_AGE`  variable adjusts the processes age that gpus_monitor has to take in account.

```python
PROCESS_AGE = 2  # min (gpus_monitor only consider >=2min aged processes)
```

6. Executing `gpus_monitor` when machine starts up.

```bash
crontab -e
```
Add the following line to the brandnew opened file :

```bash
@reboot python3 /path/to/gpu_monitor/src/gpus_monitor/main.py
```

# Ideas to enhanced the project :

- Log system (owner, total calculation time by user)
- Manage cases in email sending (subject): processes finished well or not (Send Traceback)
- Centralized system that scan every machine on a given IP adresses range.
- Better errors management (SMTP connection failed, no Cuda GPU on the machine,..)
- Documenting the project
- Rewrite it in oriented object fashion 


If you have any ideas to improve this project, don't hesitate to make a merge request ! :)