Cluster status Sat, 11 Jul 2026 16:52:01 +0000 report from beholder03

Resource usage (overall)
Resource usage (by namespace)
Ceph file system status
Etcd cluster status
Detailed network health report
nVidia Driver and GPU reports
  Imp
  Dretch
  Belial
  Fierna
  Tiamat
  Vecna
  Asmodeus
  Zariel
  Demogorgon

Resource usage (overall)

 Resource                                                                    Requested          Limit  Allocatable     Free 
  cpu                                                                      (42%) 655.4    (48%) 758.9         1.6k    817.1 
  ├─ asmodeus                                                              (78%) 200.4    (78%) 200.1        256.0     55.6 
  │  ├─ dnsutils-asmodeus                                                       100.0m         100.0m                       
  │  ├─ gpu-a100-zsh-shm                                                          32.0           32.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-9q5ls               5.0m            0.0                       
  │  ├─ kube-router-9q4k5                                                       250.0m            0.0                       
  │  ├─ step16-coolchic-gpu-run-asmodeus-a100-hq                                  20.0           20.0                       
  │  ├─ step18-ftic-gpu-run-asmodeus-a100-hq-40cpu                                40.0           40.0                       
  │  ├─ step19-deep-iqa-models-gpu-run-asmodeus-a100                             100.0          100.0                       
  │  └─ ulas-bingoel-model-pod-5                                                   8.0            8.0                       
  ├─ beholder01                                                            (4%) 900.0m    (0%) 100.0m         24.0     23.1 
  │  ├─ dnsutils-beholder01                                                     100.0m         100.0m                       
  │  ├─ kube-apiserver-beholder01                                               250.0m            0.0                       
  │  ├─ kube-controller-manager-beholder01                                      200.0m            0.0                       
  │  ├─ kube-router-6bz7x                                                       250.0m            0.0                       
  │  └─ kube-scheduler-beholder01                                               100.0m            0.0                       
  ├─ beholder02                                                              (13%) 3.2      (13%) 3.1         24.0     20.8 
  │  ├─ dnsutils-beholder02                                                     100.0m         100.0m                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-8rfmd               5.0m            0.0                       
  │  ├─ kube-apiserver-beholder02                                               250.0m            0.0                       
  │  ├─ kube-controller-manager-beholder02                                      200.0m            0.0                       
  │  ├─ kube-router-hgnjl                                                       250.0m            0.0                       
  │  ├─ kube-scheduler-beholder02                                               100.0m            0.0                       
  │  ├─ mc-rj-ddcb7fb6-b2qh6                                                       2.0            2.0                       
  │  ├─ nginx-rsn-2024-57d49484d-47rfc                                          250.0m            1.0                       
  │  ├─ virt-api-85578d9bb7-5fwdl                                                 5.0m            0.0                       
  │  ├─ virt-controller-674bcccb6-pvgxj                                          10.0m            0.0                       
  │  └─ virt-operator-b4d8f7f58-6jjxp                                            10.0m            0.0                       
  ├─ beholder03                                                               (7%) 1.6    (2%) 600.0m         24.0     22.4 
  │  ├─ coredns-66bc5c9577-d6czh                                                100.0m            0.0                       
  │  ├─ dnsutils-beholder03                                                     100.0m         100.0m                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-master-66685h4zx         100.0m            0.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-8jfkj               5.0m            0.0                       
  │  ├─ gpu-operator-f96dbb6fc-nk2jk                                            200.0m         500.0m                       
  │  ├─ kube-apiserver-beholder03                                               250.0m            0.0                       
  │  ├─ kube-controller-manager-beholder03                                      200.0m            0.0                       
  │  ├─ kube-router-2dszl                                                       250.0m            0.0                       
  │  ├─ kube-scheduler-beholder03                                               100.0m            0.0                       
  │  ├─ ldap-67b47cf9b9-v6vvm                                                   250.0m            0.0                       
  │  ├─ virt-api-85578d9bb7-6mvxx                                                 5.0m            0.0                       
  │  ├─ virt-controller-674bcccb6-f2xfj                                          10.0m            0.0                       
  │  └─ virt-operator-b4d8f7f58-pnbkz                                            10.0m            0.0                       
  ├─ belial                                                                   (5%) 3.7       (9%) 7.1         80.0     72.9 
  │  ├─ coredns-66bc5c9577-kxlwp                                                100.0m            0.0                       
  │  ├─ dnsutils-belial                                                         100.0m         100.0m                       
  │  ├─ gatekeeper-controller-manager-66f474f785-bq2vs                          100.0m            1.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-j7bdx               5.0m            0.0                       
  │  ├─ kube-router-b8htr                                                       250.0m            0.0                       
  │  ├─ ledavio-text-search-5d8f755795-tqp4s                                       1.0            2.0                       
  │  ├─ ollama-pod                                                                 1.0            1.0                       
  │  ├─ storage-pod                                                             100.0m            1.0                       
  │  ├─ transfer-aalbi                                                             1.0            2.0                       
  │  └─ virt-handler-rvzhr                                                       10.0m            0.0                       
  ├─ demogorgon                                                             (92%) 88.4   (158%) 152.1         96.0      0.0 
  │  ├─ coolchic5-gpu-run-demogorgon-a100-32cpu-3gpu-2                            32.0           32.0                       
  │  ├─ dnsutils-demogorgon                                                     100.0m         100.0m                       
  │  ├─ felix-petersen-job-29                                                      6.0            8.0                       
  │  ├─ gpu-demogorgon                                                            48.0           48.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-pxws7               5.0m            0.0                       
  │  ├─ kube-router-5g7fj                                                       250.0m            0.0                       
  │  ├─ pycharm                                                                    1.0           32.0                       
  │  └─ pycharmv2                                                                  1.0           32.0                       
  ├─ fierna                                                                   (7%) 5.6       (9%) 7.1         80.0     72.9 
  │  ├─ dnsutils-fierna                                                         100.0m         100.0m                       
  │  ├─ gatekeeper-controller-manager-66f474f785-dvhjf                          100.0m            1.0                       
  │  ├─ gatekeeper-controller-manager-66f474f785-s9kgj                          100.0m            1.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-ldxbm               5.0m            0.0                       
  │  ├─ kube-router-8g7hh                                                       250.0m            0.0                       
  │  ├─ ledavio-similarity-search-9b864cc89-ln9lk                                  4.0            4.0                       
  │  ├─ till-aust-ubuntu-entry-pod                                                 1.0            1.0                       
  │  └─ virt-handler-xmxj8                                                       10.0m            0.0                       
  ├─ kiaransalee                                                           (0%) 355.0m    (0%) 100.0m        192.0    191.6 
  │  ├─ dnsutils-kiaransalee                                                    100.0m         100.0m                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-jn8tv               5.0m            0.0                       
  │  └─ kube-router-f8tf4                                                       250.0m            0.0                       
  ├─ mindflayer01                                                           (33%) 21.0     (31%) 20.1         64.0     43.0 
  │  ├─ cdi-apiserver-7745487599-s5g6s                                          100.0m            0.0                       
  │  ├─ cdi-deployment-6c99bb8fcf-h79j8                                         100.0m            0.0                       
  │  ├─ cdi-uploadproxy-684cf5d896-9x5rh                                        100.0m            0.0                       
  │  ├─ dnsutils-mindflayer01                                                   100.0m         100.0m                       
  │  ├─ file-pod                                                                200.0m            1.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-gc-767f4bbfwdphf          10.0m            0.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-lv7bq               5.0m            0.0                       
  │  ├─ kube-router-zhxjw                                                       250.0m            0.0                       
  │  ├─ ubuntu-test-pod                                                         100.0m            1.0                       
  │  ├─ urs-waldmann-tb-access-pod                                                16.0           16.0                       
  │  ├─ valentin-schmuker-storage                                                  2.0            2.0                       
  │  ├─ virt-handler-tqrj4                                                       10.0m            0.0                       
  │  └─ virt-launcher-lightfield-analysis-8zbf6                                    2.0          15.0m                       
  ├─ mindflayer02                                                             (3%) 1.9       (3%) 2.1         64.0     61.9 
  │  ├─ dex-586d7ff58f-vpklf                                                    250.0m            0.0                       
  │  ├─ dex-loginapp-6d47648b8b-p7v2x                                           100.0m            0.0                       
  │  ├─ dex-mysql-589f4586bc-4n5vj                                              100.0m            0.0                       
  │  ├─ dnsutils-mindflayer02                                                   100.0m         100.0m                       
  │  ├─ gatekeeper-audit-59d4b6fd4c-gtwjm                                       100.0m            1.0                       
  │  ├─ kube-router-zc9q4                                                       250.0m            0.0                       
  │  ├─ mediawiki-77f9c84df5-p6k9g                                              250.0m            0.0                       
  │  ├─ mediawiki-mariadb-7ffb6c9b8d-ng7s6                                      250.0m            0.0                       
  │  ├─ registry-74dfd5b88d-dvkg8                                               100.0m            0.0                       
  │  ├─ registry-auth-86fdd9f85-k7nr2                                           100.0m            0.0                       
  │  ├─ registry-browser-7f4cbdf96b-d4b66                                       200.0m            0.0                       
  │  └─ ubuntu-test-pod                                                         100.0m            1.0                       
  ├─ mindflayer03                                                           (41%) 26.5     (41%) 26.1         64.0     37.5 
  │  ├─ cdi-operator-76f7d8c545-vrql7                                           100.0m            0.0                       
  │  ├─ dnsutils-mindflayer03                                                   100.0m         100.0m                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-vlfn5               5.0m            0.0                       
  │  ├─ gpu-pod-aalbi                                                             10.0           10.0                       
  │  ├─ kube-router-t8kl8                                                       250.0m            0.0                       
  │  ├─ urs-waldmann-ubuntu-pod                                                   16.0           16.0                       
  │  └─ virt-handler-hndfm                                                       10.0m            0.0                       
  ├─ tiamat                                                                 (11%) 27.4     (26%) 66.1        256.0    189.9 
  │  ├─ dnsutils-tiamat                                                         100.0m         100.0m                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-jg9c7               5.0m            0.0                       
  │  ├─ kube-router-9m4q5                                                       250.0m            0.0                       
  │  ├─ train-baseline-full-dataset-filtered-split-r8hdq                           2.0           16.0                       
  │  └─ ubuntu-gpu1                                                               25.0           50.0                       
  ├─ vecna                                                                  (21%) 20.4     (21%) 20.1         96.0     75.6 
  │  ├─ bee-finetune-vecna-terminal                                               16.0           16.0                       
  │  ├─ dnsutils-vecna                                                          100.0m         100.0m                       
  │  ├─ felix-petersen-job-20                                                      4.0            4.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-mvz86               5.0m            0.0                       
  │  └─ kube-router-zg4xh                                                       250.0m            0.0                       
  └─ zariel                                                                (99%) 254.4    (99%) 254.1        256.0      1.6 
     ├─ allin1                                                                     4.0            4.0                       
     ├─ dnsutils-zariel                                                         100.0m         100.0m                       
     ├─ gpu-operator-1777509885-node-feature-discovery-worker-5fsqh               5.0m            0.0                       
     ├─ kube-router-cqsw4                                                       250.0m            0.0                       
     └─ zariel-a2v2-shark-julian-schaefer-zimmermann                             250.0          250.0                       
  devices.kubevirt.io/kvm                                                     (0%) 1.0       (0%) 1.0         4.0k     4.0k 
  ├─ belial                                                                   (0%) 0.0       (0%) 0.0         1.0k     1.0k 
  ├─ fierna                                                                   (0%) 0.0       (0%) 0.0         1.0k     1.0k 
  ├─ mindflayer01                                                             (0%) 1.0       (0%) 1.0         1.0k    999.0 
  │  └─ virt-launcher-lightfield-analysis-8zbf6                                    1.0            1.0                       
  └─ mindflayer03                                                             (0%) 0.0       (0%) 0.0         1.0k     1.0k 
  devices.kubevirt.io/tun                                                     (0%) 1.0       (0%) 1.0         4.0k     4.0k 
  ├─ belial                                                                   (0%) 0.0       (0%) 0.0         1.0k     1.0k 
  ├─ fierna                                                                   (0%) 0.0       (0%) 0.0         1.0k     1.0k 
  ├─ mindflayer01                                                             (0%) 1.0       (0%) 1.0         1.0k    999.0 
  │  └─ virt-launcher-lightfield-analysis-8zbf6                                    1.0            1.0                       
  └─ mindflayer03                                                             (0%) 0.0       (0%) 0.0         1.0k     1.0k 
  devices.kubevirt.io/vhost-net                                               (0%) 1.0       (0%) 1.0         4.0k     4.0k 
  ├─ belial                                                                   (0%) 0.0       (0%) 0.0         1.0k     1.0k 
  ├─ fierna                                                                   (0%) 0.0       (0%) 0.0         1.0k     1.0k 
  ├─ mindflayer01                                                             (0%) 1.0       (0%) 1.0         1.0k    999.0 
  │  └─ virt-launcher-lightfield-analysis-8zbf6                                    1.0            1.0                       
  └─ mindflayer03                                                             (0%) 0.0       (0%) 0.0         1.0k     1.0k 
  ephemeral-storage                                                        (1%) 139.9G   (2%) 211.0Gi        11.3T    11.1T 
  ├─ asmodeus                                                                 (0%) 0.0       (0%) 0.0        94.6G    94.6G 
  ├─ beholder01                                                               (0%) 0.0       (0%) 0.0         1.7T     1.7T 
  ├─ beholder02                                                               (0%) 0.0       (0%) 0.0         1.7T     1.7T 
  ├─ beholder03                                                               (0%) 0.0       (0%) 0.0         1.7T     1.7T 
  ├─ belial                                                                   (0%) 0.0       (0%) 0.0       189.2G   189.2G 
  ├─ demogorgon                                                            (5%) 30.0Gi    (9%) 60.0Gi       706.7G   642.3G 
  │  ├─ pycharm                                                                 15.0Gi         30.0Gi                       
  │  └─ pycharmv2                                                               15.0Gi         30.0Gi                       
  ├─ fierna                                                               (0%) 256.0Mi     (1%) 1.0Gi       189.2G   188.1G 
  │  └─ ledavio-similarity-search-9b864cc89-ln9lk                              256.0Mi          1.0Gi                       
  ├─ kiaransalee                                                              (0%) 0.0       (0%) 0.0         1.7T     1.7T 
  ├─ mindflayer01                                                           (0%) 50.0M       (0%) 0.0       211.5G   211.5G 
  │  └─ virt-launcher-lightfield-analysis-8zbf6                                  50.0M            0.0                       
  ├─ mindflayer02                                                             (0%) 0.0       (0%) 0.0       211.5G   211.5G 
  ├─ mindflayer03                                                             (0%) 0.0       (0%) 0.0       211.5G   211.5G 
  ├─ tiamat                                                              (65%) 100.0Gi  (98%) 150.0Gi       164.4G     3.3G 
  │  └─ ubuntu-gpu1                                                            100.0Gi        150.0Gi                       
  ├─ vecna                                                                    (0%) 0.0       (0%) 0.0       849.0G   849.0G 
  └─ zariel                                                                   (0%) 0.0       (0%) 0.0         1.7T     1.7T 
  memory                                                                    (31%) 4.3T     (34%) 4.7T       12.7Ti    8.4Ti 
  ├─ asmodeus                                                            (37%) 736.4Gi  (38%) 764.6Gi        2.0Ti    1.2Ti 
  │  ├─ dnsutils-asmodeus                                                      100.0Mi        100.0Mi                       
  │  ├─ gpu-a100-zsh-shm                                                       256.0Gi        284.0Gi                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-9q5ls             64.0Mi        512.0Mi                       
  │  ├─ kube-router-9q4k5                                                      250.0Mi            0.0                       
  │  ├─ step16-coolchic-gpu-run-asmodeus-a100-hq                               100.0Gi        100.0Gi                       
  │  ├─ step18-ftic-gpu-run-asmodeus-a100-hq-40cpu                             100.0Gi        100.0Gi                       
  │  ├─ step19-deep-iqa-models-gpu-run-asmodeus-a100                           200.0Gi        200.0Gi                       
  │  └─ ulas-bingoel-model-pod-5                                                80.0Gi         80.0Gi                       
  ├─ beholder01                                                           (0%) 350.0Mi   (0%) 100.0Mi       92.9Gi   92.5Gi 
  │  ├─ dnsutils-beholder01                                                    100.0Mi        100.0Mi                       
  │  └─ kube-router-6bz7x                                                      250.0Mi            0.0                       
  ├─ beholder02                                                             (3%) 2.6Gi     (2%) 1.6Gi       92.9Gi   90.3Gi 
  │  ├─ dnsutils-beholder02                                                    100.0Mi        100.0Mi                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-8rfmd             64.0Mi        512.0Mi                       
  │  ├─ kube-router-hgnjl                                                      250.0Mi            0.0                       
  │  ├─ mc-rj-ddcb7fb6-b2qh6                                                     1.0Gi          1.0Gi                       
  │  ├─ virt-api-85578d9bb7-5fwdl                                              500.0Mi            0.0                       
  │  ├─ virt-controller-674bcccb6-pvgxj                                        275.0Mi            0.0                       
  │  └─ virt-operator-b4d8f7f58-6jjxp                                          450.0Mi            0.0                       
  ├─ beholder03                                                             (2%) 1.9Gi     (5%) 5.1Gi       92.9Gi   87.8Gi 
  │  ├─ coredns-66bc5c9577-d6czh                                                70.0Mi        170.0Mi                       
  │  ├─ dnsutils-beholder03                                                    100.0Mi        100.0Mi                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-master-66685h4zx        128.0Mi          4.0Gi                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-8jfkj             64.0Mi        512.0Mi                       
  │  ├─ gpu-operator-f96dbb6fc-nk2jk                                           100.0Mi        350.0Mi                       
  │  ├─ kube-router-2dszl                                                      250.0Mi            0.0                       
  │  ├─ virt-api-85578d9bb7-6mvxx                                              500.0Mi            0.0                       
  │  ├─ virt-controller-674bcccb6-f2xfj                                        275.0Mi            0.0                       
  │  └─ virt-operator-b4d8f7f58-pnbkz                                          450.0Mi            0.0                       
  ├─ belial                                                                (9%) 67.1Gi    (9%) 70.3Gi      754.4Gi  684.2Gi 
  │  ├─ coredns-66bc5c9577-kxlwp                                                70.0Mi        170.0Mi                       
  │  ├─ dnsutils-belial                                                        100.0Mi        100.0Mi                       
  │  ├─ gatekeeper-controller-manager-66f474f785-bq2vs                         256.0Mi        512.0Mi                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-j7bdx             64.0Mi        512.0Mi                       
  │  ├─ kube-router-b8htr                                                      250.0Mi            0.0                       
  │  ├─ ledavio-text-search-5d8f755795-tqp4s                                    32.0Gi         32.0Gi                       
  │  ├─ ollama-pod                                                              32.0Gi         32.0Gi                       
  │  ├─ storage-pod                                                            100.0Mi          1.0Gi                       
  │  ├─ transfer-aalbi                                                           2.0Gi          4.0Gi                       
  │  └─ virt-handler-rvzhr                                                     325.0Mi            0.0                       
  ├─ demogorgon                                                          (21%) 422.4Gi  (24%) 480.6Gi        2.0Ti    1.5Ti 
  │  ├─ coolchic5-gpu-run-demogorgon-a100-32cpu-3gpu-2                         200.0Gi        200.0Gi                       
  │  ├─ dnsutils-demogorgon                                                    100.0Mi        100.0Mi                       
  │  ├─ felix-petersen-job-29                                                   50.0Gi        100.0Gi                       
  │  ├─ gpu-demogorgon                                                          80.0Gi         80.0Gi                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-pxws7             64.0Mi        512.0Mi                       
  │  ├─ kube-router-5g7fj                                                      250.0Mi            0.0                       
  │  ├─ pycharm                                                                 46.0Gi         50.0Gi                       
  │  └─ pycharmv2                                                               46.0Gi         50.0Gi                       
  ├─ fierna                                                              (31%) 233.2Gi  (31%) 233.6Gi      754.4Gi  520.8Gi 
  │  ├─ dnsutils-fierna                                                        100.0Mi        100.0Mi                       
  │  ├─ gatekeeper-controller-manager-66f474f785-dvhjf                         256.0Mi        512.0Mi                       
  │  ├─ gatekeeper-controller-manager-66f474f785-s9kgj                         256.0Mi        512.0Mi                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-ldxbm             64.0Mi        512.0Mi                       
  │  ├─ kube-router-8g7hh                                                      250.0Mi            0.0                       
  │  ├─ ledavio-similarity-search-9b864cc89-ln9lk                               32.0Gi         32.0Gi                       
  │  ├─ till-aust-ubuntu-entry-pod                                             200.0Gi        200.0Gi                       
  │  └─ virt-handler-xmxj8                                                     325.0Mi            0.0                       
  ├─ kiaransalee                                                             (0%) 3.7G   (0%) 612.0Mi        1.5Ti     1.6T 
  │  ├─ dnsutils-kiaransalee                                                   100.0Mi        100.0Mi                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-jn8tv             64.0Mi        512.0Mi                       
  │  ├─ jupyter-elenasolar                                                        1.1G            0.0                       
  │  ├─ jupyter-giordano-2ddemarzo                                                1.1G            0.0                       
  │  ├─ jupyter-saroyehun                                                         1.1G            0.0                       
  │  └─ kube-router-f8tf4                                                      250.0Mi            0.0                       
  ├─ mindflayer01                                                          (19%) 77.9G   (45%) 181.1G      376.5Gi  207.8Gi 
  │  ├─ alertmanager-kube-prometheus-stack-1777-alertmanager-0                 200.0Mi            0.0                       
  │  ├─ cdi-apiserver-7745487599-s5g6s                                         150.0Mi            0.0                       
  │  ├─ cdi-deployment-6c99bb8fcf-h79j8                                        150.0Mi            0.0                       
  │  ├─ cdi-uploadproxy-684cf5d896-9x5rh                                       150.0Mi            0.0                       
  │  ├─ dnsutils-mindflayer01                                                  100.0Mi        100.0Mi                       
  │  ├─ file-pod                                                               256.0Mi          1.0Gi                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-gc-767f4bbfwdphf        128.0Mi          1.0Gi                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-lv7bq             64.0Mi        512.0Mi                       
  │  ├─ kube-router-zhxjw                                                      250.0Mi            0.0                       
  │  ├─ ubuntu-test-pod                                                        512.0Mi          4.0Gi                       
  │  ├─ urs-waldmann-tb-access-pod                                              64.0Gi        160.0Gi                       
  │  ├─ valentin-schmuker-storage                                                2.0Gi          2.0Gi                       
  │  ├─ virt-handler-tqrj4                                                     325.0Mi            0.0                       
  │  └─ virt-launcher-lightfield-analysis-8zbf6                                   4.6G          60.0M                       
  ├─ mindflayer02                                                         (0%) 706.0Mi     (0%) 1.6Gi      376.5Gi  374.9Gi 
  │  ├─ dnsutils-mindflayer02                                                  100.0Mi        100.0Mi                       
  │  ├─ gatekeeper-audit-59d4b6fd4c-gtwjm                                      256.0Mi        512.0Mi                       
  │  ├─ kube-router-zc9q4                                                      250.0Mi            0.0                       
  │  └─ ubuntu-test-pod                                                        100.0Mi          1.0Gi                       
  ├─ mindflayer03                                                         (20%) 74.9Gi  (45%) 170.6Gi      376.5Gi  205.9Gi 
  │  ├─ cdi-operator-76f7d8c545-vrql7                                          150.0Mi            0.0                       
  │  ├─ dnsutils-mindflayer03                                                  100.0Mi        100.0Mi                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-vlfn5             64.0Mi        512.0Mi                       
  │  ├─ gpu-pod-aalbi                                                           10.0Gi         10.0Gi                       
  │  ├─ kube-router-t8kl8                                                      250.0Mi            0.0                       
  │  ├─ urs-waldmann-ubuntu-pod                                                 64.0Gi        160.0Gi                       
  │  └─ virt-handler-hndfm                                                     325.0Mi            0.0                       
  ├─ tiamat                                                              (28%) 280.4Gi  (43%) 430.6Gi     1007.6Gi  577.0Gi 
  │  ├─ dnsutils-tiamat                                                        100.0Mi        100.0Mi                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-jg9c7             64.0Mi        512.0Mi                       
  │  ├─ kube-router-9m4q5                                                      250.0Mi            0.0                       
  │  ├─ train-baseline-full-dataset-filtered-split-r8hdq                        30.0Gi         30.0Gi                       
  │  └─ ubuntu-gpu1                                                            250.0Gi        400.0Gi                       
  ├─ vecna                                                                 (5%) 82.4Gi    (5%) 82.6Gi        1.5Ti    1.4Ti 
  │  ├─ bee-finetune-vecna-terminal                                             32.0Gi         32.0Gi                       
  │  ├─ dnsutils-vecna                                                         100.0Mi        100.0Mi                       
  │  ├─ felix-petersen-job-20                                                   50.0Gi         50.0Gi                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-mvz86             64.0Mi        512.0Mi                       
  │  └─ kube-router-zg4xh                                                      250.0Mi            0.0                       
  └─ zariel                                                                (98%) 1.9Ti    (99%) 2.0Ti        2.0Ti   14.9Gi 
     ├─ allin1                                                                  64.0Gi         80.0Gi                       
     ├─ dnsutils-zariel                                                        100.0Mi        100.0Mi                       
     ├─ gpu-operator-1777509885-node-feature-discovery-worker-5fsqh             64.0Mi        512.0Mi                       
     ├─ kube-router-cqsw4                                                      250.0Mi            0.0                       
     └─ zariel-a2v2-shark-julian-schaefer-zimmermann                             1.9Ti          1.9Ti                       
  nvidia.com/gpu                                                            (45%) 28.0     (45%) 28.0         62.0     34.0 
  ├─ asmodeus                                                               (100%) 4.0     (100%) 4.0          4.0      0.0 
  │  ├─ step16-coolchic-gpu-run-asmodeus-a100-hq                                   1.0            1.0                       
  │  ├─ step18-ftic-gpu-run-asmodeus-a100-hq-40cpu                                 1.0            1.0                       
  │  ├─ step19-deep-iqa-models-gpu-run-asmodeus-a100                               1.0            1.0                       
  │  └─ ulas-bingoel-model-pod-5                                                   1.0            1.0                       
  ├─ belial                                                                  (25%) 2.0      (25%) 2.0          8.0      6.0 
  │  ├─ ledavio-text-search-5d8f755795-tqp4s                                       1.0            1.0                       
  │  └─ ollama-pod                                                                 1.0            1.0                       
  ├─ demogorgon                                                              (88%) 7.0      (88%) 7.0          8.0      1.0 
  │  ├─ coolchic5-gpu-run-demogorgon-a100-32cpu-3gpu-2                             3.0            3.0                       
  │  ├─ felix-petersen-job-29                                                      1.0            1.0                       
  │  ├─ gpu-demogorgon                                                             1.0            1.0                       
  │  ├─ pycharm                                                                    1.0            1.0                       
  │  └─ pycharmv2                                                                  1.0            1.0                       
  ├─ fierna                                                                  (12%) 1.0      (12%) 1.0          8.0      7.0 
  │  └─ ledavio-similarity-search-9b864cc89-ln9lk                                  1.0            1.0                       
  ├─ kiaransalee                                                             (33%) 2.0      (33%) 2.0          6.0      4.0 
  │  ├─ jupyter-giordano-2ddemarzo                                                 1.0            1.0                       
  │  └─ jupyter-saroyehun                                                          1.0            1.0                       
  ├─ tiamat                                                                  (50%) 2.0      (50%) 2.0          4.0      2.0 
  │  ├─ train-baseline-full-dataset-filtered-split-r8hdq                           1.0            1.0                       
  │  └─ ubuntu-gpu1                                                                1.0            1.0                       
  ├─ vecna                                                                   (12%) 2.0      (12%) 2.0         16.0     14.0 
  │  ├─ bee-finetune-vecna-terminal                                                1.0            1.0                       
  │  └─ felix-petersen-job-20                                                      1.0            1.0                       
  └─ zariel                                                                 (100%) 8.0     (100%) 8.0          8.0      0.0 
     └─ zariel-a2v2-shark-julian-schaefer-zimmermann                               8.0            8.0                       
  nvidia.com/mig-3g.40gb                                                      (0%) 0.0       (0%) 0.0          2.0      2.0 
  └─ kiaransalee                                                              (0%) 0.0       (0%) 0.0          2.0      2.0 
  nvidia.com/mig-4g.40gb                                                     (50%) 1.0      (50%) 1.0          2.0      1.0 
  └─ kiaransalee                                                             (50%) 1.0      (50%) 1.0          2.0      1.0 
     └─ jupyter-elenasolar                                                         1.0            1.0                       
  pods                                                                     (14%) 218.0    (14%) 218.0         1.5k     1.3k 
  ├─ asmodeus                                                               (15%) 16.0     (15%) 16.0        110.0     94.0 
  │  ├─ dnsutils-asmodeus                                                          1.0            1.0                       
  │  ├─ gpu-a100-zsh-shm                                                           1.0            1.0                       
  │  ├─ gpu-feature-discovery-jgxzx                                                1.0            1.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-9q5ls                1.0            1.0                       
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-qdkvw            1.0            1.0                       
  │  ├─ kube-proxy-kxtvf                                                           1.0            1.0                       
  │  ├─ kube-router-9q4k5                                                          1.0            1.0                       
  │  ├─ nvidia-container-toolkit-daemonset-866wd                                   1.0            1.0                       
  │  ├─ nvidia-dcgm-exporter-smxtb                                                 1.0            1.0                       
  │  ├─ nvidia-device-plugin-daemonset-xjtg9                                       1.0            1.0                       
  │  ├─ nvidia-mig-manager-56kzk                                                   1.0            1.0                       
  │  ├─ nvidia-operator-validator-pz657                                            1.0            1.0                       
  │  ├─ step16-coolchic-gpu-run-asmodeus-a100-hq                                   1.0            1.0                       
  │  ├─ step18-ftic-gpu-run-asmodeus-a100-hq-40cpu                                 1.0            1.0                       
  │  ├─ step19-deep-iqa-models-gpu-run-asmodeus-a100                               1.0            1.0                       
  │  └─ ulas-bingoel-model-pod-5                                                   1.0            1.0                       
  ├─ beholder01                                                               (7%) 8.0       (7%) 8.0        110.0    102.0 
  │  ├─ dnsutils-beholder01                                                        1.0            1.0                       
  │  ├─ kube-apiserver-beholder01                                                  1.0            1.0                       
  │  ├─ kube-controller-manager-beholder01                                         1.0            1.0                       
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-g8bwl            1.0            1.0                       
  │  ├─ kube-proxy-jw87r                                                           1.0            1.0                       
  │  ├─ kube-router-6bz7x                                                          1.0            1.0                       
  │  ├─ kube-scheduler-beholder01                                                  1.0            1.0                       
  │  └─ vm-proxy-frontend-766dd9b967-swbkr                                         1.0            1.0                       
  ├─ beholder02                                                             (14%) 15.0     (14%) 15.0        110.0     95.0 
  │  ├─ dnsutils-beholder02                                                        1.0            1.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-8rfmd                1.0            1.0                       
  │  ├─ hub-848f5d5578-47s2j                                                       1.0            1.0                       
  │  ├─ kube-apiserver-beholder02                                                  1.0            1.0                       
  │  ├─ kube-controller-manager-beholder02                                         1.0            1.0                       
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-pgkbs            1.0            1.0                       
  │  ├─ kube-proxy-cmzf2                                                           1.0            1.0                       
  │  ├─ kube-router-hgnjl                                                          1.0            1.0                       
  │  ├─ kube-scheduler-beholder02                                                  1.0            1.0                       
  │  ├─ mc-rj-ddcb7fb6-b2qh6                                                       1.0            1.0                       
  │  ├─ nginx-ip-2025-7fd66b99dd-2khh6                                             1.0            1.0                       
  │  ├─ nginx-rsn-2024-57d49484d-47rfc                                             1.0            1.0                       
  │  ├─ virt-api-85578d9bb7-5fwdl                                                  1.0            1.0                       
  │  ├─ virt-controller-674bcccb6-pvgxj                                            1.0            1.0                       
  │  └─ virt-operator-b4d8f7f58-6jjxp                                              1.0            1.0                       
  ├─ beholder03                                                             (15%) 16.0     (15%) 16.0        110.0     94.0 
  │  ├─ coredns-66bc5c9577-d6czh                                                   1.0            1.0                       
  │  ├─ dnsutils-beholder03                                                        1.0            1.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-master-66685h4zx            1.0            1.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-8jfkj                1.0            1.0                       
  │  ├─ gpu-operator-f96dbb6fc-nk2jk                                               1.0            1.0                       
  │  ├─ kube-apiserver-beholder03                                                  1.0            1.0                       
  │  ├─ kube-controller-manager-beholder03                                         1.0            1.0                       
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-jrpqw            1.0            1.0                       
  │  ├─ kube-proxy-cftmz                                                           1.0            1.0                       
  │  ├─ kube-router-2dszl                                                          1.0            1.0                       
  │  ├─ kube-scheduler-beholder03                                                  1.0            1.0                       
  │  ├─ ldap-67b47cf9b9-v6vvm                                                      1.0            1.0                       
  │  ├─ memcached-6b68cdd947-w4k2q                                                 1.0            1.0                       
  │  ├─ virt-api-85578d9bb7-6mvxx                                                  1.0            1.0                       
  │  ├─ virt-controller-674bcccb6-f2xfj                                            1.0            1.0                       
  │  └─ virt-operator-b4d8f7f58-pnbkz                                              1.0            1.0                       
  ├─ belial                                                                 (15%) 17.0     (15%) 17.0        110.0     93.0 
  │  ├─ coredns-66bc5c9577-kxlwp                                                   1.0            1.0                       
  │  ├─ dnsutils-belial                                                            1.0            1.0                       
  │  ├─ gatekeeper-controller-manager-66f474f785-bq2vs                             1.0            1.0                       
  │  ├─ gpu-feature-discovery-s2tzn                                                1.0            1.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-j7bdx                1.0            1.0                       
  │  ├─ kube-proxy-xxgbc                                                           1.0            1.0                       
  │  ├─ kube-router-b8htr                                                          1.0            1.0                       
  │  ├─ ledavio-text-search-5d8f755795-tqp4s                                       1.0            1.0                       
  │  ├─ nvidia-container-toolkit-daemonset-bt92r                                   1.0            1.0                       
  │  ├─ nvidia-dcgm-exporter-rtjj5                                                 1.0            1.0                       
  │  ├─ nvidia-device-plugin-daemonset-j96xt                                       1.0            1.0                       
  │  ├─ nvidia-operator-validator-9gr7w                                            1.0            1.0                       
  │  ├─ ollama-pod                                                                 1.0            1.0                       
  │  ├─ storage-pod                                                                1.0            1.0                       
  │  ├─ transfer-aalbi                                                             1.0            1.0                       
  │  ├─ virt-handler-rvzhr                                                         1.0            1.0                       
  │  └─ whoami-74dc54d675-d6p8r                                                    1.0            1.0                       
  ├─ demogorgon                                                             (14%) 15.0     (14%) 15.0        110.0     95.0 
  │  ├─ coolchic5-gpu-run-demogorgon-a100-32cpu-3gpu-2                             1.0            1.0                       
  │  ├─ dnsutils-demogorgon                                                        1.0            1.0                       
  │  ├─ felix-petersen-job-29                                                      1.0            1.0                       
  │  ├─ gpu-demogorgon                                                             1.0            1.0                       
  │  ├─ gpu-feature-discovery-mqtsq                                                1.0            1.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-pxws7                1.0            1.0                       
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-mdrd4            1.0            1.0                       
  │  ├─ kube-proxy-xdkh7                                                           1.0            1.0                       
  │  ├─ kube-router-5g7fj                                                          1.0            1.0                       
  │  ├─ nvidia-container-toolkit-daemonset-t2bv4                                   1.0            1.0                       
  │  ├─ nvidia-dcgm-exporter-v8zxc                                                 1.0            1.0                       
  │  ├─ nvidia-device-plugin-daemonset-fq2lz                                       1.0            1.0                       
  │  ├─ nvidia-operator-validator-qww8n                                            1.0            1.0                       
  │  ├─ pycharm                                                                    1.0            1.0                       
  │  └─ pycharmv2                                                                  1.0            1.0                       
  ├─ fierna                                                                 (16%) 18.0     (16%) 18.0        110.0     92.0 
  │  ├─ dnsutils-fierna                                                            1.0            1.0                       
  │  ├─ gatekeeper-controller-manager-66f474f785-dvhjf                             1.0            1.0                       
  │  ├─ gatekeeper-controller-manager-66f474f785-s9kgj                             1.0            1.0                       
  │  ├─ gpu-feature-discovery-sspcw                                                1.0            1.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-ldxbm                1.0            1.0                       
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-8f4nd            1.0            1.0                       
  │  ├─ kube-proxy-pgw9t                                                           1.0            1.0                       
  │  ├─ kube-router-8g7hh                                                          1.0            1.0                       
  │  ├─ ledavio-similarity-search-9b864cc89-ln9lk                                  1.0            1.0                       
  │  ├─ nvidia-container-toolkit-daemonset-rbvmw                                   1.0            1.0                       
  │  ├─ nvidia-dcgm-exporter-fblhj                                                 1.0            1.0                       
  │  ├─ nvidia-device-plugin-daemonset-vsr66                                       1.0            1.0                       
  │  ├─ nvidia-operator-validator-8r84t                                            1.0            1.0                       
  │  ├─ proxy-5495d795d5-jjk7j                                                     1.0            1.0                       
  │  ├─ till-aust-ubuntu-entry-pod                                                 1.0            1.0                       
  │  ├─ user-scheduler-5cf5ffbc54-vjlq8                                            1.0            1.0                       
  │  ├─ virt-handler-xmxj8                                                         1.0            1.0                       
  │  └─ whoami-74dc54d675-vsljt                                                    1.0            1.0                       
  ├─ kiaransalee                                                            (15%) 16.0     (15%) 16.0        110.0     94.0 
  │  ├─ continuous-image-puller-6fs4k                                              1.0            1.0                       
  │  ├─ continuous-image-puller-6z8bj                                              1.0            1.0                       
  │  ├─ dnsutils-kiaransalee                                                       1.0            1.0                       
  │  ├─ gpu-feature-discovery-52lgs                                                1.0            1.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-jn8tv                1.0            1.0                       
  │  ├─ jupyter-elenasolar                                                         1.0            1.0                       
  │  ├─ jupyter-giordano-2ddemarzo                                                 1.0            1.0                       
  │  ├─ jupyter-saroyehun                                                          1.0            1.0                       
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-xc2hx            1.0            1.0                       
  │  ├─ kube-proxy-l8zn4                                                           1.0            1.0                       
  │  ├─ kube-router-f8tf4                                                          1.0            1.0                       
  │  ├─ nvidia-container-toolkit-daemonset-p2dgh                                   1.0            1.0                       
  │  ├─ nvidia-dcgm-exporter-xhdc4                                                 1.0            1.0                       
  │  ├─ nvidia-device-plugin-daemonset-wl2gh                                       1.0            1.0                       
  │  ├─ nvidia-mig-manager-5rrnr                                                   1.0            1.0                       
  │  └─ nvidia-operator-validator-99cc8                                            1.0            1.0                       
  ├─ mindflayer01                                                           (23%) 25.0     (23%) 25.0        110.0     85.0 
  │  ├─ alertmanager-kube-prometheus-stack-1777-alertmanager-0                     1.0            1.0                       
  │  ├─ cdi-apiserver-7745487599-s5g6s                                             1.0            1.0                       
  │  ├─ cdi-deployment-6c99bb8fcf-h79j8                                            1.0            1.0                       
  │  ├─ cdi-uploadproxy-684cf5d896-9x5rh                                           1.0            1.0                       
  │  ├─ dnsutils-mindflayer01                                                      1.0            1.0                       
  │  ├─ file-pod                                                                   1.0            1.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-gc-767f4bbfwdphf            1.0            1.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-lv7bq                1.0            1.0                       
  │  ├─ hub-78d6dd898d-hb67q                                                       1.0            1.0                       
  │  ├─ kube-prometheus-stack-1777-operator-6bcf46cfd5-2j6lk                       1.0            1.0                       
  │  ├─ kube-prometheus-stack-1777533213-grafana-6ccc5bf477-mw65j                  1.0            1.0                       
  │  ├─ kube-prometheus-stack-1777533213-kube-state-metrics-7f89b5fsvf6            1.0            1.0                       
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-f9z45            1.0            1.0                       
  │  ├─ kube-proxy-49vx2                                                           1.0            1.0                       
  │  ├─ kube-router-zhxjw                                                          1.0            1.0                       
  │  ├─ prometheus-kube-prometheus-stack-1777-prometheus-0                         1.0            1.0                       
  │  ├─ proxy-5bc89cc587-srb6x                                                     1.0            1.0                       
  │  ├─ proxy-7f79cc645f-mgpsq                                                     1.0            1.0                       
  │  ├─ ubuntu-test-pod                                                            1.0            1.0                       
  │  ├─ urs-waldmann-tb-access-pod                                                 1.0            1.0                       
  │  ├─ user-scheduler-5cf5ffbc54-n9qgd                                            1.0            1.0                       
  │  ├─ user-scheduler-c7db6c584-rs664                                             1.0            1.0                       
  │  ├─ valentin-schmuker-storage                                                  1.0            1.0                       
  │  ├─ virt-handler-tqrj4                                                         1.0            1.0                       
  │  └─ virt-launcher-lightfield-analysis-8zbf6                                    1.0            1.0                       
  ├─ mindflayer02                                                           (22%) 24.0     (22%) 24.0        110.0     86.0 
  │  ├─ cert-manager-79559475b4-7kv54                                              1.0            1.0                       
  │  ├─ cert-manager-cainjector-966fc8fbc-zql8j                                    1.0            1.0                       
  │  ├─ cert-manager-webhook-854cf5f458-wwf4d                                      1.0            1.0                       
  │  ├─ dex-586d7ff58f-vpklf                                                       1.0            1.0                       
  │  ├─ dex-loginapp-6d47648b8b-p7v2x                                              1.0            1.0                       
  │  ├─ dex-mysql-589f4586bc-4n5vj                                                 1.0            1.0                       
  │  ├─ dnsutils-mindflayer02                                                      1.0            1.0                       
  │  ├─ gatekeeper-audit-59d4b6fd4c-gtwjm                                          1.0            1.0                       
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-qz6sb            1.0            1.0                       
  │  ├─ kube-proxy-dfgxf                                                           1.0            1.0                       
  │  ├─ kube-router-zc9q4                                                          1.0            1.0                       
  │  ├─ local-path-provisioner-759479454f-7pqw8                                    1.0            1.0                       
  │  ├─ mediawiki-77f9c84df5-p6k9g                                                 1.0            1.0                       
  │  ├─ mediawiki-mariadb-7ffb6c9b8d-ng7s6                                         1.0            1.0                       
  │  ├─ nginx-k8s-5889449f8b-dv6xq                                                 1.0            1.0                       
  │  ├─ nginx-rec-2026-75dd946d4d-df9gc                                            1.0            1.0                       
  │  ├─ nginx-self-service-password-54767ddc56-d556f                               1.0            1.0                       
  │  ├─ pdf-55ccd6f459-s7nbj                                                       1.0            1.0                       
  │  ├─ registry-74dfd5b88d-dvkg8                                                  1.0            1.0                       
  │  ├─ registry-auth-86fdd9f85-k7nr2                                              1.0            1.0                       
  │  ├─ registry-browser-7f4cbdf96b-d4b66                                          1.0            1.0                       
  │  ├─ traefik-deployment-d8ccbfdd4-8xqsn                                         1.0            1.0                       
  │  ├─ ubuntu-test-pod                                                            1.0            1.0                       
  │  └─ user-scheduler-c7db6c584-2pxhd                                             1.0            1.0                       
  ├─ mindflayer03                                                             (8%) 9.0       (8%) 9.0        110.0    101.0 
  │  ├─ cdi-operator-76f7d8c545-vrql7                                              1.0            1.0                       
  │  ├─ dnsutils-mindflayer03                                                      1.0            1.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-vlfn5                1.0            1.0                       
  │  ├─ gpu-pod-aalbi                                                              1.0            1.0                       
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-fmsql            1.0            1.0                       
  │  ├─ kube-proxy-d2lmv                                                           1.0            1.0                       
  │  ├─ kube-router-t8kl8                                                          1.0            1.0                       
  │  ├─ urs-waldmann-ubuntu-pod                                                    1.0            1.0                       
  │  └─ virt-handler-hndfm                                                         1.0            1.0                       
  ├─ tiamat                                                                 (13%) 14.0     (13%) 14.0        110.0     96.0 
  │  ├─ continuous-image-puller-wtkhd                                              1.0            1.0                       
  │  ├─ dnsutils-tiamat                                                            1.0            1.0                       
  │  ├─ gpu-feature-discovery-ggqsw                                                1.0            1.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-jg9c7                1.0            1.0                       
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-zwnjs            1.0            1.0                       
  │  ├─ kube-proxy-n8m88                                                           1.0            1.0                       
  │  ├─ kube-router-9m4q5                                                          1.0            1.0                       
  │  ├─ nvidia-container-toolkit-daemonset-7nh8q                                   1.0            1.0                       
  │  ├─ nvidia-dcgm-exporter-8hbwd                                                 1.0            1.0                       
  │  ├─ nvidia-device-plugin-daemonset-gbv2h                                       1.0            1.0                       
  │  ├─ nvidia-mig-manager-475gc                                                   1.0            1.0                       
  │  ├─ nvidia-operator-validator-mjbvb                                            1.0            1.0                       
  │  ├─ train-baseline-full-dataset-filtered-split-r8hdq                           1.0            1.0                       
  │  └─ ubuntu-gpu1                                                                1.0            1.0                       
  ├─ vecna                                                                  (11%) 12.0     (11%) 12.0        110.0     98.0 
  │  ├─ bee-finetune-vecna-terminal                                                1.0            1.0                       
  │  ├─ dnsutils-vecna                                                             1.0            1.0                       
  │  ├─ felix-petersen-job-20                                                      1.0            1.0                       
  │  ├─ gpu-feature-discovery-dg72r                                                1.0            1.0                       
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-mvz86                1.0            1.0                       
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-5d5rl            1.0            1.0                       
  │  ├─ kube-proxy-6r98t                                                           1.0            1.0                       
  │  ├─ kube-router-zg4xh                                                          1.0            1.0                       
  │  ├─ nvidia-container-toolkit-daemonset-l7ddn                                   1.0            1.0                       
  │  ├─ nvidia-dcgm-exporter-774vj                                                 1.0            1.0                       
  │  ├─ nvidia-device-plugin-daemonset-mkhc4                                       1.0            1.0                       
  │  └─ nvidia-operator-validator-4jkmc                                            1.0            1.0                       
  └─ zariel                                                                 (12%) 13.0     (12%) 13.0        110.0     97.0 
     ├─ allin1                                                                     1.0            1.0                       
     ├─ dnsutils-zariel                                                            1.0            1.0                       
     ├─ gpu-feature-discovery-ctcnd                                                1.0            1.0                       
     ├─ gpu-operator-1777509885-node-feature-discovery-worker-5fsqh                1.0            1.0                       
     ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-wdj8l            1.0            1.0                       
     ├─ kube-proxy-gsqm7                                                           1.0            1.0                       
     ├─ kube-router-cqsw4                                                          1.0            1.0                       
     ├─ nvidia-container-toolkit-daemonset-98jbp                                   1.0            1.0                       
     ├─ nvidia-dcgm-exporter-55cl4                                                 1.0            1.0                       
     ├─ nvidia-device-plugin-daemonset-clcv7                                       1.0            1.0                       
     ├─ nvidia-mig-manager-jll8z                                                   1.0            1.0                       
     ├─ nvidia-operator-validator-8mf54                                            1.0            1.0                       
     └─ zariel-a2v2-shark-julian-schaefer-zimmermann                               1.0            1.0                       




Resource usage by namespace

 Resource                              Requested    Limit  Allocatable  Free 
  auth                                                                       
  ├─ beholder03                                                              
  │  ├─ cpu                               250.0m      0.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer02                                                            
     ├─ cpu                               450.0m      0.0                    
     └─ pods                                 4.0      4.0                    
  cdi                                                                        
  ├─ mindflayer01                                                            
  │  ├─ cpu                                  2.3    15.0m                    
  │  ├─ devices.kubevirt.io/kvm              1.0      1.0                    
  │  ├─ devices.kubevirt.io/tun              1.0      1.0                    
  │  ├─ devices.kubevirt.io/vhost-net        1.0      1.0                    
  │  ├─ ephemeral-storage                  50.0M      0.0                    
  │  ├─ memory                              5.1G    60.0M                    
  │  └─ pods                                 4.0      4.0                    
  └─ mindflayer03                                                            
     ├─ cpu                               100.0m      0.0                    
     ├─ memory                           150.0Mi      0.0                    
     └─ pods                                 1.0      1.0                    
  cert-manager                               3.0      3.0                    
  └─ mindflayer02                            3.0      3.0                    
     └─ pods                                 3.0      3.0                    
  gatekeeper-system                                                          
  ├─ belial                                                                  
  │  ├─ cpu                               100.0m      1.0                    
  │  ├─ memory                           256.0Mi  512.0Mi                    
  │  └─ pods                                 1.0      1.0                    
  ├─ fierna                                                                  
  │  ├─ cpu                               200.0m      2.0                    
  │  ├─ memory                           512.0Mi    1.0Gi                    
  │  └─ pods                                 2.0      2.0                    
  └─ mindflayer02                                                            
     ├─ cpu                               100.0m      1.0                    
     ├─ memory                           256.0Mi  512.0Mi                    
     └─ pods                                 1.0      1.0                    
  gpu-operator                                                               
  ├─ asmodeus                                                                
  │  ├─ cpu                                 5.0m      0.0                    
  │  ├─ memory                            64.0Mi  512.0Mi                    
  │  └─ pods                                 7.0      7.0                    
  ├─ beholder02                                                              
  │  ├─ cpu                                 5.0m      0.0                    
  │  ├─ memory                            64.0Mi  512.0Mi                    
  │  └─ pods                                 1.0      1.0                    
  ├─ beholder03                                                              
  │  ├─ cpu                               305.0m   500.0m                    
  │  ├─ memory                           292.0Mi    4.8Gi                    
  │  └─ pods                                 3.0      3.0                    
  ├─ belial                                                                  
  │  ├─ cpu                                 5.0m      0.0                    
  │  ├─ memory                            64.0Mi  512.0Mi                    
  │  └─ pods                                 6.0      6.0                    
  ├─ demogorgon                                                              
  │  ├─ cpu                                 5.0m      0.0                    
  │  ├─ memory                            64.0Mi  512.0Mi                    
  │  └─ pods                                 6.0      6.0                    
  ├─ fierna                                                                  
  │  ├─ cpu                                 5.0m      0.0                    
  │  ├─ memory                            64.0Mi  512.0Mi                    
  │  └─ pods                                 6.0      6.0                    
  ├─ kiaransalee                                                             
  │  ├─ cpu                                 5.0m      0.0                    
  │  ├─ memory                            64.0Mi  512.0Mi                    
  │  └─ pods                                 7.0      7.0                    
  ├─ mindflayer01                                                            
  │  ├─ cpu                                15.0m      0.0                    
  │  ├─ memory                           192.0Mi    1.5Gi                    
  │  └─ pods                                 2.0      2.0                    
  ├─ mindflayer03                                                            
  │  ├─ cpu                                 5.0m      0.0                    
  │  ├─ memory                            64.0Mi  512.0Mi                    
  │  └─ pods                                 1.0      1.0                    
  ├─ tiamat                                                                  
  │  ├─ cpu                                 5.0m      0.0                    
  │  ├─ memory                            64.0Mi  512.0Mi                    
  │  └─ pods                                 7.0      7.0                    
  ├─ vecna                                                                   
  │  ├─ cpu                                 5.0m      0.0                    
  │  ├─ memory                            64.0Mi  512.0Mi                    
  │  └─ pods                                 6.0      6.0                    
  └─ zariel                                                                  
     ├─ cpu                                 5.0m      0.0                    
     ├─ memory                            64.0Mi  512.0Mi                    
     └─ pods                                 7.0      7.0                    
  jupyterhub                                                                 
  ├─ beholder02                              1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ kiaransalee                                                             
  │  ├─ memory                              3.2G      0.0                    
  │  ├─ nvidia.com/gpu                       2.0      2.0                    
  │  ├─ nvidia.com/mig-4g.40gb               1.0      1.0                    
  │  └─ pods                                 4.0      4.0                    
  ├─ mindflayer01                            2.0      2.0                    
  │  └─ pods                                 2.0      2.0                    
  └─ mindflayer02                            1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  jupyterhub-kuckling                        2.0      2.0                    
  ├─ fierna                                  1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ tiamat                                  1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  jupyterhub-students                        5.0      5.0                    
  ├─ fierna                                  1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ kiaransalee                             1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer01                            3.0      3.0                    
     └─ pods                                 3.0      3.0                    
  kube-system                                                                
  ├─ asmodeus                                                                
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 3.0      3.0                    
  ├─ beholder01                                                              
  │  ├─ cpu                               900.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 6.0      6.0                    
  ├─ beholder02                                                              
  │  ├─ cpu                               900.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 6.0      6.0                    
  ├─ beholder03                                                              
  │  ├─ cpu                                  1.0   100.0m                    
  │  ├─ memory                           420.0Mi  270.0Mi                    
  │  └─ pods                                 7.0      7.0                    
  ├─ belial                                                                  
  │  ├─ cpu                               450.0m   100.0m                    
  │  ├─ memory                           420.0Mi  270.0Mi                    
  │  └─ pods                                 4.0      4.0                    
  ├─ demogorgon                                                              
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 3.0      3.0                    
  ├─ fierna                                                                  
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 3.0      3.0                    
  ├─ kiaransalee                                                             
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 3.0      3.0                    
  ├─ mindflayer01                                                            
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 3.0      3.0                    
  ├─ mindflayer02                                                            
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 3.0      3.0                    
  ├─ mindflayer03                                                            
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 3.0      3.0                    
  ├─ tiamat                                                                  
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 3.0      3.0                    
  ├─ vecna                                                                   
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 3.0      3.0                    
  └─ zariel                                                                  
     ├─ cpu                               350.0m   100.0m                    
     ├─ memory                           350.0Mi  100.0Mi                    
     └─ pods                                 3.0      3.0                    
  kubevirt                                                                   
  ├─ beholder02                                                              
  │  ├─ cpu                                25.0m      0.0                    
  │  ├─ memory                             1.2Gi      0.0                    
  │  └─ pods                                 3.0      3.0                    
  ├─ beholder03                                                              
  │  ├─ cpu                                25.0m      0.0                    
  │  ├─ memory                             1.2Gi      0.0                    
  │  └─ pods                                 3.0      3.0                    
  ├─ belial                                                                  
  │  ├─ cpu                                10.0m      0.0                    
  │  ├─ memory                           325.0Mi      0.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ fierna                                                                  
  │  ├─ cpu                                10.0m      0.0                    
  │  ├─ memory                           325.0Mi      0.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ mindflayer01                                                            
  │  ├─ cpu                                10.0m      0.0                    
  │  ├─ memory                           325.0Mi      0.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer03                                                            
     ├─ cpu                                10.0m      0.0                    
     ├─ memory                           325.0Mi      0.0                    
     └─ pods                                 1.0      1.0                    
  local-path-storage                         1.0      1.0                    
  └─ mindflayer02                            1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  monitoring                                                                 
  ├─ asmodeus                                1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ beholder01                              1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ beholder02                              1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ beholder03                              1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ demogorgon                              1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ fierna                                  1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ kiaransalee                             1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ mindflayer01                                                            
  │  ├─ memory                           200.0Mi      0.0                    
  │  └─ pods                                 6.0      6.0                    
  ├─ mindflayer02                            1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ mindflayer03                            1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ tiamat                                  1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ vecna                                   1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ zariel                                  1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  registry                                                                   
  ├─ belial                                  1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer02                                                            
     ├─ cpu                               400.0m      0.0                    
     └─ pods                                 3.0      3.0                    
  traefik                                    2.0      2.0                    
  ├─ fierna                                  1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer02                            1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-angela-albi                                                           
  ├─ belial                                                                  
  │  ├─ cpu                                  1.0      2.0                    
  │  ├─ memory                             2.0Gi    4.0Gi                    
  │  └─ pods                                 1.0      1.0                    
  ├─ mindflayer03                                                            
  │  ├─ cpu                                 10.0     10.0                    
  │  ├─ memory                            10.0Gi   10.0Gi                    
  │  └─ pods                                 1.0      1.0                    
  └─ tiamat                                                                  
     ├─ cpu                                  2.0     16.0                    
     ├─ memory                            30.0Gi   30.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-bastian-goldluecke                                                    
  └─ beholder02                                                              
     ├─ cpu                                  2.0      2.0                    
     ├─ memory                             1.0Gi    1.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-christoph-hanselka                                                    
  └─ belial                                                                  
     ├─ cpu                                  1.0      1.0                    
     ├─ memory                            32.0Gi   32.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-eduard-buss                                                           
  └─ demogorgon                                                              
     ├─ cpu                                  2.0     64.0                    
     ├─ ephemeral-storage                 30.0Gi   60.0Gi                    
     ├─ memory                            92.0Gi  100.0Gi                    
     ├─ nvidia.com/gpu                       2.0      2.0                    
     └─ pods                                 2.0      2.0                    
  user-felix-petersen                                                        
  ├─ demogorgon                                                              
  │  ├─ cpu                                  6.0      8.0                    
  │  ├─ memory                            50.0Gi  100.0Gi                    
  │  ├─ nvidia.com/gpu                       1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ vecna                                                                   
     ├─ cpu                                  4.0      4.0                    
     ├─ memory                            50.0Gi   50.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-giovanna-ratini                                                       
  └─ mindflayer02                                                            
     ├─ cpu                               100.0m      1.0                    
     ├─ memory                           100.0Mi    1.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-isaac-breinyn                                                         
  └─ mindflayer01                                                            
     ├─ cpu                               200.0m      1.0                    
     ├─ memory                           256.0Mi    1.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-isabela-sudac                                                         
  └─ belial                                                                  
     ├─ cpu                               100.0m      1.0                    
     ├─ memory                           100.0Mi    1.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-julian-jandeleit                                                      
  └─ demogorgon                                                              
     ├─ cpu                                 48.0     48.0                    
     ├─ memory                            80.0Gi   80.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-julian-zimmermann                                                     
  └─ zariel                                                                  
     ├─ cpu                                250.0    250.0                    
     ├─ memory                             1.9Ti    1.9Ti                    
     ├─ nvidia.com/gpu                       8.0      8.0                    
     └─ pods                                 1.0      1.0                    
  user-mattia-montanari                                                      
  ├─ mindflayer01                                                            
  │  ├─ cpu                               100.0m      1.0                    
  │  ├─ memory                           512.0Mi    4.0Gi                    
  │  └─ pods                                 1.0      1.0                    
  └─ vecna                                                                   
     ├─ cpu                                 16.0     16.0                    
     ├─ memory                            32.0Gi   32.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-mike-battistella                                                      
  ├─ belial                                                                  
  │  ├─ cpu                                  1.0      2.0                    
  │  ├─ memory                            32.0Gi   32.0Gi                    
  │  ├─ nvidia.com/gpu                       1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ fierna                                                                  
     ├─ cpu                                  4.0      4.0                    
     ├─ ephemeral-storage                256.0Mi    1.0Gi                    
     ├─ memory                            32.0Gi   32.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-mohsen-jenadeleh                                                      
  ├─ asmodeus                                                                
  │  ├─ cpu                                160.0    160.0                    
  │  ├─ memory                           400.0Gi  400.0Gi                    
  │  ├─ nvidia.com/gpu                       3.0      3.0                    
  │  └─ pods                                 3.0      3.0                    
  └─ demogorgon                                                              
     ├─ cpu                                 32.0     32.0                    
     ├─ memory                           200.0Gi  200.0Gi                    
     ├─ nvidia.com/gpu                       3.0      3.0                    
     └─ pods                                 1.0      1.0                    
  user-segun-aroyehun                                                        
  └─ tiamat                                                                  
     ├─ cpu                                 25.0     50.0                    
     ├─ ephemeral-storage                100.0Gi  150.0Gi                    
     ├─ memory                           250.0Gi  400.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-sifei-li                                                              
  ├─ asmodeus                                                                
  │  ├─ cpu                                 32.0     32.0                    
  │  ├─ memory                           256.0Gi  284.0Gi                    
  │  └─ pods                                 1.0      1.0                    
  └─ zariel                                                                  
     ├─ cpu                                  4.0      4.0                    
     ├─ memory                            64.0Gi   80.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-till-aust                                                             
  └─ fierna                                                                  
     ├─ cpu                                  1.0      1.0                    
     ├─ memory                           200.0Gi  200.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-ulas-bingoel                                                          
  └─ asmodeus                                                                
     ├─ cpu                                  8.0      8.0                    
     ├─ memory                            80.0Gi   80.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-urs-waldmann                                                          
  ├─ mindflayer01                                                            
  │  ├─ cpu                                 16.0     16.0                    
  │  ├─ memory                            64.0Gi  160.0Gi                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer03                                                            
     ├─ cpu                                 16.0     16.0                    
     ├─ memory                            64.0Gi  160.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-valentin-schmuker                                                     
  └─ mindflayer01                                                            
     ├─ cpu                                  2.0      2.0                    
     ├─ memory                             2.0Gi    2.0Gi                    
     └─ pods                                 1.0      1.0                    
  web                                                                        
  ├─ beholder01                              1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ beholder02                                                              
  │  ├─ cpu                               250.0m      1.0                    
  │  └─ pods                                 2.0      2.0                    
  ├─ beholder03                              1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer02                                                            
     ├─ cpu                               500.0m      0.0                    
     └─ pods                                 5.0      5.0                    




Ceph file system report

  cluster:
    id:     3fee6f38-ba9f-11ec-9328-e188936dcafd
    health: HEALTH_OK
 
  services:
    mon: 5 daemons, quorum beholder03,beholder01,beholder02,mindflayer02,mindflayer03 (age 4M) [leader: beholder03]
    mgr: mindflayer02.ympgrs(active, since 4M), standbys: beholder03.nprqzk, mindflayer03.rzdvrr, beholder02.akktmp, mindflayer01.mkuopd, beholder01.verxwn
    mds: 4/4 daemons up, 2 standby
    osd: 24 osds: 24 up (since 4M), 24 in (since 6M)
 
  data:
    volumes: 1/1 healthy
    pools:   3 pools, 545 pgs
    objects: 88.76M objects, 104 TiB
    usage:   208 TiB used, 74 TiB / 282 TiB avail
    pgs:     543 active+clean
             2   active+clean+scrubbing+deep
 
  io:
    client:   34 MiB/s wr, 0 op/s rd, 13 op/s wr
 
HEALTH_OK




Etcd cluster

+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|     ENDPOINT     |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.1.1:4252 | 7126e7a3a9cc42ca |  3.4.30 |  156 MB |     false |      false |    349248 |  807653464 |          807653464 |        |
| 192.168.1.2:4252 | 39d72894bf6c7600 |  3.4.30 |  156 MB |      true |      false |    349248 |  807653464 |          807653464 |        |
| 192.168.1.3:4252 | bbf4a2b99c3fd692 |  3.4.30 |  156 MB |     false |      false |    349248 |  807653464 |          807653464 |        |
| 192.168.2.1:4252 | 5cb9997dd1c2246b |  3.4.30 |  156 MB |     false |      false |    349248 |  807653464 |          807653464 |        |
| 192.168.2.3:4252 |  cbc1cf89959ea4e |  3.4.30 |  156 MB |     false |      false |    349248 |  807653464 |          807653464 |        |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+




Detailed network health

API and web servers

beholder01
SSH port open yes
Report available yes
External interface up ok
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs
local raid mounted /raid
Test pod responding dnsutils-beholder01
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
beholder02
SSH port open yes
Report available yes
External interface up ok
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs
local raid mounted /raid
Test pod responding dnsutils-beholder02
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
beholder03
SSH port open yes
Report available yes
External interface up ok
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs
local raid mounted /raid
Test pod responding dnsutils-beholder03
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes




Ceph osd nodes

mindflayer01
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs
local raid mounted /raid
Test pod responding dnsutils-mindflayer01
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
mindflayer02
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs
local raid mounted /raid
Test pod responding dnsutils-mindflayer02
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
mindflayer03
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs
local raid mounted /raid
Test pod responding dnsutils-mindflayer03
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes




Compute nodes

vecna
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-vecna
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
kiaransalee
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-kiaransalee
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
belial
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-belial
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
fierna
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-fierna
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
demogorgon
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-demogorgon
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
tiamat
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-tiamat
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
asmodeus
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-asmodeus
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
zariel
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-zariel
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes




nVidia driver and GPU status

dretch

belial

Sat Jul 11 16:53:08 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Quadro RTX 6000                Off |   00000000:1B:00.0 Off |                  Off |
| 33%   33C    P8             22W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  Quadro RTX 6000                Off |   00000000:1C:00.0 Off |                  Off |
| 33%   32C    P8             27W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  Quadro RTX 6000                Off |   00000000:1D:00.0 Off |                  Off |
| 33%   33C    P8             24W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  Quadro RTX 6000                Off |   00000000:1E:00.0 Off |                  Off |
| 33%   34C    P8             22W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  Quadro RTX 6000                Off |   00000000:3D:00.0 Off |                  Off |
| 33%   34C    P8             28W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  Quadro RTX 6000                Off |   00000000:3F:00.0 Off |                  Off |
| 33%   33C    P8             26W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  Quadro RTX 6000                Off |   00000000:40:00.0 Off |                  Off |
| 33%   33C    P8             20W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  Quadro RTX 6000                Off |   00000000:41:00.0 Off |                  Off |
| 33%   34C    P8             20W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

fierna

Sat Jul 11 16:53:11 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Quadro RTX 6000                On  |   00000000:1B:00.0 Off |                  Off |
| 33%   33C    P8             32W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  Quadro RTX 6000                On  |   00000000:1C:00.0 Off |                  Off |
| 33%   32C    P8             29W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  Quadro RTX 6000                On  |   00000000:1D:00.0 Off |                  Off |
| 33%   36C    P8             20W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  Quadro RTX 6000                On  |   00000000:1E:00.0 Off |                  Off |
| 33%   33C    P8             22W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  Quadro RTX 6000                On  |   00000000:3D:00.0 Off |                  Off |
| 33%   33C    P8             31W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  Quadro RTX 6000                On  |   00000000:3F:00.0 Off |                  Off |
| 33%   33C    P8             21W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  Quadro RTX 6000                On  |   00000000:40:00.0 Off |                  Off |
| 33%   34C    P8             30W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  Quadro RTX 6000                On  |   00000000:41:00.0 Off |                  Off |
| 33%   33C    P8             31W /  260W |     786MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    7   N/A  N/A          801557      C   python                                  782MiB |
+-----------------------------------------------------------------------------------------+

tiamat

Sat Jul 11 16:53:13 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100-SXM4-40GB          On  |   00000000:01:00.0 Off |                    0 |
| N/A   28C    P0             53W /  400W |       0MiB /  40960MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A100-SXM4-40GB          On  |   00000000:41:00.0 Off |                    0 |
| N/A   30C    P0             56W /  400W |       0MiB /  40960MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A100-SXM4-40GB          On  |   00000000:81:00.0 Off |                    0 |
| N/A   37C    P0            105W /  400W |   11347MiB /  40960MiB |     14%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A100-SXM4-40GB          On  |   00000000:C1:00.0 Off |                    0 |
| N/A   26C    P0             54W /  400W |       0MiB /  40960MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    2   N/A  N/A         1840586      C   python                                11336MiB |
+-----------------------------------------------------------------------------------------+

vecna

Sat Jul 11 18:53:16 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla V100-SXM3-32GB           Off |   00000000:34:00.0 Off |                    0 |
| N/A   34C    P0             48W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  Tesla V100-SXM3-32GB           Off |   00000000:36:00.0 Off |                    0 |
| N/A   34C    P0             48W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  Tesla V100-SXM3-32GB           Off |   00000000:39:00.0 Off |                    0 |
| N/A   36C    P0             54W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  Tesla V100-SXM3-32GB           Off |   00000000:3B:00.0 Off |                    0 |
| N/A   37C    P0             51W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  Tesla V100-SXM3-32GB           Off |   00000000:57:00.0 Off |                    0 |
| N/A   34C    P0             48W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  Tesla V100-SXM3-32GB           Off |   00000000:59:00.0 Off |                    0 |
| N/A   37C    P0             53W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  Tesla V100-SXM3-32GB           Off |   00000000:5C:00.0 Off |                    0 |
| N/A   35C    P0             52W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  Tesla V100-SXM3-32GB           Off |   00000000:5E:00.0 Off |                    0 |
| N/A   39C    P0             53W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   8  Tesla V100-SXM3-32GB           Off |   00000000:B7:00.0 Off |                    0 |
| N/A   34C    P0             50W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   9  Tesla V100-SXM3-32GB           Off |   00000000:B9:00.0 Off |                    0 |
| N/A   34C    P0             49W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|  10  Tesla V100-SXM3-32GB           Off |   00000000:BC:00.0 Off |                    0 |
| N/A   36C    P0             51W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|  11  Tesla V100-SXM3-32GB           Off |   00000000:BE:00.0 Off |                    0 |
| N/A   39C    P0             49W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|  12  Tesla V100-SXM3-32GB           Off |   00000000:E0:00.0 Off |                    0 |
| N/A   36C    P0             48W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|  13  Tesla V100-SXM3-32GB           Off |   00000000:E2:00.0 Off |                    0 |
| N/A   36C    P0             49W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|  14  Tesla V100-SXM3-32GB           Off |   00000000:E5:00.0 Off |                    0 |
| N/A   40C    P0             51W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|  15  Tesla V100-SXM3-32GB           Off |   00000000:E7:00.0 Off |                    0 |
| N/A   38C    P0             49W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

asmodeus

Sat Jul 11 16:53:18 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100-SXM4-80GB          On  |   00000000:01:00.0 Off |                    0 |
| N/A   47C    P0            224W /  500W |   15909MiB /  81920MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A100-SXM4-80GB          On  |   00000000:41:00.0 Off |                    0 |
| N/A   53C    P0            325W /  500W |   19016MiB /  81920MiB |     99%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A100-SXM4-80GB          On  |   00000000:81:00.0 Off |                    0 |
| N/A   31C    P0             95W /  500W |     663MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A100-SXM4-80GB          On  |   00000000:C1:00.0 Off |                    0 |
| N/A   27C    P0             64W /  500W |       5MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A          188674      C   ...c-5/coolchic5-venv/bin/python       6498MiB |
|    0   N/A  N/A          193321      C   ...c-5/coolchic5-venv/bin/python       9394MiB |
|    1   N/A  N/A          217439      C   ...c-5/coolchic5-venv/bin/python      18864MiB |
|    1   N/A  N/A          388245      C   python                                   12MiB |
|    1   N/A  N/A          388252      C   python                                   12MiB |
|    2   N/A  N/A         1007577      C   python                                  652MiB |
+-----------------------------------------------------------------------------------------+

zariel

Sat Jul 11 18:53:21 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100-SXM4-40GB          On  |   00000000:07:00.0 Off |                    0 |
| N/A   58C    P0            250W /  400W |   31458MiB /  40960MiB |     95%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A100-SXM4-40GB          On  |   00000000:0F:00.0 Off |                    0 |
| N/A   54C    P0            382W /  400W |   31454MiB /  40960MiB |     89%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A100-SXM4-40GB          On  |   00000000:47:00.0 Off |                    0 |
| N/A   53C    P0            406W /  400W |   31454MiB /  40960MiB |     96%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A100-SXM4-40GB          On  |   00000000:4E:00.0 Off |                    0 |
| N/A   54C    P0            287W /  400W |   31454MiB /  40960MiB |     92%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA A100-SXM4-40GB          On  |   00000000:87:00.0 Off |                    0 |
| N/A   67C    P0            372W /  400W |   31454MiB /  40960MiB |     93%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA A100-SXM4-40GB          On  |   00000000:90:00.0 Off |                    0 |
| N/A   66C    P0            426W /  400W |   31454MiB /  40960MiB |     74%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA A100-SXM4-40GB          On  |   00000000:B7:00.0 Off |                    0 |
| N/A   67C    P0            411W /  400W |   31454MiB /  40960MiB |     93%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA A100-SXM4-40GB          On  |   00000000:BD:00.0 Off |                    0 |
| N/A   63C    P0            414W /  400W |   31454MiB /  40960MiB |     94%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A         1986888      C   /usr/bin/python3                      31444MiB |
|    1   N/A  N/A         1986889      C   /usr/bin/python3                      31440MiB |
|    2   N/A  N/A         1986890      C   /usr/bin/python3                      31440MiB |
|    3   N/A  N/A         1986891      C   /usr/bin/python3                      31440MiB |
|    4   N/A  N/A         1986892      C   /usr/bin/python3                      31440MiB |
|    5   N/A  N/A         1986893      C   /usr/bin/python3                      31440MiB |
|    6   N/A  N/A         1986894      C   /usr/bin/python3                      31440MiB |
|    7   N/A  N/A         1986895      C   /usr/bin/python3                      31440MiB |
+-----------------------------------------------------------------------------------------+

demogorgon

Sat Jul 11 16:53:24 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A40                     Off |   00000000:01:00.0 Off |                    0 |
|  0%   32C    P8             35W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A40                     Off |   00000000:25:00.0 Off |                    0 |
|  0%   32C    P8             35W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A40                     Off |   00000000:41:00.0 Off |                    0 |
|  0%   32C    P8             34W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A40                     Off |   00000000:61:00.0 Off |                    0 |
|  0%   31C    P8             35W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA A40                     Off |   00000000:81:00.0 Off |                    0 |
|  0%   33C    P8             36W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA A40                     Off |   00000000:A1:00.0 Off |                    0 |
|  0%   32C    P8             35W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA A40                     Off |   00000000:C1:00.0 Off |                    0 |
|  0%   34C    P8             36W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA A40                     Off |   00000000:E1:00.0 Off |                    0 |
|  0%   33C    P8             34W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

kiaransalee

Sat Jul 11 16:53:27 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.08             Driver Version: 550.127.08     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:26:00.0 Off |                    0 |
| N/A   29C    P0             77W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H100 80GB HBM3          On  |   00000000:2F:00.0 Off |                    0 |
| N/A   35C    P0             81W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA H100 80GB HBM3          On  |   00000000:46:00.0 Off |                    0 |
| N/A   35C    P0             77W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA H100 80GB HBM3          On  |   00000000:54:00.0 Off |                    0 |
| N/A   31C    P0             80W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA H100 80GB HBM3          On  |   00000000:A6:00.0 Off |                    0 |
| N/A   50C    P0             81W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA H100 80GB HBM3          On  |   00000000:AF:00.0 Off |                    0 |
| N/A   35C    P0             81W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA H100 80GB HBM3          On  |   00000000:C6:00.0 Off |                   On |
| N/A   52C    P0            322W /  700W |   36268MiB /  81559MiB |     N/A      Default |
|                                         |                        |              Enabled |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA H100 80GB HBM3          On  |   00000000:CF:00.0 Off |                   On |
| N/A   28C    P0             77W /  700W |      89MiB /  81559MiB |     N/A      Default |
|                                         |                        |              Enabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| MIG devices:                                                                            |
+------------------+----------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |                     Memory-Usage |        Vol|      Shared           |
|      ID  ID  Dev |                       BAR1-Usage | SM     Unc| CE ENC DEC OFA JPG    |
|                  |                                  |        ECC|                       |
|==================+==================================+===========+=======================|
|  6    1   0   0  |           36230MiB / 40320MiB    | 64      0 |  4   0    4    0    4 |
|                  |                 3MiB / 65535MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  6    2   0   1  |              38MiB / 40320MiB    | 60      0 |  3   0    3    0    3 |
|                  |                 0MiB / 65535MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  7    1   0   0  |              38MiB / 40320MiB    | 60      0 |  3   0    3    0    3 |
|                  |                 0MiB / 65535MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  7    2   0   1  |              51MiB / 40320MiB    | 64      0 |  4   0    4    0    4 |
|                  |                 0MiB / 65535MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    6    1    0    3654249      C   /opt/conda/bin/python                       36170MiB |
+-----------------------------------------------------------------------------------------+