Cluster status Thu, 21 May 2026 15:32:01 +0000 report from beholder01

Resource usage (overall)
Resource usage (by namespace)
Ceph file system status
Etcd cluster status
Detailed network health report
nVidia Driver and GPU reports
  Imp
  Dretch
  Belial
  Fierna
  Tiamat
  Vecna
  Asmodeus
  Zariel
  Demogorgon

Resource usage (overall)

 Resource                                                                    Requested          Limit  Allocatable      Free 
  cpu                                                                      (43%) 680.4    (49%) 774.9         1.6k     801.1 
  ├─ asmodeus                                                              (89%) 228.4    (89%) 228.1        256.0      27.6 
  │  ├─ asmodeus-single-gpu-julian-schaefer-zimmermann                           128.0          128.0                        
  │  ├─ dnsutils-asmodeus                                                       100.0m         100.0m                        
  │  ├─ gpu-a100-zsh-shm                                                          32.0           32.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-9q5ls               5.0m            0.0                        
  │  ├─ kube-router-qbh4m                                                       250.0m            0.0                        
  │  ├─ step16-coolchic-gpu-run-asmodeus-a100-hq                                  20.0           20.0                        
  │  ├─ step18-ftic-gpu-run-asmodeus-a100-hq-40cpu                                40.0           40.0                        
  │  └─ ulas-bingoel-model-pod-5                                                   8.0            8.0                        
  ├─ beholder01                                                            (4%) 900.0m    (0%) 100.0m         24.0      23.1 
  │  ├─ dnsutils-beholder01                                                     100.0m         100.0m                        
  │  ├─ kube-apiserver-beholder01                                               250.0m            0.0                        
  │  ├─ kube-controller-manager-beholder01                                      200.0m            0.0                        
  │  ├─ kube-router-4x8js                                                       250.0m            0.0                        
  │  └─ kube-scheduler-beholder01                                               100.0m            0.0                        
  ├─ beholder02                                                              (13%) 3.2      (13%) 3.1         24.0      20.8 
  │  ├─ dnsutils-beholder02                                                     100.0m         100.0m                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-8rfmd               5.0m            0.0                        
  │  ├─ kube-apiserver-beholder02                                               250.0m            0.0                        
  │  ├─ kube-controller-manager-beholder02                                      200.0m            0.0                        
  │  ├─ kube-router-2dtfp                                                       250.0m            0.0                        
  │  ├─ kube-scheduler-beholder02                                               100.0m            0.0                        
  │  ├─ mc-rj-ddcb7fb6-b2qh6                                                       2.0            2.0                        
  │  ├─ nginx-rsn-2024-57d49484d-47rfc                                          250.0m            1.0                        
  │  ├─ virt-api-85578d9bb7-5fwdl                                                 5.0m            0.0                        
  │  ├─ virt-controller-674bcccb6-pvgxj                                          10.0m            0.0                        
  │  └─ virt-operator-b4d8f7f58-6jjxp                                            10.0m            0.0                        
  ├─ beholder03                                                               (7%) 1.6    (2%) 600.0m         24.0      22.4 
  │  ├─ coredns-66bc5c9577-d6czh                                                100.0m            0.0                        
  │  ├─ dnsutils-beholder03                                                     100.0m         100.0m                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-master-66685h4zx         100.0m            0.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-8jfkj               5.0m            0.0                        
  │  ├─ gpu-operator-f96dbb6fc-nk2jk                                            200.0m         500.0m                        
  │  ├─ kube-apiserver-beholder03                                               250.0m            0.0                        
  │  ├─ kube-controller-manager-beholder03                                      200.0m            0.0                        
  │  ├─ kube-router-7r8zt                                                       250.0m            0.0                        
  │  ├─ kube-scheduler-beholder03                                               100.0m            0.0                        
  │  ├─ ldap-67b47cf9b9-v6vvm                                                   250.0m            0.0                        
  │  ├─ virt-api-85578d9bb7-6mvxx                                                 5.0m            0.0                        
  │  ├─ virt-controller-674bcccb6-f2xfj                                          10.0m            0.0                        
  │  └─ virt-operator-b4d8f7f58-pnbkz                                            10.0m            0.0                        
  ├─ belial                                                                 (36%) 28.6     (68%) 54.1         80.0      25.9 
  │  ├─ cool-pod                                                                   2.0            2.0                        
  │  ├─ coredns-66bc5c9577-kxlwp                                                100.0m            0.0                        
  │  ├─ dnsutils-belial                                                         100.0m         100.0m                        
  │  ├─ gatekeeper-controller-manager-66f474f785-bq2vs                          100.0m            1.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-j7bdx               5.0m            0.0                        
  │  ├─ kube-router-k9stn                                                       250.0m            0.0                        
  │  ├─ ollama-pod                                                                 1.0            1.0                        
  │  ├─ ubuntu-gpu1                                                               25.0           50.0                        
  │  └─ virt-handler-rvzhr                                                       10.0m            0.0                        
  ├─ demogorgon                                                             (80%) 76.4   (146%) 140.1         96.0       0.0 
  │  ├─ a2v2-four-gpu-jcsz                                                        20.0           20.0                        
  │  ├─ dnsutils-demogorgon                                                     100.0m         100.0m                        
  │  ├─ felix-petersen-job-29                                                      6.0            8.0                        
  │  ├─ gpu-demogorgon                                                            48.0           48.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-pxws7               5.0m            0.0                        
  │  ├─ kube-router-xt9gr                                                       250.0m            0.0                        
  │  ├─ pycharm                                                                    1.0           32.0                        
  │  └─ pycharmv2                                                                  1.0           32.0                        
  ├─ fierna                                                                 (21%) 16.6     (31%) 25.1         80.0      54.9 
  │  ├─ dnsutils-fierna                                                         100.0m         100.0m                        
  │  ├─ gatekeeper-controller-manager-66f474f785-dvhjf                          100.0m            1.0                        
  │  ├─ gatekeeper-controller-manager-66f474f785-s9kgj                          100.0m            1.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-ldxbm               5.0m            0.0                        
  │  ├─ interact                                                                  10.0           16.0                        
  │  ├─ kube-router-ftwg9                                                       250.0m            0.0                        
  │  ├─ ledavio-similarity-search-9b864cc89-ln9lk                                  4.0            4.0                        
  │  ├─ ledavio-text-search-5d8f755795-t6x2w                                       1.0            2.0                        
  │  ├─ till-aust-ubuntu-entry-pod                                                 1.0            1.0                        
  │  └─ virt-handler-xmxj8                                                       10.0m            0.0                        
  ├─ kiaransalee                                                           (0%) 355.0m    (0%) 100.0m        192.0     191.6 
  │  ├─ dnsutils-kiaransalee                                                    100.0m         100.0m                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-jn8tv               5.0m            0.0                        
  │  └─ kube-router-gpmnw                                                       250.0m            0.0                        
  ├─ mindflayer01                                                           (33%) 21.1     (33%) 21.1         64.0      42.9 
  │  ├─ cdi-apiserver-7745487599-s5g6s                                          100.0m            0.0                        
  │  ├─ cdi-deployment-6c99bb8fcf-h79j8                                         100.0m            0.0                        
  │  ├─ cdi-uploadproxy-684cf5d896-9x5rh                                        100.0m            0.0                        
  │  ├─ dnsutils-mindflayer01                                                   100.0m         100.0m                        
  │  ├─ file-pod                                                                200.0m            1.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-gc-767f4bbfwdphf          10.0m            0.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-lv7bq               5.0m            0.0                        
  │  ├─ kube-router-xn2j8                                                       250.0m            0.0                        
  │  ├─ ubuntu-test-pod                                                         200.0m            2.0                        
  │  ├─ urs-waldmann-tb-access-pod                                                16.0           16.0                        
  │  ├─ valentin-schmuker-storage                                                  2.0            2.0                        
  │  ├─ virt-handler-tqrj4                                                       10.0m            0.0                        
  │  └─ virt-launcher-lightfield-analysis-8zbf6                                    2.0          15.0m                        
  ├─ mindflayer02                                                             (3%) 1.9       (3%) 2.1         64.0      61.9 
  │  ├─ dex-5b5d847b8-2hqxg                                                     250.0m            0.0                        
  │  ├─ dex-loginapp-77c84b57fd-wbc65                                           100.0m            0.0                        
  │  ├─ dex-mysql-589f4586bc-4n5vj                                              100.0m            0.0                        
  │  ├─ dnsutils-mindflayer02                                                   100.0m         100.0m                        
  │  ├─ gatekeeper-audit-59d4b6fd4c-gtwjm                                       100.0m            1.0                        
  │  ├─ kube-router-dsqlv                                                       250.0m            0.0                        
  │  ├─ mediawiki-77f9c84df5-p6k9g                                              250.0m            0.0                        
  │  ├─ mediawiki-mariadb-7ffb6c9b8d-ng7s6                                      250.0m            0.0                        
  │  ├─ registry-7996dcb999-kffqb                                               100.0m            0.0                        
  │  ├─ registry-auth-57776bfc77-7smsm                                          100.0m            0.0                        
  │  ├─ registry-browser-7f4cbdf96b-d4b66                                       200.0m            0.0                        
  │  └─ ubuntu-test-pod                                                         100.0m            1.0                        
  ├─ mindflayer03                                                           (41%) 26.5     (41%) 26.1         64.0      37.5 
  │  ├─ cdi-operator-76f7d8c545-vrql7                                           100.0m            0.0                        
  │  ├─ dnsutils-mindflayer03                                                   100.0m         100.0m                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-vlfn5               5.0m            0.0                        
  │  ├─ gpu-pod-aalbi                                                             10.0           10.0                        
  │  ├─ kube-router-9ljfj                                                       250.0m            0.0                        
  │  ├─ urs-waldmann-ubuntu-pod                                                   16.0           16.0                        
  │  └─ virt-handler-hndfm                                                       10.0m            0.0                        
  ├─ tiamat                                                                (0%) 355.0m    (0%) 100.0m        256.0     255.6 
  │  ├─ dnsutils-tiamat                                                         100.0m         100.0m                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-jg9c7               5.0m            0.0                        
  │  └─ kube-router-w2xvk                                                       250.0m            0.0                        
  ├─ vecna                                                                  (21%) 20.4     (21%) 20.1         96.0      75.6 
  │  ├─ bee-finetune-vecna-terminal                                               16.0           16.0                        
  │  ├─ dnsutils-vecna                                                          100.0m         100.0m                        
  │  ├─ felix-petersen-job-20                                                      4.0            4.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-mvz86               5.0m            0.0                        
  │  └─ kube-router-dl426                                                       250.0m            0.0                        
  └─ zariel                                                                (99%) 254.4    (99%) 254.1        256.0       1.6 
     ├─ allin1                                                                     4.0            4.0                        
     ├─ dnsutils-zariel                                                         100.0m         100.0m                        
     ├─ gpu-operator-1777509885-node-feature-discovery-worker-5fsqh               5.0m            0.0                        
     ├─ kube-router-9w62j                                                       250.0m            0.0                        
     └─ zariel-a2v-revision2-runs-jcsz                                           250.0          250.0                        
  devices.kubevirt.io/kvm                                                     (0%) 1.0       (0%) 1.0         4.0k      4.0k 
  ├─ belial                                                                   (0%) 0.0       (0%) 0.0         1.0k      1.0k 
  ├─ fierna                                                                   (0%) 0.0       (0%) 0.0         1.0k      1.0k 
  ├─ mindflayer01                                                             (0%) 1.0       (0%) 1.0         1.0k     999.0 
  │  └─ virt-launcher-lightfield-analysis-8zbf6                                    1.0            1.0                        
  └─ mindflayer03                                                             (0%) 0.0       (0%) 0.0         1.0k      1.0k 
  devices.kubevirt.io/tun                                                     (0%) 1.0       (0%) 1.0         4.0k      4.0k 
  ├─ belial                                                                   (0%) 0.0       (0%) 0.0         1.0k      1.0k 
  ├─ fierna                                                                   (0%) 0.0       (0%) 0.0         1.0k      1.0k 
  ├─ mindflayer01                                                             (0%) 1.0       (0%) 1.0         1.0k     999.0 
  │  └─ virt-launcher-lightfield-analysis-8zbf6                                    1.0            1.0                        
  └─ mindflayer03                                                             (0%) 0.0       (0%) 0.0         1.0k      1.0k 
  devices.kubevirt.io/vhost-net                                               (0%) 1.0       (0%) 1.0         4.0k      4.0k 
  ├─ belial                                                                   (0%) 0.0       (0%) 0.0         1.0k      1.0k 
  ├─ fierna                                                                   (0%) 0.0       (0%) 0.0         1.0k      1.0k 
  ├─ mindflayer01                                                             (0%) 1.0       (0%) 1.0         1.0k     999.0 
  │  └─ virt-launcher-lightfield-analysis-8zbf6                                    1.0            1.0                        
  └─ mindflayer03                                                             (0%) 0.0       (0%) 0.0         1.0k      1.0k 
  ephemeral-storage                                                        (1%) 139.9G   (2%) 211.0Gi        11.3T     11.1T 
  ├─ asmodeus                                                                 (0%) 0.0       (0%) 0.0        94.6G     94.6G 
  ├─ beholder01                                                               (0%) 0.0       (0%) 0.0         1.7T      1.7T 
  ├─ beholder02                                                               (0%) 0.0       (0%) 0.0         1.7T      1.7T 
  ├─ beholder03                                                               (0%) 0.0       (0%) 0.0         1.7T      1.7T 
  ├─ belial                                                              (57%) 100.0Gi  (85%) 150.0Gi       189.2G     28.2G 
  │  └─ ubuntu-gpu1                                                            100.0Gi        150.0Gi                        
  ├─ demogorgon                                                            (5%) 30.0Gi    (9%) 60.0Gi       706.7G    642.3G 
  │  ├─ pycharm                                                                 15.0Gi         30.0Gi                        
  │  └─ pycharmv2                                                               15.0Gi         30.0Gi                        
  ├─ fierna                                                               (0%) 256.0Mi     (1%) 1.0Gi       189.2G    188.1G 
  │  └─ ledavio-similarity-search-9b864cc89-ln9lk                              256.0Mi          1.0Gi                        
  ├─ kiaransalee                                                              (0%) 0.0       (0%) 0.0         1.7T      1.7T 
  ├─ mindflayer01                                                           (0%) 50.0M       (0%) 0.0       211.5G    211.5G 
  │  └─ virt-launcher-lightfield-analysis-8zbf6                                  50.0M            0.0                        
  ├─ mindflayer02                                                             (0%) 0.0       (0%) 0.0       211.5G    211.5G 
  ├─ mindflayer03                                                             (0%) 0.0       (0%) 0.0       211.5G    211.5G 
  ├─ tiamat                                                                   (0%) 0.0       (0%) 0.0       164.4G    164.4G 
  ├─ vecna                                                                    (0%) 0.0       (0%) 0.0       849.0G    849.0G 
  └─ zariel                                                                   (0%) 0.0       (0%) 0.0         1.7T      1.7T 
  memory                                                                    (44%) 6.1T     (47%) 6.6T       12.7Ti     6.7Ti 
  ├─ asmodeus                                                              (78%) 1.5Ti    (79%) 1.6Ti        2.0Ti   415.1Gi 
  │  ├─ asmodeus-single-gpu-julian-schaefer-zimmermann                           1.0Ti          1.0Ti                        
  │  ├─ dnsutils-asmodeus                                                      100.0Mi        100.0Mi                        
  │  ├─ gpu-a100-zsh-shm                                                       256.0Gi        284.0Gi                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-9q5ls             64.0Mi        512.0Mi                        
  │  ├─ kube-router-qbh4m                                                      250.0Mi            0.0                        
  │  ├─ step16-coolchic-gpu-run-asmodeus-a100-hq                               100.0Gi        100.0Gi                        
  │  ├─ step18-ftic-gpu-run-asmodeus-a100-hq-40cpu                             100.0Gi        100.0Gi                        
  │  └─ ulas-bingoel-model-pod-5                                                80.0Gi         80.0Gi                        
  ├─ beholder01                                                           (0%) 350.0Mi   (0%) 100.0Mi       92.9Gi    92.5Gi 
  │  ├─ dnsutils-beholder01                                                    100.0Mi        100.0Mi                        
  │  └─ kube-router-4x8js                                                      250.0Mi            0.0                        
  ├─ beholder02                                                             (3%) 2.6Gi     (2%) 1.6Gi       92.9Gi    90.3Gi 
  │  ├─ dnsutils-beholder02                                                    100.0Mi        100.0Mi                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-8rfmd             64.0Mi        512.0Mi                        
  │  ├─ kube-router-2dtfp                                                      250.0Mi            0.0                        
  │  ├─ mc-rj-ddcb7fb6-b2qh6                                                     1.0Gi          1.0Gi                        
  │  ├─ virt-api-85578d9bb7-5fwdl                                              500.0Mi            0.0                        
  │  ├─ virt-controller-674bcccb6-pvgxj                                        275.0Mi            0.0                        
  │  └─ virt-operator-b4d8f7f58-6jjxp                                          450.0Mi            0.0                        
  ├─ beholder03                                                             (2%) 1.9Gi     (5%) 5.1Gi       92.9Gi    87.8Gi 
  │  ├─ coredns-66bc5c9577-d6czh                                                70.0Mi        170.0Mi                        
  │  ├─ dnsutils-beholder03                                                    100.0Mi        100.0Mi                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-master-66685h4zx        128.0Mi          4.0Gi                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-8jfkj             64.0Mi        512.0Mi                        
  │  ├─ gpu-operator-f96dbb6fc-nk2jk                                           100.0Mi        350.0Mi                        
  │  ├─ kube-router-7r8zt                                                      250.0Mi            0.0                        
  │  ├─ virt-api-85578d9bb7-6mvxx                                              500.0Mi            0.0                        
  │  ├─ virt-controller-674bcccb6-f2xfj                                        275.0Mi            0.0                        
  │  └─ virt-operator-b4d8f7f58-pnbkz                                          450.0Mi            0.0                        
  ├─ belial                                                              (42%) 315.0Gi  (62%) 465.3Gi      754.4Gi   289.2Gi 
  │  ├─ cool-pod                                                                32.0Gi         32.0Gi                        
  │  ├─ coredns-66bc5c9577-kxlwp                                                70.0Mi        170.0Mi                        
  │  ├─ dnsutils-belial                                                        100.0Mi        100.0Mi                        
  │  ├─ gatekeeper-controller-manager-66f474f785-bq2vs                         256.0Mi        512.0Mi                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-j7bdx             64.0Mi        512.0Mi                        
  │  ├─ kube-router-k9stn                                                      250.0Mi            0.0                        
  │  ├─ ollama-pod                                                              32.0Gi         32.0Gi                        
  │  ├─ ubuntu-gpu1                                                            250.0Gi        400.0Gi                        
  │  └─ virt-handler-rvzhr                                                     325.0Mi            0.0                        
  ├─ demogorgon                                                            (62%) 1.2Ti    (65%) 1.3Ti        2.0Ti   699.2Gi 
  │  ├─ a2v2-four-gpu-jcsz                                                       1.0Ti          1.0Ti                        
  │  ├─ dnsutils-demogorgon                                                    100.0Mi        100.0Mi                        
  │  ├─ felix-petersen-job-29                                                   50.0Gi        100.0Gi                        
  │  ├─ gpu-demogorgon                                                          80.0Gi         80.0Gi                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-pxws7             64.0Mi        512.0Mi                        
  │  ├─ kube-router-xt9gr                                                      250.0Mi            0.0                        
  │  ├─ pycharm                                                                 46.0Gi         50.0Gi                        
  │  └─ pycharmv2                                                               46.0Gi         50.0Gi                        
  ├─ fierna                                                              (44%) 329.2Gi  (44%) 329.6Gi      754.4Gi   424.8Gi 
  │  ├─ dnsutils-fierna                                                        100.0Mi        100.0Mi                        
  │  ├─ gatekeeper-controller-manager-66f474f785-dvhjf                         256.0Mi        512.0Mi                        
  │  ├─ gatekeeper-controller-manager-66f474f785-s9kgj                         256.0Mi        512.0Mi                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-ldxbm             64.0Mi        512.0Mi                        
  │  ├─ interact                                                                64.0Gi         64.0Gi                        
  │  ├─ kube-router-ftwg9                                                      250.0Mi            0.0                        
  │  ├─ ledavio-similarity-search-9b864cc89-ln9lk                               32.0Gi         32.0Gi                        
  │  ├─ ledavio-text-search-5d8f755795-t6x2w                                    32.0Gi         32.0Gi                        
  │  ├─ till-aust-ubuntu-entry-pod                                             200.0Gi        200.0Gi                        
  │  └─ virt-handler-xmxj8                                                     325.0Mi            0.0                        
  ├─ kiaransalee                                                             (1%) 9.0G   (0%) 612.0Mi        1.5Ti      1.6T 
  │  ├─ dnsutils-kiaransalee                                                   100.0Mi        100.0Mi                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-jn8tv             64.0Mi        512.0Mi                        
  │  ├─ jupyter-farahadeeba109                                                    1.1G            0.0                        
  │  ├─ jupyter-hanna27m                                                          1.1G            0.0                        
  │  ├─ jupyter-huygenssteiner                                                    1.1G            0.0                        
  │  ├─ jupyter-kmayer24                                                          1.1G            0.0                        
  │  ├─ jupyter-lorenzrck                                                         1.1G            0.0                        
  │  ├─ jupyter-mariaruxandracojocaru                                             1.1G            0.0                        
  │  ├─ jupyter-mibar3                                                            1.1G            0.0                        
  │  ├─ jupyter-samkopecek                                                        1.1G            0.0                        
  │  └─ kube-router-gpmnw                                                      250.0Mi            0.0                        
  ├─ mindflayer01                                                          (19%) 78.0G   (45%) 182.2G      376.5Gi   206.8Gi 
  │  ├─ alertmanager-kube-prometheus-stack-1777-alertmanager-0                 200.0Mi            0.0                        
  │  ├─ cdi-apiserver-7745487599-s5g6s                                         150.0Mi            0.0                        
  │  ├─ cdi-deployment-6c99bb8fcf-h79j8                                        150.0Mi            0.0                        
  │  ├─ cdi-uploadproxy-684cf5d896-9x5rh                                       150.0Mi            0.0                        
  │  ├─ dnsutils-mindflayer01                                                  100.0Mi        100.0Mi                        
  │  ├─ file-pod                                                               256.0Mi          1.0Gi                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-gc-767f4bbfwdphf        128.0Mi          1.0Gi                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-lv7bq             64.0Mi        512.0Mi                        
  │  ├─ kube-router-xn2j8                                                      250.0Mi            0.0                        
  │  ├─ ubuntu-test-pod                                                        612.0Mi          5.0Gi                        
  │  ├─ urs-waldmann-tb-access-pod                                              64.0Gi        160.0Gi                        
  │  ├─ valentin-schmuker-storage                                                2.0Gi          2.0Gi                        
  │  ├─ virt-handler-tqrj4                                                     325.0Mi            0.0                        
  │  └─ virt-launcher-lightfield-analysis-8zbf6                                   4.6G          60.0M                        
  ├─ mindflayer02                                                         (0%) 706.0Mi     (0%) 1.6Gi      376.5Gi   374.9Gi 
  │  ├─ dnsutils-mindflayer02                                                  100.0Mi        100.0Mi                        
  │  ├─ gatekeeper-audit-59d4b6fd4c-gtwjm                                      256.0Mi        512.0Mi                        
  │  ├─ kube-router-dsqlv                                                      250.0Mi            0.0                        
  │  └─ ubuntu-test-pod                                                        100.0Mi          1.0Gi                        
  ├─ mindflayer03                                                         (20%) 74.9Gi  (45%) 170.6Gi      376.5Gi   205.9Gi 
  │  ├─ cdi-operator-76f7d8c545-vrql7                                          150.0Mi            0.0                        
  │  ├─ dnsutils-mindflayer03                                                  100.0Mi        100.0Mi                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-vlfn5             64.0Mi        512.0Mi                        
  │  ├─ gpu-pod-aalbi                                                           10.0Gi         10.0Gi                        
  │  ├─ kube-router-9ljfj                                                      250.0Mi            0.0                        
  │  ├─ urs-waldmann-ubuntu-pod                                                 64.0Gi        160.0Gi                        
  │  └─ virt-handler-hndfm                                                     325.0Mi            0.0                        
  ├─ tiamat                                                               (0%) 414.0Mi   (0%) 612.0Mi     1007.6Gi  1007.0Gi 
  │  ├─ dnsutils-tiamat                                                        100.0Mi        100.0Mi                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-jg9c7             64.0Mi        512.0Mi                        
  │  └─ kube-router-w2xvk                                                      250.0Mi            0.0                        
  ├─ vecna                                                                 (5%) 82.4Gi    (5%) 82.6Gi        1.5Ti     1.4Ti 
  │  ├─ bee-finetune-vecna-terminal                                             32.0Gi         32.0Gi                        
  │  ├─ dnsutils-vecna                                                         100.0Mi        100.0Mi                        
  │  ├─ felix-petersen-job-20                                                   50.0Gi         50.0Gi                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-mvz86             64.0Mi        512.0Mi                        
  │  └─ kube-router-dl426                                                      250.0Mi            0.0                        
  └─ zariel                                                                (98%) 1.9Ti    (99%) 2.0Ti        2.0Ti    14.9Gi 
     ├─ allin1                                                                  64.0Gi         80.0Gi                        
     ├─ dnsutils-zariel                                                        100.0Mi        100.0Mi                        
     ├─ gpu-operator-1777509885-node-feature-discovery-worker-5fsqh             64.0Mi        512.0Mi                        
     ├─ kube-router-9w62j                                                      250.0Mi            0.0                        
     └─ zariel-a2v-revision2-runs-jcsz                                           1.9Ti          1.9Ti                        
  nvidia.com/gpu                                                            (52%) 32.0     (52%) 32.0         62.0      30.0 
  ├─ asmodeus                                                               (100%) 4.0     (100%) 4.0          4.0       0.0 
  │  ├─ asmodeus-single-gpu-julian-schaefer-zimmermann                             1.0            1.0                        
  │  ├─ step16-coolchic-gpu-run-asmodeus-a100-hq                                   1.0            1.0                        
  │  ├─ step18-ftic-gpu-run-asmodeus-a100-hq-40cpu                                 1.0            1.0                        
  │  └─ ulas-bingoel-model-pod-5                                                   1.0            1.0                        
  ├─ belial                                                                  (25%) 2.0      (25%) 2.0          8.0       6.0 
  │  ├─ ollama-pod                                                                 1.0            1.0                        
  │  └─ ubuntu-gpu1                                                                1.0            1.0                        
  ├─ demogorgon                                                             (100%) 8.0     (100%) 8.0          8.0       0.0 
  │  ├─ a2v2-four-gpu-jcsz                                                         4.0            4.0                        
  │  ├─ felix-petersen-job-29                                                      1.0            1.0                        
  │  ├─ gpu-demogorgon                                                             1.0            1.0                        
  │  ├─ pycharm                                                                    1.0            1.0                        
  │  └─ pycharmv2                                                                  1.0            1.0                        
  ├─ fierna                                                                  (38%) 3.0      (38%) 3.0          8.0       5.0 
  │  ├─ interact                                                                   1.0            1.0                        
  │  ├─ ledavio-similarity-search-9b864cc89-ln9lk                                  1.0            1.0                        
  │  └─ ledavio-text-search-5d8f755795-t6x2w                                       1.0            1.0                        
  ├─ kiaransalee                                                            (100%) 6.0     (100%) 6.0          6.0       0.0 
  │  ├─ jupyter-farahadeeba109                                                     1.0            1.0                        
  │  ├─ jupyter-huygenssteiner                                                     1.0            1.0                        
  │  ├─ jupyter-kmayer24                                                           1.0            1.0                        
  │  ├─ jupyter-lorenzrck                                                          1.0            1.0                        
  │  ├─ jupyter-mariaruxandracojocaru                                              1.0            1.0                        
  │  └─ jupyter-samkopecek                                                         1.0            1.0                        
  ├─ tiamat                                                                   (0%) 0.0       (0%) 0.0          4.0       4.0 
  ├─ vecna                                                                    (6%) 1.0       (6%) 1.0         16.0      15.0 
  │  └─ felix-petersen-job-20                                                      1.0            1.0                        
  └─ zariel                                                                 (100%) 8.0     (100%) 8.0          8.0       0.0 
     └─ zariel-a2v-revision2-runs-jcsz                                             8.0            8.0                        
  nvidia.com/mig-3g.40gb                                                     (50%) 1.0      (50%) 1.0          2.0       1.0 
  └─ kiaransalee                                                             (50%) 1.0      (50%) 1.0          2.0       1.0 
     └─ jupyter-mibar3                                                             1.0            1.0                        
  nvidia.com/mig-4g.40gb                                                     (50%) 1.0      (50%) 1.0          2.0       1.0 
  └─ kiaransalee                                                             (50%) 1.0      (50%) 1.0          2.0       1.0 
     └─ jupyter-hanna27m                                                           1.0            1.0                        
  pods                                                                     (14%) 223.0    (14%) 223.0         1.5k      1.3k 
  ├─ asmodeus                                                               (15%) 16.0     (15%) 16.0        110.0      94.0 
  │  ├─ asmodeus-single-gpu-julian-schaefer-zimmermann                             1.0            1.0                        
  │  ├─ dnsutils-asmodeus                                                          1.0            1.0                        
  │  ├─ gpu-a100-zsh-shm                                                           1.0            1.0                        
  │  ├─ gpu-feature-discovery-jgxzx                                                1.0            1.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-9q5ls                1.0            1.0                        
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-qdkvw            1.0            1.0                        
  │  ├─ kube-proxy-kxtvf                                                           1.0            1.0                        
  │  ├─ kube-router-qbh4m                                                          1.0            1.0                        
  │  ├─ nvidia-container-toolkit-daemonset-866wd                                   1.0            1.0                        
  │  ├─ nvidia-dcgm-exporter-smxtb                                                 1.0            1.0                        
  │  ├─ nvidia-device-plugin-daemonset-xjtg9                                       1.0            1.0                        
  │  ├─ nvidia-mig-manager-56kzk                                                   1.0            1.0                        
  │  ├─ nvidia-operator-validator-pz657                                            1.0            1.0                        
  │  ├─ step16-coolchic-gpu-run-asmodeus-a100-hq                                   1.0            1.0                        
  │  ├─ step18-ftic-gpu-run-asmodeus-a100-hq-40cpu                                 1.0            1.0                        
  │  └─ ulas-bingoel-model-pod-5                                                   1.0            1.0                        
  ├─ beholder01                                                               (7%) 8.0       (7%) 8.0        110.0     102.0 
  │  ├─ dnsutils-beholder01                                                        1.0            1.0                        
  │  ├─ kube-apiserver-beholder01                                                  1.0            1.0                        
  │  ├─ kube-controller-manager-beholder01                                         1.0            1.0                        
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-g8bwl            1.0            1.0                        
  │  ├─ kube-proxy-jw87r                                                           1.0            1.0                        
  │  ├─ kube-router-4x8js                                                          1.0            1.0                        
  │  ├─ kube-scheduler-beholder01                                                  1.0            1.0                        
  │  └─ vm-proxy-frontend-766dd9b967-swbkr                                         1.0            1.0                        
  ├─ beholder02                                                             (14%) 15.0     (14%) 15.0        110.0      95.0 
  │  ├─ dnsutils-beholder02                                                        1.0            1.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-8rfmd                1.0            1.0                        
  │  ├─ hub-848f5d5578-47s2j                                                       1.0            1.0                        
  │  ├─ kube-apiserver-beholder02                                                  1.0            1.0                        
  │  ├─ kube-controller-manager-beholder02                                         1.0            1.0                        
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-pgkbs            1.0            1.0                        
  │  ├─ kube-proxy-cmzf2                                                           1.0            1.0                        
  │  ├─ kube-router-2dtfp                                                          1.0            1.0                        
  │  ├─ kube-scheduler-beholder02                                                  1.0            1.0                        
  │  ├─ mc-rj-ddcb7fb6-b2qh6                                                       1.0            1.0                        
  │  ├─ nginx-ip-2025-7fd66b99dd-2khh6                                             1.0            1.0                        
  │  ├─ nginx-rsn-2024-57d49484d-47rfc                                             1.0            1.0                        
  │  ├─ virt-api-85578d9bb7-5fwdl                                                  1.0            1.0                        
  │  ├─ virt-controller-674bcccb6-pvgxj                                            1.0            1.0                        
  │  └─ virt-operator-b4d8f7f58-6jjxp                                              1.0            1.0                        
  ├─ beholder03                                                             (15%) 16.0     (15%) 16.0        110.0      94.0 
  │  ├─ coredns-66bc5c9577-d6czh                                                   1.0            1.0                        
  │  ├─ dnsutils-beholder03                                                        1.0            1.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-master-66685h4zx            1.0            1.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-8jfkj                1.0            1.0                        
  │  ├─ gpu-operator-f96dbb6fc-nk2jk                                               1.0            1.0                        
  │  ├─ kube-apiserver-beholder03                                                  1.0            1.0                        
  │  ├─ kube-controller-manager-beholder03                                         1.0            1.0                        
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-jrpqw            1.0            1.0                        
  │  ├─ kube-proxy-cftmz                                                           1.0            1.0                        
  │  ├─ kube-router-7r8zt                                                          1.0            1.0                        
  │  ├─ kube-scheduler-beholder03                                                  1.0            1.0                        
  │  ├─ ldap-67b47cf9b9-v6vvm                                                      1.0            1.0                        
  │  ├─ memcached-6b68cdd947-w4k2q                                                 1.0            1.0                        
  │  ├─ virt-api-85578d9bb7-6mvxx                                                  1.0            1.0                        
  │  ├─ virt-controller-674bcccb6-f2xfj                                            1.0            1.0                        
  │  └─ virt-operator-b4d8f7f58-pnbkz                                              1.0            1.0                        
  ├─ belial                                                                 (15%) 16.0     (15%) 16.0        110.0      94.0 
  │  ├─ cool-pod                                                                   1.0            1.0                        
  │  ├─ coredns-66bc5c9577-kxlwp                                                   1.0            1.0                        
  │  ├─ dnsutils-belial                                                            1.0            1.0                        
  │  ├─ gatekeeper-controller-manager-66f474f785-bq2vs                             1.0            1.0                        
  │  ├─ gpu-feature-discovery-s2tzn                                                1.0            1.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-j7bdx                1.0            1.0                        
  │  ├─ kube-proxy-xxgbc                                                           1.0            1.0                        
  │  ├─ kube-router-k9stn                                                          1.0            1.0                        
  │  ├─ nvidia-container-toolkit-daemonset-bt92r                                   1.0            1.0                        
  │  ├─ nvidia-dcgm-exporter-rtjj5                                                 1.0            1.0                        
  │  ├─ nvidia-device-plugin-daemonset-j96xt                                       1.0            1.0                        
  │  ├─ nvidia-operator-validator-9gr7w                                            1.0            1.0                        
  │  ├─ ollama-pod                                                                 1.0            1.0                        
  │  ├─ ubuntu-gpu1                                                                1.0            1.0                        
  │  ├─ virt-handler-rvzhr                                                         1.0            1.0                        
  │  └─ whoami-74dc54d675-d6p8r                                                    1.0            1.0                        
  ├─ demogorgon                                                             (14%) 15.0     (14%) 15.0        110.0      95.0 
  │  ├─ a2v2-four-gpu-jcsz                                                         1.0            1.0                        
  │  ├─ dnsutils-demogorgon                                                        1.0            1.0                        
  │  ├─ felix-petersen-job-29                                                      1.0            1.0                        
  │  ├─ gpu-demogorgon                                                             1.0            1.0                        
  │  ├─ gpu-feature-discovery-mqtsq                                                1.0            1.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-pxws7                1.0            1.0                        
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-mdrd4            1.0            1.0                        
  │  ├─ kube-proxy-xdkh7                                                           1.0            1.0                        
  │  ├─ kube-router-xt9gr                                                          1.0            1.0                        
  │  ├─ nvidia-container-toolkit-daemonset-t2bv4                                   1.0            1.0                        
  │  ├─ nvidia-dcgm-exporter-v8zxc                                                 1.0            1.0                        
  │  ├─ nvidia-device-plugin-daemonset-fq2lz                                       1.0            1.0                        
  │  ├─ nvidia-operator-validator-qww8n                                            1.0            1.0                        
  │  ├─ pycharm                                                                    1.0            1.0                        
  │  └─ pycharmv2                                                                  1.0            1.0                        
  ├─ fierna                                                                 (21%) 23.0     (21%) 23.0        110.0      87.0 
  │  ├─ dnsutils-fierna                                                            1.0            1.0                        
  │  ├─ gatekeeper-controller-manager-66f474f785-dvhjf                             1.0            1.0                        
  │  ├─ gatekeeper-controller-manager-66f474f785-s9kgj                             1.0            1.0                        
  │  ├─ gpu-feature-discovery-sspcw                                                1.0            1.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-ldxbm                1.0            1.0                        
  │  ├─ interact                                                                   1.0            1.0                        
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-8f4nd            1.0            1.0                        
  │  ├─ kube-proxy-pgw9t                                                           1.0            1.0                        
  │  ├─ kube-router-ftwg9                                                          1.0            1.0                        
  │  ├─ ledavio-similarity-search-9b864cc89-ln9lk                                  1.0            1.0                        
  │  ├─ ledavio-text-search-5d8f755795-t6x2w                                       1.0            1.0                        
  │  ├─ nvidia-container-toolkit-daemonset-rbvmw                                   1.0            1.0                        
  │  ├─ nvidia-dcgm-exporter-fblhj                                                 1.0            1.0                        
  │  ├─ nvidia-device-plugin-daemonset-vsr66                                       1.0            1.0                        
  │  ├─ nvidia-operator-validator-8r84t                                            1.0            1.0                        
  │  ├─ proxy-5495d795d5-jjk7j                                                     1.0            1.0                        
  │  ├─ proxy-5bc89cc587-fwm6j                                                     1.0            1.0                        
  │  ├─ proxy-7f79cc645f-52qjx                                                     1.0            1.0                        
  │  ├─ till-aust-ubuntu-entry-pod                                                 1.0            1.0                        
  │  ├─ user-scheduler-5cf5ffbc54-wrnrk                                            1.0            1.0                        
  │  ├─ user-scheduler-c7db6c584-6vbss                                             1.0            1.0                        
  │  ├─ virt-handler-xmxj8                                                         1.0            1.0                        
  │  └─ whoami-74dc54d675-vsljt                                                    1.0            1.0                        
  ├─ kiaransalee                                                            (19%) 21.0     (19%) 21.0        110.0      89.0 
  │  ├─ continuous-image-puller-6fs4k                                              1.0            1.0                        
  │  ├─ continuous-image-puller-6z8bj                                              1.0            1.0                        
  │  ├─ dnsutils-kiaransalee                                                       1.0            1.0                        
  │  ├─ gpu-feature-discovery-52lgs                                                1.0            1.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-jn8tv                1.0            1.0                        
  │  ├─ jupyter-farahadeeba109                                                     1.0            1.0                        
  │  ├─ jupyter-hanna27m                                                           1.0            1.0                        
  │  ├─ jupyter-huygenssteiner                                                     1.0            1.0                        
  │  ├─ jupyter-kmayer24                                                           1.0            1.0                        
  │  ├─ jupyter-lorenzrck                                                          1.0            1.0                        
  │  ├─ jupyter-mariaruxandracojocaru                                              1.0            1.0                        
  │  ├─ jupyter-mibar3                                                             1.0            1.0                        
  │  ├─ jupyter-samkopecek                                                         1.0            1.0                        
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-xc2hx            1.0            1.0                        
  │  ├─ kube-proxy-l8zn4                                                           1.0            1.0                        
  │  ├─ kube-router-gpmnw                                                          1.0            1.0                        
  │  ├─ nvidia-container-toolkit-daemonset-p2dgh                                   1.0            1.0                        
  │  ├─ nvidia-dcgm-exporter-xhdc4                                                 1.0            1.0                        
  │  ├─ nvidia-device-plugin-daemonset-wl2gh                                       1.0            1.0                        
  │  ├─ nvidia-mig-manager-5rrnr                                                   1.0            1.0                        
  │  └─ nvidia-operator-validator-99cc8                                            1.0            1.0                        
  ├─ mindflayer01                                                           (21%) 23.0     (21%) 23.0        110.0      87.0 
  │  ├─ alertmanager-kube-prometheus-stack-1777-alertmanager-0                     1.0            1.0                        
  │  ├─ cdi-apiserver-7745487599-s5g6s                                             1.0            1.0                        
  │  ├─ cdi-deployment-6c99bb8fcf-h79j8                                            1.0            1.0                        
  │  ├─ cdi-uploadproxy-684cf5d896-9x5rh                                           1.0            1.0                        
  │  ├─ dnsutils-mindflayer01                                                      1.0            1.0                        
  │  ├─ file-pod                                                                   1.0            1.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-gc-767f4bbfwdphf            1.0            1.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-lv7bq                1.0            1.0                        
  │  ├─ hub-78d6dd898d-hb67q                                                       1.0            1.0                        
  │  ├─ kube-prometheus-stack-1777-operator-6bcf46cfd5-2j6lk                       1.0            1.0                        
  │  ├─ kube-prometheus-stack-1777533213-grafana-6ccc5bf477-mw65j                  1.0            1.0                        
  │  ├─ kube-prometheus-stack-1777533213-kube-state-metrics-7f89b5fsvf6            1.0            1.0                        
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-f9z45            1.0            1.0                        
  │  ├─ kube-proxy-49vx2                                                           1.0            1.0                        
  │  ├─ kube-router-xn2j8                                                          1.0            1.0                        
  │  ├─ prometheus-kube-prometheus-stack-1777-prometheus-0                         1.0            1.0                        
  │  ├─ ubuntu-test-pod                                                            2.0            2.0                        
  │  ├─ urs-waldmann-tb-access-pod                                                 1.0            1.0                        
  │  ├─ user-scheduler-5cf5ffbc54-n9qgd                                            1.0            1.0                        
  │  ├─ valentin-schmuker-storage                                                  1.0            1.0                        
  │  ├─ virt-handler-tqrj4                                                         1.0            1.0                        
  │  └─ virt-launcher-lightfield-analysis-8zbf6                                    1.0            1.0                        
  ├─ mindflayer02                                                           (22%) 24.0     (22%) 24.0        110.0      86.0 
  │  ├─ cert-manager-79559475b4-7kv54                                              1.0            1.0                        
  │  ├─ cert-manager-cainjector-966fc8fbc-zql8j                                    1.0            1.0                        
  │  ├─ cert-manager-webhook-854cf5f458-wwf4d                                      1.0            1.0                        
  │  ├─ dex-5b5d847b8-2hqxg                                                        1.0            1.0                        
  │  ├─ dex-loginapp-77c84b57fd-wbc65                                              1.0            1.0                        
  │  ├─ dex-mysql-589f4586bc-4n5vj                                                 1.0            1.0                        
  │  ├─ dnsutils-mindflayer02                                                      1.0            1.0                        
  │  ├─ gatekeeper-audit-59d4b6fd4c-gtwjm                                          1.0            1.0                        
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-qz6sb            1.0            1.0                        
  │  ├─ kube-proxy-dfgxf                                                           1.0            1.0                        
  │  ├─ kube-router-dsqlv                                                          1.0            1.0                        
  │  ├─ local-path-provisioner-759479454f-7pqw8                                    1.0            1.0                        
  │  ├─ mediawiki-77f9c84df5-p6k9g                                                 1.0            1.0                        
  │  ├─ mediawiki-mariadb-7ffb6c9b8d-ng7s6                                         1.0            1.0                        
  │  ├─ nginx-k8s-5889449f8b-dv6xq                                                 1.0            1.0                        
  │  ├─ nginx-rec-2026-75dd946d4d-df9gc                                            1.0            1.0                        
  │  ├─ nginx-self-service-password-54767ddc56-d556f                               1.0            1.0                        
  │  ├─ pdf-55ccd6f459-s7nbj                                                       1.0            1.0                        
  │  ├─ registry-7996dcb999-kffqb                                                  1.0            1.0                        
  │  ├─ registry-auth-57776bfc77-7smsm                                             1.0            1.0                        
  │  ├─ registry-browser-7f4cbdf96b-d4b66                                          1.0            1.0                        
  │  ├─ traefik-deployment-d8ccbfdd4-8xqsn                                         1.0            1.0                        
  │  ├─ ubuntu-test-pod                                                            1.0            1.0                        
  │  └─ user-scheduler-c7db6c584-2pxhd                                             1.0            1.0                        
  ├─ mindflayer03                                                             (8%) 9.0       (8%) 9.0        110.0     101.0 
  │  ├─ cdi-operator-76f7d8c545-vrql7                                              1.0            1.0                        
  │  ├─ dnsutils-mindflayer03                                                      1.0            1.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-vlfn5                1.0            1.0                        
  │  ├─ gpu-pod-aalbi                                                              1.0            1.0                        
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-fmsql            1.0            1.0                        
  │  ├─ kube-proxy-d2lmv                                                           1.0            1.0                        
  │  ├─ kube-router-9ljfj                                                          1.0            1.0                        
  │  ├─ urs-waldmann-ubuntu-pod                                                    1.0            1.0                        
  │  └─ virt-handler-hndfm                                                         1.0            1.0                        
  ├─ tiamat                                                                 (11%) 12.0     (11%) 12.0        110.0      98.0 
  │  ├─ continuous-image-puller-wtkhd                                              1.0            1.0                        
  │  ├─ dnsutils-tiamat                                                            1.0            1.0                        
  │  ├─ gpu-feature-discovery-ggqsw                                                1.0            1.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-jg9c7                1.0            1.0                        
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-zwnjs            1.0            1.0                        
  │  ├─ kube-proxy-n8m88                                                           1.0            1.0                        
  │  ├─ kube-router-w2xvk                                                          1.0            1.0                        
  │  ├─ nvidia-container-toolkit-daemonset-7nh8q                                   1.0            1.0                        
  │  ├─ nvidia-dcgm-exporter-8hbwd                                                 1.0            1.0                        
  │  ├─ nvidia-device-plugin-daemonset-gbv2h                                       1.0            1.0                        
  │  ├─ nvidia-mig-manager-475gc                                                   1.0            1.0                        
  │  └─ nvidia-operator-validator-mjbvb                                            1.0            1.0                        
  ├─ vecna                                                                  (11%) 12.0     (11%) 12.0        110.0      98.0 
  │  ├─ bee-finetune-vecna-terminal                                                1.0            1.0                        
  │  ├─ dnsutils-vecna                                                             1.0            1.0                        
  │  ├─ felix-petersen-job-20                                                      1.0            1.0                        
  │  ├─ gpu-feature-discovery-dg72r                                                1.0            1.0                        
  │  ├─ gpu-operator-1777509885-node-feature-discovery-worker-mvz86                1.0            1.0                        
  │  ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-5d5rl            1.0            1.0                        
  │  ├─ kube-proxy-6r98t                                                           1.0            1.0                        
  │  ├─ kube-router-dl426                                                          1.0            1.0                        
  │  ├─ nvidia-container-toolkit-daemonset-l7ddn                                   1.0            1.0                        
  │  ├─ nvidia-dcgm-exporter-774vj                                                 1.0            1.0                        
  │  ├─ nvidia-device-plugin-daemonset-mkhc4                                       1.0            1.0                        
  │  └─ nvidia-operator-validator-4jkmc                                            1.0            1.0                        
  └─ zariel                                                                 (12%) 13.0     (12%) 13.0        110.0      97.0 
     ├─ allin1                                                                     1.0            1.0                        
     ├─ dnsutils-zariel                                                            1.0            1.0                        
     ├─ gpu-feature-discovery-ctcnd                                                1.0            1.0                        
     ├─ gpu-operator-1777509885-node-feature-discovery-worker-5fsqh                1.0            1.0                        
     ├─ kube-prometheus-stack-1777533213-prometheus-node-exporter-wdj8l            1.0            1.0                        
     ├─ kube-proxy-gsqm7                                                           1.0            1.0                        
     ├─ kube-router-9w62j                                                          1.0            1.0                        
     ├─ nvidia-container-toolkit-daemonset-98jbp                                   1.0            1.0                        
     ├─ nvidia-dcgm-exporter-55cl4                                                 1.0            1.0                        
     ├─ nvidia-device-plugin-daemonset-clcv7                                       1.0            1.0                        
     ├─ nvidia-mig-manager-jll8z                                                   1.0            1.0                        
     ├─ nvidia-operator-validator-8mf54                                            1.0            1.0                        
     └─ zariel-a2v-revision2-runs-jcsz                                             1.0            1.0                        




Resource usage by namespace

 Resource                              Requested    Limit  Allocatable  Free 
  auth                                                                       
  ├─ beholder03                                                              
  │  ├─ cpu                               250.0m      0.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer02                                                            
     ├─ cpu                               450.0m      0.0                    
     └─ pods                                 4.0      4.0                    
  cdi                                                                        
  ├─ mindflayer01                                                            
  │  ├─ cpu                                  2.3    15.0m                    
  │  ├─ devices.kubevirt.io/kvm              1.0      1.0                    
  │  ├─ devices.kubevirt.io/tun              1.0      1.0                    
  │  ├─ devices.kubevirt.io/vhost-net        1.0      1.0                    
  │  ├─ ephemeral-storage                  50.0M      0.0                    
  │  ├─ memory                              5.1G    60.0M                    
  │  └─ pods                                 4.0      4.0                    
  └─ mindflayer03                                                            
     ├─ cpu                               100.0m      0.0                    
     ├─ memory                           150.0Mi      0.0                    
     └─ pods                                 1.0      1.0                    
  cert-manager                               3.0      3.0                    
  └─ mindflayer02                            3.0      3.0                    
     └─ pods                                 3.0      3.0                    
  gatekeeper-system                                                          
  ├─ belial                                                                  
  │  ├─ cpu                               100.0m      1.0                    
  │  ├─ memory                           256.0Mi  512.0Mi                    
  │  └─ pods                                 1.0      1.0                    
  ├─ fierna                                                                  
  │  ├─ cpu                               200.0m      2.0                    
  │  ├─ memory                           512.0Mi    1.0Gi                    
  │  └─ pods                                 2.0      2.0                    
  └─ mindflayer02                                                            
     ├─ cpu                               100.0m      1.0                    
     ├─ memory                           256.0Mi  512.0Mi                    
     └─ pods                                 1.0      1.0                    
  gpu-operator                                                               
  ├─ asmodeus                                                                
  │  ├─ cpu                                 5.0m      0.0                    
  │  ├─ memory                            64.0Mi  512.0Mi                    
  │  └─ pods                                 7.0      7.0                    
  ├─ beholder02                                                              
  │  ├─ cpu                                 5.0m      0.0                    
  │  ├─ memory                            64.0Mi  512.0Mi                    
  │  └─ pods                                 1.0      1.0                    
  ├─ beholder03                                                              
  │  ├─ cpu                               305.0m   500.0m                    
  │  ├─ memory                           292.0Mi    4.8Gi                    
  │  └─ pods                                 3.0      3.0                    
  ├─ belial                                                                  
  │  ├─ cpu                                 5.0m      0.0                    
  │  ├─ memory                            64.0Mi  512.0Mi                    
  │  └─ pods                                 6.0      6.0                    
  ├─ demogorgon                                                              
  │  ├─ cpu                                 5.0m      0.0                    
  │  ├─ memory                            64.0Mi  512.0Mi                    
  │  └─ pods                                 6.0      6.0                    
  ├─ fierna                                                                  
  │  ├─ cpu                                 5.0m      0.0                    
  │  ├─ memory                            64.0Mi  512.0Mi                    
  │  └─ pods                                 6.0      6.0                    
  ├─ kiaransalee                                                             
  │  ├─ cpu                                 5.0m      0.0                    
  │  ├─ memory                            64.0Mi  512.0Mi                    
  │  └─ pods                                 7.0      7.0                    
  ├─ mindflayer01                                                            
  │  ├─ cpu                                15.0m      0.0                    
  │  ├─ memory                           192.0Mi    1.5Gi                    
  │  └─ pods                                 2.0      2.0                    
  ├─ mindflayer03                                                            
  │  ├─ cpu                                 5.0m      0.0                    
  │  ├─ memory                            64.0Mi  512.0Mi                    
  │  └─ pods                                 1.0      1.0                    
  ├─ tiamat                                                                  
  │  ├─ cpu                                 5.0m      0.0                    
  │  ├─ memory                            64.0Mi  512.0Mi                    
  │  └─ pods                                 7.0      7.0                    
  ├─ vecna                                                                   
  │  ├─ cpu                                 5.0m      0.0                    
  │  ├─ memory                            64.0Mi  512.0Mi                    
  │  └─ pods                                 6.0      6.0                    
  └─ zariel                                                                  
     ├─ cpu                                 5.0m      0.0                    
     ├─ memory                            64.0Mi  512.0Mi                    
     └─ pods                                 7.0      7.0                    
  jupyterhub                                                                 
  ├─ beholder02                              1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ fierna                                  2.0      2.0                    
  │  └─ pods                                 2.0      2.0                    
  ├─ kiaransalee                                                             
  │  ├─ memory                              8.6G      0.0                    
  │  ├─ nvidia.com/gpu                       6.0      6.0                    
  │  ├─ nvidia.com/mig-3g.40gb               1.0      1.0                    
  │  ├─ nvidia.com/mig-4g.40gb               1.0      1.0                    
  │  └─ pods                                 9.0      9.0                    
  └─ mindflayer02                            1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  jupyterhub-kuckling                        2.0      2.0                    
  ├─ fierna                                  1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ tiamat                                  1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  jupyterhub-students                        5.0      5.0                    
  ├─ fierna                                  2.0      2.0                    
  │  └─ pods                                 2.0      2.0                    
  ├─ kiaransalee                             1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer01                            2.0      2.0                    
     └─ pods                                 2.0      2.0                    
  kube-system                                                                
  ├─ asmodeus                                                                
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 3.0      3.0                    
  ├─ beholder01                                                              
  │  ├─ cpu                               900.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 6.0      6.0                    
  ├─ beholder02                                                              
  │  ├─ cpu                               900.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 6.0      6.0                    
  ├─ beholder03                                                              
  │  ├─ cpu                                  1.0   100.0m                    
  │  ├─ memory                           420.0Mi  270.0Mi                    
  │  └─ pods                                 7.0      7.0                    
  ├─ belial                                                                  
  │  ├─ cpu                               450.0m   100.0m                    
  │  ├─ memory                           420.0Mi  270.0Mi                    
  │  └─ pods                                 4.0      4.0                    
  ├─ demogorgon                                                              
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 3.0      3.0                    
  ├─ fierna                                                                  
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 3.0      3.0                    
  ├─ kiaransalee                                                             
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 3.0      3.0                    
  ├─ mindflayer01                                                            
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 3.0      3.0                    
  ├─ mindflayer02                                                            
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 3.0      3.0                    
  ├─ mindflayer03                                                            
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 3.0      3.0                    
  ├─ tiamat                                                                  
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 3.0      3.0                    
  ├─ vecna                                                                   
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 3.0      3.0                    
  └─ zariel                                                                  
     ├─ cpu                               350.0m   100.0m                    
     ├─ memory                           350.0Mi  100.0Mi                    
     └─ pods                                 3.0      3.0                    
  kubevirt                                                                   
  ├─ beholder02                                                              
  │  ├─ cpu                                25.0m      0.0                    
  │  ├─ memory                             1.2Gi      0.0                    
  │  └─ pods                                 3.0      3.0                    
  ├─ beholder03                                                              
  │  ├─ cpu                                25.0m      0.0                    
  │  ├─ memory                             1.2Gi      0.0                    
  │  └─ pods                                 3.0      3.0                    
  ├─ belial                                                                  
  │  ├─ cpu                                10.0m      0.0                    
  │  ├─ memory                           325.0Mi      0.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ fierna                                                                  
  │  ├─ cpu                                10.0m      0.0                    
  │  ├─ memory                           325.0Mi      0.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ mindflayer01                                                            
  │  ├─ cpu                                10.0m      0.0                    
  │  ├─ memory                           325.0Mi      0.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer03                                                            
     ├─ cpu                                10.0m      0.0                    
     ├─ memory                           325.0Mi      0.0                    
     └─ pods                                 1.0      1.0                    
  local-path-storage                         1.0      1.0                    
  └─ mindflayer02                            1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  monitoring                                                                 
  ├─ asmodeus                                1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ beholder01                              1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ beholder02                              1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ beholder03                              1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ demogorgon                              1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ fierna                                  1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ kiaransalee                             1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ mindflayer01                                                            
  │  ├─ memory                           200.0Mi      0.0                    
  │  └─ pods                                 6.0      6.0                    
  ├─ mindflayer02                            1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ mindflayer03                            1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ tiamat                                  1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ vecna                                   1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ zariel                                  1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  registry                                                                   
  ├─ belial                                  1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer02                                                            
     ├─ cpu                               400.0m      0.0                    
     └─ pods                                 3.0      3.0                    
  traefik                                    2.0      2.0                    
  ├─ fierna                                  1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer02                            1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-alex-chan                                                             
  └─ belial                                                                  
     ├─ cpu                                  2.0      2.0                    
     ├─ memory                            32.0Gi   32.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-angela-albi                                                           
  └─ mindflayer03                                                            
     ├─ cpu                                 10.0     10.0                    
     ├─ memory                            10.0Gi   10.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-bastian-goldluecke                                                    
  └─ beholder02                                                              
     ├─ cpu                                  2.0      2.0                    
     ├─ memory                             1.0Gi    1.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-christoph-hanselka                                                    
  └─ belial                                                                  
     ├─ cpu                                  1.0      1.0                    
     ├─ memory                            32.0Gi   32.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-eduard-buss                                                           
  └─ demogorgon                                                              
     ├─ cpu                                  2.0     64.0                    
     ├─ ephemeral-storage                 30.0Gi   60.0Gi                    
     ├─ memory                            92.0Gi  100.0Gi                    
     ├─ nvidia.com/gpu                       2.0      2.0                    
     └─ pods                                 2.0      2.0                    
  user-felix-petersen                                                        
  ├─ demogorgon                                                              
  │  ├─ cpu                                  6.0      8.0                    
  │  ├─ memory                            50.0Gi  100.0Gi                    
  │  ├─ nvidia.com/gpu                       1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ vecna                                                                   
     ├─ cpu                                  4.0      4.0                    
     ├─ memory                            50.0Gi   50.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-giovanna-ratini                                                       
  └─ mindflayer02                                                            
     ├─ cpu                               100.0m      1.0                    
     ├─ memory                           100.0Mi    1.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-isaac-breinyn                                                         
  └─ mindflayer01                                                            
     ├─ cpu                               200.0m      1.0                    
     ├─ memory                           256.0Mi    1.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-isabela-sudac                                                         
  └─ mindflayer01                                                            
     ├─ cpu                               100.0m      1.0                    
     ├─ memory                           100.0Mi    1.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-julian-jandeleit                                                      
  └─ demogorgon                                                              
     ├─ cpu                                 48.0     48.0                    
     ├─ memory                            80.0Gi   80.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-julian-zimmermann                                                     
  ├─ asmodeus                                                                
  │  ├─ cpu                                128.0    128.0                    
  │  ├─ memory                             1.0Ti    1.0Ti                    
  │  ├─ nvidia.com/gpu                       1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ demogorgon                                                              
  │  ├─ cpu                                 20.0     20.0                    
  │  ├─ memory                             1.0Ti    1.0Ti                    
  │  ├─ nvidia.com/gpu                       4.0      4.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ zariel                                                                  
     ├─ cpu                                250.0    250.0                    
     ├─ memory                             1.9Ti    1.9Ti                    
     ├─ nvidia.com/gpu                       8.0      8.0                    
     └─ pods                                 1.0      1.0                    
  user-lucas-lecarpentier                                                    
  └─ fierna                                                                  
     ├─ cpu                                 10.0     16.0                    
     ├─ memory                            64.0Gi   64.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-mattia-montanari                                                      
  ├─ mindflayer01                                                            
  │  ├─ cpu                               100.0m      1.0                    
  │  ├─ memory                           512.0Mi    4.0Gi                    
  │  └─ pods                                 1.0      1.0                    
  └─ vecna                                                                   
     ├─ cpu                                 16.0     16.0                    
     ├─ memory                            32.0Gi   32.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-mike-battistella                                                      
  └─ fierna                                                                  
     ├─ cpu                                  5.0      6.0                    
     ├─ ephemeral-storage                256.0Mi    1.0Gi                    
     ├─ memory                            64.0Gi   64.0Gi                    
     ├─ nvidia.com/gpu                       2.0      2.0                    
     └─ pods                                 2.0      2.0                    
  user-mohsen-jenadeleh                                                      
  └─ asmodeus                                                                
     ├─ cpu                                 60.0     60.0                    
     ├─ memory                           200.0Gi  200.0Gi                    
     ├─ nvidia.com/gpu                       2.0      2.0                    
     └─ pods                                 2.0      2.0                    
  user-segun-aroyehun                                                        
  └─ belial                                                                  
     ├─ cpu                                 25.0     50.0                    
     ├─ ephemeral-storage                100.0Gi  150.0Gi                    
     ├─ memory                           250.0Gi  400.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-sifei-li                                                              
  ├─ asmodeus                                                                
  │  ├─ cpu                                 32.0     32.0                    
  │  ├─ memory                           256.0Gi  284.0Gi                    
  │  └─ pods                                 1.0      1.0                    
  └─ zariel                                                                  
     ├─ cpu                                  4.0      4.0                    
     ├─ memory                            64.0Gi   80.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-till-aust                                                             
  └─ fierna                                                                  
     ├─ cpu                                  1.0      1.0                    
     ├─ memory                           200.0Gi  200.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-ulas-bingoel                                                          
  └─ asmodeus                                                                
     ├─ cpu                                  8.0      8.0                    
     ├─ memory                            80.0Gi   80.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-urs-waldmann                                                          
  ├─ mindflayer01                                                            
  │  ├─ cpu                                 16.0     16.0                    
  │  ├─ memory                            64.0Gi  160.0Gi                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer03                                                            
     ├─ cpu                                 16.0     16.0                    
     ├─ memory                            64.0Gi  160.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-valentin-schmuker                                                     
  └─ mindflayer01                                                            
     ├─ cpu                                  2.0      2.0                    
     ├─ memory                             2.0Gi    2.0Gi                    
     └─ pods                                 1.0      1.0                    
  web                                                                        
  ├─ beholder01                              1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ beholder02                                                              
  │  ├─ cpu                               250.0m      1.0                    
  │  └─ pods                                 2.0      2.0                    
  ├─ beholder03                              1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer02                                                            
     ├─ cpu                               500.0m      0.0                    
     └─ pods                                 5.0      5.0                    




Ceph file system report

  cluster:
    id:     3fee6f38-ba9f-11ec-9328-e188936dcafd
    health: HEALTH_WARN
            1 clients failing to respond to cache pressure
 
  services:
    mon: 5 daemons, quorum beholder03,beholder01,beholder02,mindflayer02,mindflayer03 (age 2M) [leader: beholder03]
    mgr: mindflayer02.ympgrs(active, since 3M), standbys: beholder03.nprqzk, mindflayer03.rzdvrr, beholder02.akktmp, mindflayer01.mkuopd, beholder01.verxwn
    mds: 4/4 daemons up, 2 standby
    osd: 24 osds: 24 up (since 3M), 24 in (since 4M)
 
  data:
    volumes: 1/1 healthy
    pools:   3 pools, 545 pgs
    objects: 83.05M objects, 102 TiB
    usage:   205 TiB used, 76 TiB / 282 TiB avail
    pgs:     543 active+clean
             1   active+clean+scrubbing+deep
             1   active+clean+scrubbing
 
  io:
    client:   149 KiB/s wr, 0 op/s rd, 4 op/s wr
 
HEALTH_WARN 1 clients failing to respond to cache pressure
[WRN] MDS_CLIENT_RECALL: 1 clients failing to respond to cache pressure
    mds.cephfs.beholder02.ohjxrm(mds.0): Client asmodeus:asmodeus3 failing to respond to cache pressure client_id: 3450717




Etcd cluster

+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|     ENDPOINT     |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.1.1:4252 | 7126e7a3a9cc42ca |  3.4.30 |  156 MB |     false |      false |    349248 |  773419629 |          773419629 |        |
| 192.168.1.2:4252 | 39d72894bf6c7600 |  3.4.30 |  156 MB |      true |      false |    349248 |  773419629 |          773419629 |        |
| 192.168.1.3:4252 | bbf4a2b99c3fd692 |  3.4.30 |  156 MB |     false |      false |    349248 |  773419630 |          773419630 |        |
| 192.168.2.1:4252 | 5cb9997dd1c2246b |  3.4.30 |  156 MB |     false |      false |    349248 |  773419630 |          773419630 |        |
| 192.168.2.3:4252 |  cbc1cf89959ea4e |  3.4.30 |  156 MB |     false |      false |    349248 |  773419630 |          773419630 |        |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+




Detailed network health

API and web servers

beholder01
SSH port open yes
Report available yes
External interface up ok
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs
local raid mounted /raid
Test pod responding dnsutils-beholder01
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
beholder02
SSH port open yes
Report available yes
External interface up ok
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs
local raid mounted /raid
Test pod responding dnsutils-beholder02
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
beholder03
SSH port open yes
Report available yes
External interface up ok
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs
local raid mounted /raid
Test pod responding dnsutils-beholder03
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes




Ceph osd nodes

mindflayer01
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs
local raid mounted /raid
Test pod responding dnsutils-mindflayer01
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
mindflayer02
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs
local raid mounted /raid
Test pod responding dnsutils-mindflayer02
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
mindflayer03
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs
local raid mounted /raid
Test pod responding dnsutils-mindflayer03
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes




Compute nodes

vecna
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-vecna
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
kiaransalee
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-kiaransalee
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
belial
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-belial
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
fierna
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-fierna
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
demogorgon
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-demogorgon
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
tiamat
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-tiamat
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
asmodeus
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-asmodeus
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
zariel
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-zariel
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes




nVidia driver and GPU status

dretch

belial

Thu May 21 15:33:00 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Quadro RTX 6000                Off |   00000000:1B:00.0 Off |                  Off |
| 33%   33C    P8             21W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  Quadro RTX 6000                Off |   00000000:1C:00.0 Off |                  Off |
| 33%   32C    P8             27W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  Quadro RTX 6000                Off |   00000000:1D:00.0 Off |                  Off |
| 33%   33C    P8             23W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  Quadro RTX 6000                Off |   00000000:1E:00.0 Off |                  Off |
| 33%   34C    P8             23W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  Quadro RTX 6000                Off |   00000000:3D:00.0 Off |                  Off |
| 33%   33C    P8             29W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  Quadro RTX 6000                Off |   00000000:3F:00.0 Off |                  Off |
| 33%   33C    P8             25W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  Quadro RTX 6000                Off |   00000000:40:00.0 Off |                  Off |
| 33%   33C    P8             20W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  Quadro RTX 6000                Off |   00000000:41:00.0 Off |                  Off |
| 33%   34C    P8             21W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

fierna

Thu May 21 15:33:02 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Quadro RTX 6000                On  |   00000000:1B:00.0 Off |                  Off |
| 33%   32C    P8             32W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  Quadro RTX 6000                On  |   00000000:1C:00.0 Off |                  Off |
| 33%   32C    P8             29W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  Quadro RTX 6000                On  |   00000000:1D:00.0 Off |                  Off |
| 33%   35C    P8             19W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  Quadro RTX 6000                On  |   00000000:1E:00.0 Off |                  Off |
| 33%   32C    P8             22W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  Quadro RTX 6000                On  |   00000000:3D:00.0 Off |                  Off |
| 33%   32C    P8             31W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  Quadro RTX 6000                On  |   00000000:3F:00.0 Off |                  Off |
| 33%   33C    P8             21W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  Quadro RTX 6000                On  |   00000000:40:00.0 Off |                  Off |
| 33%   34C    P8             30W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  Quadro RTX 6000                On  |   00000000:41:00.0 Off |                  Off |
| 33%   33C    P8             31W /  260W |     786MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    7   N/A  N/A          801557      C   python                                  782MiB |
+-----------------------------------------------------------------------------------------+

tiamat

Thu May 21 15:33:04 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100-SXM4-40GB          On  |   00000000:01:00.0 Off |                    0 |
| N/A   28C    P0             53W /  400W |       0MiB /  40960MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A100-SXM4-40GB          On  |   00000000:41:00.0 Off |                    0 |
| N/A   28C    P0             56W /  400W |       0MiB /  40960MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A100-SXM4-40GB          On  |   00000000:81:00.0 Off |                    0 |
| N/A   27C    P0             53W /  400W |       0MiB /  40960MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A100-SXM4-40GB          On  |   00000000:C1:00.0 Off |                    0 |
| N/A   27C    P0             54W /  400W |       0MiB /  40960MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

vecna

Thu May 21 17:33:07 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla V100-SXM3-32GB           Off |   00000000:34:00.0 Off |                    0 |
| N/A   33C    P0             48W /  350W |       6MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  Tesla V100-SXM3-32GB           Off |   00000000:36:00.0 Off |                    0 |
| N/A   33C    P0             48W /  350W |       6MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  Tesla V100-SXM3-32GB           Off |   00000000:39:00.0 Off |                    0 |
| N/A   35C    P0             54W /  350W |       6MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  Tesla V100-SXM3-32GB           Off |   00000000:3B:00.0 Off |                    0 |
| N/A   36C    P0             51W /  350W |       6MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  Tesla V100-SXM3-32GB           Off |   00000000:57:00.0 Off |                    0 |
| N/A   33C    P0             48W /  350W |       6MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  Tesla V100-SXM3-32GB           Off |   00000000:59:00.0 Off |                    0 |
| N/A   36C    P0             53W /  350W |       6MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  Tesla V100-SXM3-32GB           Off |   00000000:5C:00.0 Off |                    0 |
| N/A   34C    P0             51W /  350W |       6MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  Tesla V100-SXM3-32GB           Off |   00000000:5E:00.0 Off |                    0 |
| N/A   38C    P0             53W /  350W |       6MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   8  Tesla V100-SXM3-32GB           Off |   00000000:B7:00.0 Off |                    0 |
| N/A   34C    P0             50W /  350W |       6MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   9  Tesla V100-SXM3-32GB           Off |   00000000:B9:00.0 Off |                    0 |
| N/A   33C    P0             49W /  350W |       6MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|  10  Tesla V100-SXM3-32GB           Off |   00000000:BC:00.0 Off |                    0 |
| N/A   36C    P0             51W /  350W |       6MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|  11  Tesla V100-SXM3-32GB           Off |   00000000:BE:00.0 Off |                    0 |
| N/A   38C    P0             49W /  350W |       6MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|  12  Tesla V100-SXM3-32GB           Off |   00000000:E0:00.0 Off |                    0 |
| N/A   36C    P0             48W /  350W |       6MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|  13  Tesla V100-SXM3-32GB           Off |   00000000:E2:00.0 Off |                    0 |
| N/A   36C    P0             49W /  350W |       6MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|  14  Tesla V100-SXM3-32GB           Off |   00000000:E5:00.0 Off |                    0 |
| N/A   39C    P0             51W /  350W |       6MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|  15  Tesla V100-SXM3-32GB           Off |   00000000:E7:00.0 Off |                    0 |
| N/A   38C    P0             49W /  350W |       6MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

asmodeus

Thu May 21 15:33:09 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100-SXM4-80GB          On  |   00000000:01:00.0 Off |                    0 |
| N/A   28C    P0             65W /  500W |       5MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A100-SXM4-80GB          On  |   00000000:41:00.0 Off |                    0 |
| N/A   33C    P0             93W /  500W |     143MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A100-SXM4-80GB          On  |   00000000:81:00.0 Off |                    0 |
| N/A   27C    P0             62W /  500W |       5MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A100-SXM4-80GB          On  |   00000000:C1:00.0 Off |                    0 |
| N/A   27C    P0             64W /  500W |       5MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    1   N/A  N/A          388245      C   python                                   12MiB |
|    1   N/A  N/A          388252      C   python                                   12MiB |
+-----------------------------------------------------------------------------------------+

zariel

Thu May 21 17:33:12 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100-SXM4-40GB          On  |   00000000:07:00.0 Off |                    0 |
| N/A   55C    P0            283W /  400W |   18367MiB /  40960MiB |     93%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A100-SXM4-40GB          On  |   00000000:0F:00.0 Off |                    0 |
| N/A   50C    P0            233W /  400W |   19085MiB /  40960MiB |     96%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A100-SXM4-40GB          On  |   00000000:47:00.0 Off |                    0 |
| N/A   48C    P0            267W /  400W |   19027MiB /  40960MiB |     93%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A100-SXM4-40GB          On  |   00000000:4E:00.0 Off |                    0 |
| N/A   51C    P0            283W /  400W |   18283MiB /  40960MiB |     93%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA A100-SXM4-40GB          On  |   00000000:87:00.0 Off |                    0 |
| N/A   65C    P0            146W /  400W |   17917MiB /  40960MiB |     92%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA A100-SXM4-40GB          On  |   00000000:90:00.0 Off |                    0 |
| N/A   62C    P0            200W /  400W |   18285MiB /  40960MiB |     93%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA A100-SXM4-40GB          On  |   00000000:B7:00.0 Off |                    0 |
| N/A   62C    P0            168W /  400W |   18655MiB /  40960MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA A100-SXM4-40GB          On  |   00000000:BD:00.0 Off |                    0 |
| N/A   62C    P0            176W /  400W |   18367MiB /  40960MiB |     93%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A         3468727      C   /usr/bin/python3                      18358MiB |
|    1   N/A  N/A         3468728      C   /usr/bin/python3                      19076MiB |
|    2   N/A  N/A         3468729      C   /usr/bin/python3                      19018MiB |
|    3   N/A  N/A         3468731      C   /usr/bin/python3                      18274MiB |
|    4   N/A  N/A         3468732      C   /usr/bin/python3                      17908MiB |
|    5   N/A  N/A         3468736      C   /usr/bin/python3                      18276MiB |
|    6   N/A  N/A         3468738      C   /usr/bin/python3                      18646MiB |
|    7   N/A  N/A         3468741      C   /usr/bin/python3                      18358MiB |
+-----------------------------------------------------------------------------------------+

demogorgon

Thu May 21 15:33:17 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A40                     Off |   00000000:01:00.0 Off |                    0 |
|  0%   33C    P8             36W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A40                     Off |   00000000:25:00.0 Off |                    0 |
|  0%   32C    P8             36W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A40                     Off |   00000000:41:00.0 Off |                    0 |
|  0%   33C    P8             34W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A40                     Off |   00000000:61:00.0 Off |                    0 |
|  0%   32C    P8             35W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA A40                     Off |   00000000:81:00.0 Off |                    0 |
|  0%   35C    P8             36W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA A40                     Off |   00000000:A1:00.0 Off |                    0 |
|  0%   33C    P8             36W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA A40                     Off |   00000000:C1:00.0 Off |                    0 |
|  0%   36C    P8             36W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA A40                     Off |   00000000:E1:00.0 Off |                    0 |
|  0%   35C    P8             34W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

kiaransalee

Thu May 21 15:33:19 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.08             Driver Version: 550.127.08     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:26:00.0 Off |                    0 |
| N/A   50C    P0            627W /  700W |   66280MiB /  81559MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H100 80GB HBM3          On  |   00000000:2F:00.0 Off |                    0 |
| N/A   64C    P0            671W /  700W |   70632MiB /  81559MiB |    100%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA H100 80GB HBM3          On  |   00000000:46:00.0 Off |                    0 |
| N/A   36C    P0             77W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA H100 80GB HBM3          On  |   00000000:54:00.0 Off |                    0 |
| N/A   30C    P0             78W /  700W |       4MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA H100 80GB HBM3          On  |   00000000:A6:00.0 Off |                    0 |
| N/A   82C    P0            643W /  700W |   21472MiB /  81559MiB |     89%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA H100 80GB HBM3          On  |   00000000:AF:00.0 Off |                    0 |
| N/A   48C    P0            487W /  700W |   23653MiB /  81559MiB |     70%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA H100 80GB HBM3          On  |   00000000:C6:00.0 Off |                   On |
| N/A   37C    P0            134W /  700W |   33934MiB /  81559MiB |     N/A      Default |
|                                         |                        |              Enabled |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA H100 80GB HBM3          On  |   00000000:CF:00.0 Off |                   On |
| N/A   26C    P0             76W /  700W |      89MiB /  81559MiB |     N/A      Default |
|                                         |                        |              Enabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| MIG devices:                                                                            |
+------------------+----------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |                     Memory-Usage |        Vol|      Shared           |
|      ID  ID  Dev |                       BAR1-Usage | SM     Unc| CE ENC DEC OFA JPG    |
|                  |                                  |        ECC|                       |
|==================+==================================+===========+=======================|
|  6    1   0   0  |              51MiB / 40320MiB    | 64      0 |  4   0    4    0    4 |
|                  |                 0MiB / 65535MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  6    2   0   1  |           33883MiB / 40320MiB    | 60      0 |  3   0    3    0    3 |
|                  |                 3MiB / 65535MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  7    1   0   0  |              38MiB / 40320MiB    | 60      0 |  3   0    3    0    3 |
|                  |                 0MiB / 65535MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  7    2   0   1  |              51MiB / 40320MiB    | 64      0 |  4   0    4    0    4 |
|                  |                 0MiB / 65535MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   2693272      C   /opt/conda/bin/python                       66270MiB |
|    1   N/A  N/A   3308777      C   VLLM::EngineCore                            70622MiB |
|    4   N/A  N/A   3301921      C   /home/jovyan/css_stu/.venv/bin/python3      21462MiB |
|    5   N/A  N/A   2962469      C   python                                      18908MiB |
|    5   N/A  N/A   3183680      C   python                                       4730MiB |
|    6    2    0    3286663      C   /opt/conda/bin/python                       33836MiB |
+-----------------------------------------------------------------------------------------+