Cluster status Sat, 21 Mar 2026 12:42:01 +0000 report from beholder02

Resource usage (overall)
Resource usage (by namespace)
Ceph file system status
Etcd cluster status
Detailed network health report
nVidia Driver and GPU reports
  Imp
  Dretch
  Belial
  Fierna
  Tiamat
  Vecna
  Asmodeus
  Zariel
  Demogorgon

Resource usage (overall)

 Resource                                                   Requested          Limit  Allocatable     Free 
  cpu                                                     (46%) 729.5    (53%) 840.7         1.6k    735.3 
  ├─ asmodeus                                             (39%) 100.3    (39%) 100.1        256.0    155.7 
  │  ├─ dnsutils-asmodeus                                      100.0m         100.0m                       
  │  ├─ gpu-a100-zsh-shm                                         32.0           32.0                       
  │  ├─ kube-router-hs9vg                                      250.0m            0.0                       
  │  ├─ step16-coolchic-gpu-run-asmodeus-a100-hq                 20.0           20.0                       
  │  ├─ step18-ftic-gpu-run-asmodeus-a100-hq-40cpu               40.0           40.0                       
  │  └─ ulas-bingoel-model-pod-5                                  8.0            8.0                       
  ├─ beholder01                                           (4%) 900.0m    (0%) 100.0m         24.0     23.1 
  │  ├─ dnsutils-beholder01                                    100.0m         100.0m                       
  │  ├─ kube-apiserver-beholder01                              250.0m            0.0                       
  │  ├─ kube-controller-manager-beholder01                     200.0m            0.0                       
  │  ├─ kube-router-bflkh                                      250.0m            0.0                       
  │  └─ kube-scheduler-beholder01                              100.0m            0.0                       
  ├─ beholder02                                             (13%) 3.2      (13%) 3.1         24.0     20.8 
  │  ├─ dnsutils-beholder02                                    100.0m         100.0m                       
  │  ├─ kube-apiserver-beholder02                              250.0m            0.0                       
  │  ├─ kube-controller-manager-beholder02                     200.0m            0.0                       
  │  ├─ kube-router-7ngm5                                      250.0m            0.0                       
  │  ├─ kube-scheduler-beholder02                              100.0m            0.0                       
  │  ├─ mc-rj-ddcb7fb6-b2qh6                                      2.0            2.0                       
  │  ├─ nginx-rsn-2024-57d49484d-47rfc                         250.0m            1.0                       
  │  ├─ virt-api-85578d9bb7-5fwdl                                5.0m            0.0                       
  │  ├─ virt-controller-674bcccb6-pvgxj                         10.0m            0.0                       
  │  └─ virt-operator-b4d8f7f58-6jjxp                           10.0m            0.0                       
  ├─ beholder03                                              (6%) 1.4    (2%) 400.0m         24.0     22.6 
  │  ├─ coredns-66bc5c9577-d6czh                               100.0m            0.0                       
  │  ├─ dnsutils-beholder03                                    100.0m         100.0m                       
  │  ├─ kube-apiserver-beholder03                              250.0m            0.0                       
  │  ├─ kube-controller-manager-beholder03                     200.0m            0.0                       
  │  ├─ kube-router-nd2v4                                      250.0m            0.0                       
  │  ├─ kube-scheduler-beholder03                              100.0m            0.0                       
  │  ├─ ldap-67b47cf9b9-v6vvm                                  250.0m            0.0                       
  │  ├─ nfd-master-6589cf6d4c-9xw6v                            100.0m         300.0m                       
  │  ├─ virt-api-85578d9bb7-6mvxx                                5.0m            0.0                       
  │  ├─ virt-controller-674bcccb6-f2xfj                         10.0m            0.0                       
  │  └─ virt-operator-b4d8f7f58-pnbkz                           10.0m            0.0                       
  ├─ belial                                                (88%) 70.1     (89%) 71.1         80.0      8.9 
  │  ├─ cool-pod                                                  2.0            2.0                       
  │  ├─ coredns-66bc5c9577-kxlwp                               100.0m            0.0                       
  │  ├─ dnsutils-belial                                        100.0m         100.0m                       
  │  ├─ gatekeeper-controller-manager-66f474f785-bq2vs         100.0m            1.0                       
  │  ├─ kube-router-66wg4                                      250.0m            0.0                       
  │  ├─ nfd-gc-7b6c64c4b8-gc2k5                                 10.0m          20.0m                       
  │  ├─ ollama-pod                                                1.0            1.0                       
  │  ├─ prometheus-deployment-b65d5d898-q55j8                  500.0m            1.0                       
  │  ├─ pytorch-pod                                               1.0            1.0                       
  │  ├─ till-aust-baseline-arrowhead-job-pv7w7                   32.0           32.0                       
  │  ├─ till-aust-baseline-car-job-x7tlg                         32.0           32.0                       
  │  ├─ till-aust-ubuntu-entry-pod                                1.0            1.0                       
  │  └─ virt-handler-rvzhr                                      10.0m            0.0                       
  ├─ demogorgon                                            (80%) 76.3   (146%) 140.1         96.0      0.0 
  │  ├─ a2v2-single-gpu-jcsz                                      8.0            8.0                       
  │  ├─ demogorgon-a2v2                                          12.0           12.0                       
  │  ├─ dnsutils-demogorgon                                    100.0m         100.0m                       
  │  ├─ felix-petersen-job-29                                     6.0            8.0                       
  │  ├─ gpu-demogorgon                                           48.0           48.0                       
  │  ├─ kube-router-drgpn                                      250.0m            0.0                       
  │  ├─ pycharm                                                   1.0           32.0                       
  │  └─ pycharmv2                                                 1.0           32.0                       
  ├─ fierna                                                (87%) 69.6     (90%) 72.1         80.0      7.9 
  │  ├─ dnsutils-fierna                                        100.0m         100.0m                       
  │  ├─ gatekeeper-controller-manager-66f474f785-dvhjf         100.0m            1.0                       
  │  ├─ gatekeeper-controller-manager-66f474f785-s9kgj         100.0m            1.0                       
  │  ├─ kube-router-gnsmw                                      250.0m            0.0                       
  │  ├─ ledavio-similarity-search-9b864cc89-ln9lk                 4.0            4.0                       
  │  ├─ ledavio-text-search-5d8f755795-cndsv                      1.0            2.0                       
  │  ├─ till-aust-baseline-adiac-job-66j56                       32.0           32.0                       
  │  ├─ till-aust-baseline-beef-job-ts4jw                        32.0           32.0                       
  │  └─ virt-handler-xmxj8                                      10.0m            0.0                       
  ├─ kiaransalee                                           (19%) 37.4     (32%) 62.1        192.0    129.9 
  │  ├─ bash-pod                                                 12.0           12.0                       
  │  ├─ dnsutils-kiaransalee                                   100.0m         100.0m                       
  │  ├─ kube-router-pkxjt                                      250.0m            0.0                       
  │  └─ ubuntu-gpu1                                              25.0           50.0                       
  ├─ mindflayer01                                          (64%) 41.0     (64%) 41.1         64.0     22.9 
  │  ├─ cdi-apiserver-7745487599-s5g6s                         100.0m            0.0                       
  │  ├─ cdi-deployment-6c99bb8fcf-h79j8                        100.0m            0.0                       
  │  ├─ cdi-uploadproxy-684cf5d896-9x5rh                       100.0m            0.0                       
  │  ├─ dnsutils-mindflayer01                                  100.0m         100.0m                       
  │  ├─ file-pod                                               200.0m            1.0                       
  │  ├─ kube-router-6tdlp                                      250.0m            0.0                       
  │  ├─ till-aust-fireflies-transition-job-gt9jd                 20.0           21.0                       
  │  ├─ ubuntu-test-pod                                        100.0m            1.0                       
  │  ├─ urs-waldmann-tb-access-pod                               16.0           16.0                       
  │  ├─ valentin-schmuker-storage                                 2.0            2.0                       
  │  ├─ virt-handler-tqrj4                                      10.0m            0.0                       
  │  └─ virt-launcher-lightfield-analysis-8zbf6                   2.0          15.0m                       
  ├─ mindflayer02                                            (3%) 1.9       (3%) 2.1         64.0     61.9 
  │  ├─ dex-69dddb47b7-mg5r5                                   250.0m            0.0                       
  │  ├─ dex-loginapp-5f5974b54d-96bsg                          100.0m            0.0                       
  │  ├─ dex-mysql-589f4586bc-4n5vj                             100.0m            0.0                       
  │  ├─ dnsutils-mindflayer02                                  100.0m         100.0m                       
  │  ├─ gatekeeper-audit-59d4b6fd4c-gtwjm                      100.0m            1.0                       
  │  ├─ kube-router-stlpt                                      250.0m            0.0                       
  │  ├─ mediawiki-77f9c84df5-p6k9g                             250.0m            0.0                       
  │  ├─ mediawiki-mariadb-7ffb6c9b8d-ng7s6                     250.0m            0.0                       
  │  ├─ registry-7495ddbf59-zs5ph                              100.0m            0.0                       
  │  ├─ registry-auth-bd4bb7d8b-cdjnd                          100.0m            0.0                       
  │  ├─ registry-browser-7f4cbdf96b-d4b66                      200.0m            0.0                       
  │  └─ ubuntu-test-pod                                        100.0m            1.0                       
  ├─ mindflayer03                                          (73%) 46.5     (72%) 46.1         64.0     17.5 
  │  ├─ cdi-operator-76f7d8c545-vrql7                          100.0m            0.0                       
  │  ├─ dnsutils-mindflayer03                                  100.0m         100.0m                       
  │  ├─ gpu-pod-aalbi                                            10.0           10.0                       
  │  ├─ kube-router-d5x2v                                      250.0m            0.0                       
  │  ├─ till-aust-fireflies-k-job-mj9wm                          20.0           20.0                       
  │  ├─ urs-waldmann-ubuntu-pod                                  16.0           16.0                       
  │  └─ virt-handler-hndfm                                      10.0m            0.0                       
  ├─ tiamat                                                 (4%) 10.3      (6%) 16.1        256.0    239.9 
  │  ├─ dnsutils-tiamat                                        100.0m         100.0m                       
  │  ├─ kube-router-55pc2                                      250.0m            0.0                       
  │  └─ train-octfishy-filtered-split-p5gnw                      10.0           16.0                       
  ├─ vecna                                                 (21%) 20.4     (38%) 36.1         96.0     59.9 
  │  ├─ dnsutils-vecna                                         100.0m         100.0m                       
  │  ├─ felix-petersen-job-20                                     4.0            4.0                       
  │  ├─ kube-router-d9tsf                                      250.0m            0.0                       
  │  ├─ training-job-dataset2-6xmj6                               4.0            8.0                       
  │  ├─ training-job-dataset4-tcz9x                               4.0            8.0                       
  │  ├─ training-job-dataset6-66xjw                               4.0            8.0                       
  │  └─ training-job-dataset8-7ncgm                               4.0            8.0                       
  └─ zariel                                               (98%) 250.3    (98%) 250.1        256.0      5.7 
     ├─ dnsutils-zariel                                        100.0m         100.0m                       
     ├─ kube-router-v2x67                                      250.0m            0.0                       
     └─ zariel-a2v-sifaka-pt-ft-jcsz                            250.0          250.0                       
  devices.kubevirt.io/kvm                                    (0%) 1.0       (0%) 1.0         4.0k     4.0k 
  ├─ belial                                                  (0%) 0.0       (0%) 0.0         1.0k     1.0k 
  ├─ fierna                                                  (0%) 0.0       (0%) 0.0         1.0k     1.0k 
  ├─ mindflayer01                                            (0%) 1.0       (0%) 1.0         1.0k    999.0 
  │  └─ virt-launcher-lightfield-analysis-8zbf6                   1.0            1.0                       
  └─ mindflayer03                                            (0%) 0.0       (0%) 0.0         1.0k     1.0k 
  devices.kubevirt.io/tun                                    (0%) 1.0       (0%) 1.0         4.0k     4.0k 
  ├─ belial                                                  (0%) 0.0       (0%) 0.0         1.0k     1.0k 
  ├─ fierna                                                  (0%) 0.0       (0%) 0.0         1.0k     1.0k 
  ├─ mindflayer01                                            (0%) 1.0       (0%) 1.0         1.0k    999.0 
  │  └─ virt-launcher-lightfield-analysis-8zbf6                   1.0            1.0                       
  └─ mindflayer03                                            (0%) 0.0       (0%) 0.0         1.0k     1.0k 
  devices.kubevirt.io/vhost-net                              (0%) 1.0       (0%) 1.0         4.0k     4.0k 
  ├─ belial                                                  (0%) 0.0       (0%) 0.0         1.0k     1.0k 
  ├─ fierna                                                  (0%) 0.0       (0%) 0.0         1.0k     1.0k 
  ├─ mindflayer01                                            (0%) 1.0       (0%) 1.0         1.0k    999.0 
  │  └─ virt-launcher-lightfield-analysis-8zbf6                   1.0            1.0                       
  └─ mindflayer03                                            (0%) 0.0       (0%) 0.0         1.0k     1.0k 
  ephemeral-storage                                        (0%) 54.0G   (1%) 101.0Gi        11.3T    11.2T 
  ├─ asmodeus                                                (0%) 0.0       (0%) 0.0        94.6G    94.6G 
  ├─ beholder01                                              (0%) 0.0       (0%) 0.0         1.7T     1.7T 
  ├─ beholder02                                              (0%) 0.0       (0%) 0.0         1.7T     1.7T 
  ├─ beholder03                                              (0%) 0.0       (0%) 0.0         1.7T     1.7T 
  ├─ belial                                                  (0%) 0.0       (0%) 0.0       189.2G   189.2G 
  ├─ demogorgon                                           (5%) 30.0Gi    (9%) 60.0Gi       706.7G   642.3G 
  │  ├─ pycharm                                                15.0Gi         30.0Gi                       
  │  └─ pycharmv2                                              15.0Gi         30.0Gi                       
  ├─ fierna                                              (0%) 256.0Mi     (1%) 1.0Gi       189.2G   188.1G 
  │  └─ ledavio-similarity-search-9b864cc89-ln9lk             256.0Mi          1.0Gi                       
  ├─ kiaransalee                                          (1%) 20.0Gi    (3%) 40.0Gi         1.7T     1.7T 
  │  └─ ubuntu-gpu1                                            20.0Gi         40.0Gi                       
  ├─ mindflayer01                                          (0%) 50.0M       (0%) 0.0       211.5G   211.5G 
  │  └─ virt-launcher-lightfield-analysis-8zbf6                 50.0M            0.0                       
  ├─ mindflayer02                                            (0%) 0.0       (0%) 0.0       211.5G   211.5G 
  ├─ mindflayer03                                            (0%) 0.0       (0%) 0.0       211.5G   211.5G 
  ├─ tiamat                                                  (0%) 0.0       (0%) 0.0       164.4G   164.4G 
  ├─ vecna                                                   (0%) 0.0       (0%) 0.0       849.0G   849.0G 
  └─ zariel                                                  (0%) 0.0       (0%) 0.0         1.7T     1.7T 
  memory                                                   (34%) 4.7T     (38%) 5.3T       12.7Ti    7.9Ti 
  ├─ asmodeus                                           (27%) 536.3Gi  (28%) 564.1Gi        2.0Ti    1.4Ti 
  │  ├─ dnsutils-asmodeus                                     100.0Mi        100.0Mi                       
  │  ├─ gpu-a100-zsh-shm                                      256.0Gi        284.0Gi                       
  │  ├─ kube-router-hs9vg                                     250.0Mi            0.0                       
  │  ├─ step16-coolchic-gpu-run-asmodeus-a100-hq              100.0Gi        100.0Gi                       
  │  ├─ step18-ftic-gpu-run-asmodeus-a100-hq-40cpu            100.0Gi        100.0Gi                       
  │  └─ ulas-bingoel-model-pod-5                               80.0Gi         80.0Gi                       
  ├─ beholder01                                          (0%) 350.0Mi   (0%) 100.0Mi       92.9Gi   92.5Gi 
  │  ├─ dnsutils-beholder01                                   100.0Mi        100.0Mi                       
  │  └─ kube-router-bflkh                                     250.0Mi            0.0                       
  ├─ beholder02                                            (3%) 2.5Gi     (1%) 1.1Gi       92.9Gi   90.3Gi 
  │  ├─ dnsutils-beholder02                                   100.0Mi        100.0Mi                       
  │  ├─ kube-router-7ngm5                                     250.0Mi            0.0                       
  │  ├─ mc-rj-ddcb7fb6-b2qh6                                    1.0Gi          1.0Gi                       
  │  ├─ virt-api-85578d9bb7-5fwdl                             500.0Mi            0.0                       
  │  ├─ virt-controller-674bcccb6-pvgxj                       275.0Mi            0.0                       
  │  └─ virt-operator-b4d8f7f58-6jjxp                         450.0Mi            0.0                       
  ├─ beholder03                                            (2%) 1.7Gi     (5%) 4.3Gi       92.9Gi   88.6Gi 
  │  ├─ coredns-66bc5c9577-d6czh                               70.0Mi        170.0Mi                       
  │  ├─ dnsutils-beholder03                                   100.0Mi        100.0Mi                       
  │  ├─ kube-router-nd2v4                                     250.0Mi            0.0                       
  │  ├─ nfd-master-6589cf6d4c-9xw6v                           128.0Mi          4.0Gi                       
  │  ├─ virt-api-85578d9bb7-6mvxx                             500.0Mi            0.0                       
  │  ├─ virt-controller-674bcccb6-f2xfj                       275.0Mi            0.0                       
  │  └─ virt-operator-b4d8f7f58-pnbkz                         450.0Mi            0.0                       
  ├─ belial                                              (38%) 308.8G  (44%) 328.8Gi      754.4Gi  425.7Gi 
  │  ├─ cool-pod                                               32.0Gi         32.0Gi                       
  │  ├─ coredns-66bc5c9577-kxlwp                               70.0Mi        170.0Mi                       
  │  ├─ dnsutils-belial                                       100.0Mi        100.0Mi                       
  │  ├─ gatekeeper-controller-manager-66f474f785-bq2vs        256.0Mi        512.0Mi                       
  │  ├─ kube-router-66wg4                                     250.0Mi            0.0                       
  │  ├─ nfd-gc-7b6c64c4b8-gc2k5                               128.0Mi          1.0Gi                       
  │  ├─ ollama-pod                                             32.0Gi         32.0Gi                       
  │  ├─ prometheus-deployment-b65d5d898-q55j8                  500.0M          1.0Gi                       
  │  ├─ pytorch-pod                                            12.0Gi         12.0Gi                       
  │  ├─ till-aust-baseline-arrowhead-job-pv7w7                100.0Gi        120.0Gi                       
  │  ├─ till-aust-baseline-car-job-x7tlg                      100.0Gi        120.0Gi                       
  │  ├─ till-aust-ubuntu-entry-pod                             10.0Gi         10.0Gi                       
  │  └─ virt-handler-rvzhr                                    325.0Mi            0.0                       
  ├─ demogorgon                                         (39%) 778.3Gi  (42%) 836.1Gi        2.0Ti    1.1Ti 
  │  ├─ a2v2-single-gpu-jcsz                                  256.0Gi        256.0Gi                       
  │  ├─ demogorgon-a2v2                                       300.0Gi        300.0Gi                       
  │  ├─ dnsutils-demogorgon                                   100.0Mi        100.0Mi                       
  │  ├─ felix-petersen-job-29                                  50.0Gi        100.0Gi                       
  │  ├─ gpu-demogorgon                                         80.0Gi         80.0Gi                       
  │  ├─ kube-router-drgpn                                     250.0Mi            0.0                       
  │  ├─ pycharm                                                46.0Gi         50.0Gi                       
  │  └─ pycharmv2                                              46.0Gi         50.0Gi                       
  ├─ fierna                                             (35%) 265.2Gi  (40%) 305.1Gi      754.4Gi  449.3Gi 
  │  ├─ dnsutils-fierna                                       100.0Mi        100.0Mi                       
  │  ├─ gatekeeper-controller-manager-66f474f785-dvhjf        256.0Mi        512.0Mi                       
  │  ├─ gatekeeper-controller-manager-66f474f785-s9kgj        256.0Mi        512.0Mi                       
  │  ├─ kube-router-gnsmw                                     250.0Mi            0.0                       
  │  ├─ ledavio-similarity-search-9b864cc89-ln9lk              32.0Gi         32.0Gi                       
  │  ├─ ledavio-text-search-5d8f755795-cndsv                   32.0Gi         32.0Gi                       
  │  ├─ till-aust-baseline-adiac-job-66j56                    100.0Gi        120.0Gi                       
  │  ├─ till-aust-baseline-beef-job-ts4jw                     100.0Gi        120.0Gi                       
  │  └─ virt-handler-xmxj8                                    325.0Mi            0.0                       
  ├─ kiaransalee                                          (8%) 129.2G  (11%) 166.1Gi        1.5Ti    1.3Ti 
  │  ├─ bash-pod                                               16.0Gi         16.0Gi                       
  │  ├─ dnsutils-kiaransalee                                  100.0Mi        100.0Mi                       
  │  ├─ jupyter-adrianruhe                                       1.1G            0.0                       
  │  ├─ jupyter-giordano-2ddemarzo                               1.1G            0.0                       
  │  ├─ jupyter-huygenssteiner                                   1.1G            0.0                       
  │  ├─ jupyter-samrauh                                          1.1G            0.0                       
  │  ├─ kube-router-pkxjt                                     250.0Mi            0.0                       
  │  └─ ubuntu-gpu1                                           100.0Gi        150.0Gi                       
  ├─ mindflayer01                                        (46%) 184.8G   (76%) 308.3G      376.5Gi   89.3Gi 
  │  ├─ cdi-apiserver-7745487599-s5g6s                        150.0Mi            0.0                       
  │  ├─ cdi-deployment-6c99bb8fcf-h79j8                       150.0Mi            0.0                       
  │  ├─ cdi-uploadproxy-684cf5d896-9x5rh                      150.0Mi            0.0                       
  │  ├─ dnsutils-mindflayer01                                 100.0Mi        100.0Mi                       
  │  ├─ file-pod                                              256.0Mi          1.0Gi                       
  │  ├─ kube-router-6tdlp                                     250.0Mi            0.0                       
  │  ├─ till-aust-fireflies-transition-job-gt9jd              100.0Gi        120.0Gi                       
  │  ├─ ubuntu-test-pod                                       512.0Mi          4.0Gi                       
  │  ├─ urs-waldmann-tb-access-pod                             64.0Gi        160.0Gi                       
  │  ├─ valentin-schmuker-storage                               2.0Gi          2.0Gi                       
  │  ├─ virt-handler-tqrj4                                    325.0Mi            0.0                       
  │  └─ virt-launcher-lightfield-analysis-8zbf6                  4.6G          60.0M                       
  ├─ mindflayer02                                        (0%) 706.0Mi     (0%) 1.6Gi      376.5Gi  374.9Gi 
  │  ├─ dnsutils-mindflayer02                                 100.0Mi        100.0Mi                       
  │  ├─ gatekeeper-audit-59d4b6fd4c-gtwjm                     256.0Mi        512.0Mi                       
  │  ├─ kube-router-stlpt                                     250.0Mi            0.0                       
  │  └─ ubuntu-test-pod                                       100.0Mi          1.0Gi                       
  ├─ mindflayer03                                       (46%) 174.8Gi  (77%) 290.1Gi      376.5Gi   86.4Gi 
  │  ├─ cdi-operator-76f7d8c545-vrql7                         150.0Mi            0.0                       
  │  ├─ dnsutils-mindflayer03                                 100.0Mi        100.0Mi                       
  │  ├─ gpu-pod-aalbi                                          10.0Gi         10.0Gi                       
  │  ├─ kube-router-d5x2v                                     250.0Mi            0.0                       
  │  ├─ till-aust-fireflies-k-job-mj9wm                       100.0Gi        120.0Gi                       
  │  ├─ urs-waldmann-ubuntu-pod                                64.0Gi        160.0Gi                       
  │  └─ virt-handler-hndfm                                    325.0Mi            0.0                       
  ├─ tiamat                                               (3%) 30.3Gi    (3%) 30.1Gi     1007.6Gi  977.2Gi 
  │  ├─ dnsutils-tiamat                                       100.0Mi        100.0Mi                       
  │  ├─ kube-router-55pc2                                     250.0Mi            0.0                       
  │  └─ train-octfishy-filtered-split-p5gnw                    30.0Gi         30.0Gi                       
  ├─ vecna                                               (8%) 114.3Gi  (12%) 178.1Gi        1.5Ti    1.3Ti 
  │  ├─ dnsutils-vecna                                        100.0Mi        100.0Mi                       
  │  ├─ felix-petersen-job-20                                  50.0Gi         50.0Gi                       
  │  ├─ kube-router-d9tsf                                     250.0Mi            0.0                       
  │  ├─ training-job-dataset2-6xmj6                            16.0Gi         32.0Gi                       
  │  ├─ training-job-dataset4-tcz9x                            16.0Gi         32.0Gi                       
  │  ├─ training-job-dataset6-66xjw                            16.0Gi         32.0Gi                       
  │  └─ training-job-dataset8-7ncgm                            16.0Gi         32.0Gi                       
  └─ zariel                                               (95%) 1.9Ti    (95%) 1.9Ti        2.0Ti   95.2Gi 
     ├─ dnsutils-zariel                                       100.0Mi        100.0Mi                       
     ├─ kube-router-v2x67                                     250.0Mi            0.0                       
     └─ zariel-a2v-sifaka-pt-ft-jcsz                            1.9Ti          1.9Ti                       
  nvidia.com/gpu                                           (61%) 38.0     (61%) 38.0         62.0     24.0 
  ├─ asmodeus                                               (75%) 3.0      (75%) 3.0          4.0      1.0 
  │  ├─ step16-coolchic-gpu-run-asmodeus-a100-hq                  1.0            1.0                       
  │  ├─ step18-ftic-gpu-run-asmodeus-a100-hq-40cpu                1.0            1.0                       
  │  └─ ulas-bingoel-model-pod-5                                  1.0            1.0                       
  ├─ belial                                                 (62%) 5.0      (62%) 5.0          8.0      3.0 
  │  ├─ ollama-pod                                                1.0            1.0                       
  │  ├─ pytorch-pod                                               2.0            2.0                       
  │  ├─ till-aust-baseline-arrowhead-job-pv7w7                    1.0            1.0                       
  │  └─ till-aust-baseline-car-job-x7tlg                          1.0            1.0                       
  ├─ demogorgon                                            (100%) 8.0     (100%) 8.0          8.0      0.0 
  │  ├─ a2v2-single-gpu-jcsz                                      1.0            1.0                       
  │  ├─ demogorgon-a2v2                                           3.0            3.0                       
  │  ├─ felix-petersen-job-29                                     1.0            1.0                       
  │  ├─ gpu-demogorgon                                            1.0            1.0                       
  │  ├─ pycharm                                                   1.0            1.0                       
  │  └─ pycharmv2                                                 1.0            1.0                       
  ├─ fierna                                                 (50%) 4.0      (50%) 4.0          8.0      4.0 
  │  ├─ ledavio-similarity-search-9b864cc89-ln9lk                 1.0            1.0                       
  │  ├─ ledavio-text-search-5d8f755795-cndsv                      1.0            1.0                       
  │  ├─ till-aust-baseline-adiac-job-66j56                        1.0            1.0                       
  │  └─ till-aust-baseline-beef-job-ts4jw                         1.0            1.0                       
  ├─ kiaransalee                                            (67%) 4.0      (67%) 4.0          6.0      2.0 
  │  ├─ jupyter-adrianruhe                                        1.0            1.0                       
  │  ├─ jupyter-giordano-2ddemarzo                                1.0            1.0                       
  │  ├─ jupyter-huygenssteiner                                    1.0            1.0                       
  │  └─ ubuntu-gpu1                                               1.0            1.0                       
  ├─ tiamat                                                 (25%) 1.0      (25%) 1.0          4.0      3.0 
  │  └─ train-octfishy-filtered-split-p5gnw                       1.0            1.0                       
  ├─ vecna                                                  (31%) 5.0      (31%) 5.0         16.0     11.0 
  │  ├─ felix-petersen-job-20                                     1.0            1.0                       
  │  ├─ training-job-dataset2-6xmj6                               1.0            1.0                       
  │  ├─ training-job-dataset4-tcz9x                               1.0            1.0                       
  │  ├─ training-job-dataset6-66xjw                               1.0            1.0                       
  │  └─ training-job-dataset8-7ncgm                               1.0            1.0                       
  └─ zariel                                                (100%) 8.0     (100%) 8.0          8.0      0.0 
     └─ zariel-a2v-sifaka-pt-ft-jcsz                              8.0            8.0                       
  nvidia.com/mig-1g.10gb                                    (14%) 1.0      (14%) 1.0          7.0      6.0 
  └─ kiaransalee                                            (14%) 1.0      (14%) 1.0          7.0      6.0 
     └─ jupyter-samrauh                                           1.0            1.0                       
  nvidia.com/mig-3g.40gb                                     (0%) 0.0       (0%) 0.0          1.0      1.0 
  └─ kiaransalee                                             (0%) 0.0       (0%) 0.0          1.0      1.0 
  nvidia.com/mig-4g.40gb                                     (0%) 0.0       (0%) 0.0          1.0      1.0 
  └─ kiaransalee                                             (0%) 0.0       (0%) 0.0          1.0      1.0 
  pods                                                    (11%) 174.0    (11%) 174.0         1.5k     1.4k 
  ├─ asmodeus                                                (8%) 9.0       (8%) 9.0        110.0    101.0 
  │  ├─ dnsutils-asmodeus                                         1.0            1.0                       
  │  ├─ gpu-a100-zsh-shm                                          1.0            1.0                       
  │  ├─ gpu-feature-discovery-9wxd8                               1.0            1.0                       
  │  ├─ kube-proxy-kxtvf                                          1.0            1.0                       
  │  ├─ kube-router-hs9vg                                         1.0            1.0                       
  │  ├─ nvidia-device-plugin-daemonset-pgtkn                      1.0            1.0                       
  │  ├─ step16-coolchic-gpu-run-asmodeus-a100-hq                  1.0            1.0                       
  │  ├─ step18-ftic-gpu-run-asmodeus-a100-hq-40cpu                1.0            1.0                       
  │  └─ ulas-bingoel-model-pod-5                                  1.0            1.0                       
  ├─ beholder01                                              (6%) 7.0       (6%) 7.0        110.0    103.0 
  │  ├─ dnsutils-beholder01                                       1.0            1.0                       
  │  ├─ kube-apiserver-beholder01                                 1.0            1.0                       
  │  ├─ kube-controller-manager-beholder01                        1.0            1.0                       
  │  ├─ kube-proxy-jw87r                                          1.0            1.0                       
  │  ├─ kube-router-bflkh                                         1.0            1.0                       
  │  ├─ kube-scheduler-beholder01                                 1.0            1.0                       
  │  └─ vm-proxy-frontend-766dd9b967-swbkr                        1.0            1.0                       
  ├─ beholder02                                            (12%) 13.0     (12%) 13.0        110.0     97.0 
  │  ├─ dnsutils-beholder02                                       1.0            1.0                       
  │  ├─ hub-848f5d5578-47s2j                                      1.0            1.0                       
  │  ├─ kube-apiserver-beholder02                                 1.0            1.0                       
  │  ├─ kube-controller-manager-beholder02                        1.0            1.0                       
  │  ├─ kube-proxy-cmzf2                                          1.0            1.0                       
  │  ├─ kube-router-7ngm5                                         1.0            1.0                       
  │  ├─ kube-scheduler-beholder02                                 1.0            1.0                       
  │  ├─ mc-rj-ddcb7fb6-b2qh6                                      1.0            1.0                       
  │  ├─ nginx-ip-2025-7fd66b99dd-2khh6                            1.0            1.0                       
  │  ├─ nginx-rsn-2024-57d49484d-47rfc                            1.0            1.0                       
  │  ├─ virt-api-85578d9bb7-5fwdl                                 1.0            1.0                       
  │  ├─ virt-controller-674bcccb6-pvgxj                           1.0            1.0                       
  │  └─ virt-operator-b4d8f7f58-6jjxp                             1.0            1.0                       
  ├─ beholder03                                            (12%) 13.0     (12%) 13.0        110.0     97.0 
  │  ├─ coredns-66bc5c9577-d6czh                                  1.0            1.0                       
  │  ├─ dnsutils-beholder03                                       1.0            1.0                       
  │  ├─ kube-apiserver-beholder03                                 1.0            1.0                       
  │  ├─ kube-controller-manager-beholder03                        1.0            1.0                       
  │  ├─ kube-proxy-cftmz                                          1.0            1.0                       
  │  ├─ kube-router-nd2v4                                         1.0            1.0                       
  │  ├─ kube-scheduler-beholder03                                 1.0            1.0                       
  │  ├─ ldap-67b47cf9b9-v6vvm                                     1.0            1.0                       
  │  ├─ memcached-6b68cdd947-w4k2q                                1.0            1.0                       
  │  ├─ nfd-master-6589cf6d4c-9xw6v                               1.0            1.0                       
  │  ├─ virt-api-85578d9bb7-6mvxx                                 1.0            1.0                       
  │  ├─ virt-controller-674bcccb6-f2xfj                           1.0            1.0                       
  │  └─ virt-operator-b4d8f7f58-pnbkz                             1.0            1.0                       
  ├─ belial                                                (17%) 19.0     (17%) 19.0        110.0     91.0 
  │  ├─ cool-pod                                                  1.0            1.0                       
  │  ├─ coredns-66bc5c9577-kxlwp                                  1.0            1.0                       
  │  ├─ dnsutils-belial                                           1.0            1.0                       
  │  ├─ gatekeeper-controller-manager-66f474f785-bq2vs            1.0            1.0                       
  │  ├─ gpu-feature-discovery-x4hnt                               1.0            1.0                       
  │  ├─ hub-78d6dd898d-7f6kx                                      1.0            1.0                       
  │  ├─ kube-proxy-xxgbc                                          1.0            1.0                       
  │  ├─ kube-router-66wg4                                         1.0            1.0                       
  │  ├─ nfd-gc-7b6c64c4b8-gc2k5                                   1.0            1.0                       
  │  ├─ nvidia-device-plugin-daemonset-9nzsv                      1.0            1.0                       
  │  ├─ ollama-pod                                                1.0            1.0                       
  │  ├─ prometheus-deployment-b65d5d898-q55j8                     1.0            1.0                       
  │  ├─ pytorch-pod                                               1.0            1.0                       
  │  ├─ till-aust-baseline-arrowhead-job-pv7w7                    1.0            1.0                       
  │  ├─ till-aust-baseline-car-job-x7tlg                          1.0            1.0                       
  │  ├─ till-aust-ubuntu-entry-pod                                1.0            1.0                       
  │  ├─ user-scheduler-5cf5ffbc54-tjld9                           1.0            1.0                       
  │  ├─ virt-handler-rvzhr                                        1.0            1.0                       
  │  └─ whoami-74dc54d675-d6p8r                                   1.0            1.0                       
  ├─ demogorgon                                            (10%) 11.0     (10%) 11.0        110.0     99.0 
  │  ├─ a2v2-single-gpu-jcsz                                      1.0            1.0                       
  │  ├─ demogorgon-a2v2                                           1.0            1.0                       
  │  ├─ dnsutils-demogorgon                                       1.0            1.0                       
  │  ├─ felix-petersen-job-29                                     1.0            1.0                       
  │  ├─ gpu-demogorgon                                            1.0            1.0                       
  │  ├─ gpu-feature-discovery-2d8cr                               1.0            1.0                       
  │  ├─ kube-proxy-xdkh7                                          1.0            1.0                       
  │  ├─ kube-router-drgpn                                         1.0            1.0                       
  │  ├─ nvidia-device-plugin-daemonset-2b2z8                      1.0            1.0                       
  │  ├─ pycharm                                                   1.0            1.0                       
  │  └─ pycharmv2                                                 1.0            1.0                       
  ├─ fierna                                                (16%) 18.0     (16%) 18.0        110.0     92.0 
  │  ├─ dnsutils-fierna                                           1.0            1.0                       
  │  ├─ gatekeeper-controller-manager-66f474f785-dvhjf            1.0            1.0                       
  │  ├─ gatekeeper-controller-manager-66f474f785-s9kgj            1.0            1.0                       
  │  ├─ gpu-feature-discovery-k27g5                               1.0            1.0                       
  │  ├─ kube-proxy-pgw9t                                          1.0            1.0                       
  │  ├─ kube-router-gnsmw                                         1.0            1.0                       
  │  ├─ ledavio-similarity-search-9b864cc89-ln9lk                 1.0            1.0                       
  │  ├─ ledavio-text-search-5d8f755795-cndsv                      1.0            1.0                       
  │  ├─ nvidia-device-plugin-daemonset-rqv7h                      1.0            1.0                       
  │  ├─ proxy-5495d795d5-jjk7j                                    1.0            1.0                       
  │  ├─ proxy-5bc89cc587-fwm6j                                    1.0            1.0                       
  │  ├─ proxy-7f79cc645f-52qjx                                    1.0            1.0                       
  │  ├─ till-aust-baseline-adiac-job-66j56                        1.0            1.0                       
  │  ├─ till-aust-baseline-beef-job-ts4jw                         1.0            1.0                       
  │  ├─ user-scheduler-5cf5ffbc54-wrnrk                           1.0            1.0                       
  │  ├─ user-scheduler-c7db6c584-6vbss                            1.0            1.0                       
  │  ├─ virt-handler-xmxj8                                        1.0            1.0                       
  │  └─ whoami-74dc54d675-vsljt                                   1.0            1.0                       
  ├─ kiaransalee                                           (12%) 13.0     (12%) 13.0        110.0     97.0 
  │  ├─ bash-pod                                                  1.0            1.0                       
  │  ├─ continuous-image-puller-6fs4k                             1.0            1.0                       
  │  ├─ continuous-image-puller-6z8bj                             1.0            1.0                       
  │  ├─ dnsutils-kiaransalee                                      1.0            1.0                       
  │  ├─ gpu-feature-discovery-w8j9g                               1.0            1.0                       
  │  ├─ jupyter-adrianruhe                                        1.0            1.0                       
  │  ├─ jupyter-giordano-2ddemarzo                                1.0            1.0                       
  │  ├─ jupyter-huygenssteiner                                    1.0            1.0                       
  │  ├─ jupyter-samrauh                                           1.0            1.0                       
  │  ├─ kube-proxy-l8zn4                                          1.0            1.0                       
  │  ├─ kube-router-pkxjt                                         1.0            1.0                       
  │  ├─ nvidia-device-plugin-daemonset-8jskd                      1.0            1.0                       
  │  └─ ubuntu-gpu1                                               1.0            1.0                       
  ├─ mindflayer01                                          (13%) 14.0     (13%) 14.0        110.0     96.0 
  │  ├─ cdi-apiserver-7745487599-s5g6s                            1.0            1.0                       
  │  ├─ cdi-deployment-6c99bb8fcf-h79j8                           1.0            1.0                       
  │  ├─ cdi-uploadproxy-684cf5d896-9x5rh                          1.0            1.0                       
  │  ├─ dnsutils-mindflayer01                                     1.0            1.0                       
  │  ├─ file-pod                                                  1.0            1.0                       
  │  ├─ kube-proxy-49vx2                                          1.0            1.0                       
  │  ├─ kube-router-6tdlp                                         1.0            1.0                       
  │  ├─ nvidia-device-plugin-daemonset-jq8bq                      1.0            1.0                       
  │  ├─ till-aust-fireflies-transition-job-gt9jd                  1.0            1.0                       
  │  ├─ ubuntu-test-pod                                           1.0            1.0                       
  │  ├─ urs-waldmann-tb-access-pod                                1.0            1.0                       
  │  ├─ valentin-schmuker-storage                                 1.0            1.0                       
  │  ├─ virt-handler-tqrj4                                        1.0            1.0                       
  │  └─ virt-launcher-lightfield-analysis-8zbf6                   1.0            1.0                       
  ├─ mindflayer02                                          (23%) 25.0     (23%) 25.0        110.0     85.0 
  │  ├─ cert-manager-79559475b4-7kv54                             1.0            1.0                       
  │  ├─ cert-manager-cainjector-966fc8fbc-zql8j                   1.0            1.0                       
  │  ├─ cert-manager-webhook-854cf5f458-wwf4d                     1.0            1.0                       
  │  ├─ dex-69dddb47b7-mg5r5                                      1.0            1.0                       
  │  ├─ dex-loginapp-5f5974b54d-96bsg                             1.0            1.0                       
  │  ├─ dex-mysql-589f4586bc-4n5vj                                1.0            1.0                       
  │  ├─ dnsutils-mindflayer02                                     1.0            1.0                       
  │  ├─ gatekeeper-audit-59d4b6fd4c-gtwjm                         1.0            1.0                       
  │  ├─ kube-proxy-dfgxf                                          1.0            1.0                       
  │  ├─ kube-router-stlpt                                         1.0            1.0                       
  │  ├─ kube-state-metrics-8945855d-dqg79                         1.0            1.0                       
  │  ├─ local-path-provisioner-759479454f-7pqw8                   1.0            1.0                       
  │  ├─ mediawiki-77f9c84df5-p6k9g                                1.0            1.0                       
  │  ├─ mediawiki-mariadb-7ffb6c9b8d-ng7s6                        1.0            1.0                       
  │  ├─ nginx-k8s-5889449f8b-dv6xq                                1.0            1.0                       
  │  ├─ nginx-rec-2026-75dd946d4d-df9gc                           1.0            1.0                       
  │  ├─ nginx-self-service-password-54767ddc56-d556f              1.0            1.0                       
  │  ├─ nvidia-device-plugin-daemonset-h7f9j                      1.0            1.0                       
  │  ├─ pdf-55ccd6f459-s7nbj                                      1.0            1.0                       
  │  ├─ registry-7495ddbf59-zs5ph                                 1.0            1.0                       
  │  ├─ registry-auth-bd4bb7d8b-cdjnd                             1.0            1.0                       
  │  ├─ registry-browser-7f4cbdf96b-d4b66                         1.0            1.0                       
  │  ├─ traefik-deployment-d8ccbfdd4-z58bc                        1.0            1.0                       
  │  ├─ ubuntu-test-pod                                           1.0            1.0                       
  │  └─ user-scheduler-c7db6c584-2pxhd                            1.0            1.0                       
  ├─ mindflayer03                                            (8%) 9.0       (8%) 9.0        110.0    101.0 
  │  ├─ cdi-operator-76f7d8c545-vrql7                             1.0            1.0                       
  │  ├─ dnsutils-mindflayer03                                     1.0            1.0                       
  │  ├─ gpu-pod-aalbi                                             1.0            1.0                       
  │  ├─ kube-proxy-d2lmv                                          1.0            1.0                       
  │  ├─ kube-router-d5x2v                                         1.0            1.0                       
  │  ├─ nvidia-device-plugin-daemonset-c4wlg                      1.0            1.0                       
  │  ├─ till-aust-fireflies-k-job-mj9wm                           1.0            1.0                       
  │  ├─ urs-waldmann-ubuntu-pod                                   1.0            1.0                       
  │  └─ virt-handler-hndfm                                        1.0            1.0                       
  ├─ tiamat                                                  (6%) 7.0       (6%) 7.0        110.0    103.0 
  │  ├─ continuous-image-puller-wtkhd                             1.0            1.0                       
  │  ├─ dnsutils-tiamat                                           1.0            1.0                       
  │  ├─ gpu-feature-discovery-nmjfn                               1.0            1.0                       
  │  ├─ kube-proxy-n8m88                                          1.0            1.0                       
  │  ├─ kube-router-55pc2                                         1.0            1.0                       
  │  ├─ nvidia-device-plugin-daemonset-pqfxg                      1.0            1.0                       
  │  └─ train-octfishy-filtered-split-p5gnw                       1.0            1.0                       
  ├─ vecna                                                  (9%) 10.0      (9%) 10.0        110.0    100.0 
  │  ├─ dnsutils-vecna                                            1.0            1.0                       
  │  ├─ felix-petersen-job-20                                     1.0            1.0                       
  │  ├─ gpu-feature-discovery-8mn9m                               1.0            1.0                       
  │  ├─ kube-proxy-6r98t                                          1.0            1.0                       
  │  ├─ kube-router-d9tsf                                         1.0            1.0                       
  │  ├─ nvidia-device-plugin-daemonset-hv8rs                      1.0            1.0                       
  │  ├─ training-job-dataset2-6xmj6                               1.0            1.0                       
  │  ├─ training-job-dataset4-tcz9x                               1.0            1.0                       
  │  ├─ training-job-dataset6-66xjw                               1.0            1.0                       
  │  └─ training-job-dataset8-7ncgm                               1.0            1.0                       
  └─ zariel                                                  (5%) 6.0       (5%) 6.0        110.0    104.0 
     ├─ dnsutils-zariel                                           1.0            1.0                       
     ├─ gpu-feature-discovery-tpxz6                               1.0            1.0                       
     ├─ kube-proxy-gsqm7                                          1.0            1.0                       
     ├─ kube-router-v2x67                                         1.0            1.0                       
     ├─ nvidia-device-plugin-daemonset-4q45h                      1.0            1.0                       
     └─ zariel-a2v-sifaka-pt-ft-jcsz                              1.0            1.0                       




Resource usage by namespace

 Resource                              Requested    Limit  Allocatable  Free 
  auth                                                                       
  ├─ beholder03                                                              
  │  ├─ cpu                               250.0m      0.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer02                                                            
     ├─ cpu                               450.0m      0.0                    
     └─ pods                                 4.0      4.0                    
  cdi                                                                        
  ├─ mindflayer01                                                            
  │  ├─ cpu                                  2.3    15.0m                    
  │  ├─ devices.kubevirt.io/kvm              1.0      1.0                    
  │  ├─ devices.kubevirt.io/tun              1.0      1.0                    
  │  ├─ devices.kubevirt.io/vhost-net        1.0      1.0                    
  │  ├─ ephemeral-storage                  50.0M      0.0                    
  │  ├─ memory                              5.1G    60.0M                    
  │  └─ pods                                 4.0      4.0                    
  └─ mindflayer03                                                            
     ├─ cpu                               100.0m      0.0                    
     ├─ memory                           150.0Mi      0.0                    
     └─ pods                                 1.0      1.0                    
  cert-manager                               3.0      3.0                    
  └─ mindflayer02                            3.0      3.0                    
     └─ pods                                 3.0      3.0                    
  gatekeeper-system                                                          
  ├─ belial                                                                  
  │  ├─ cpu                               100.0m      1.0                    
  │  ├─ memory                           256.0Mi  512.0Mi                    
  │  └─ pods                                 1.0      1.0                    
  ├─ fierna                                                                  
  │  ├─ cpu                               200.0m      2.0                    
  │  ├─ memory                           512.0Mi    1.0Gi                    
  │  └─ pods                                 2.0      2.0                    
  └─ mindflayer02                                                            
     ├─ cpu                               100.0m      1.0                    
     ├─ memory                           256.0Mi  512.0Mi                    
     └─ pods                                 1.0      1.0                    
  jupyterhub                                                                 
  ├─ beholder02                              1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ fierna                                  2.0      2.0                    
  │  └─ pods                                 2.0      2.0                    
  ├─ kiaransalee                                                             
  │  ├─ memory                              3.2G      0.0                    
  │  ├─ nvidia.com/gpu                       3.0      3.0                    
  │  └─ pods                                 4.0      4.0                    
  └─ mindflayer02                            1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  jupyterhub-kuckling                        2.0      2.0                    
  ├─ fierna                                  1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ tiamat                                  1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  jupyterhub-students                                                        
  ├─ belial                                  2.0      2.0                    
  │  └─ pods                                 2.0      2.0                    
  ├─ fierna                                  2.0      2.0                    
  │  └─ pods                                 2.0      2.0                    
  └─ kiaransalee                                                             
     ├─ memory                              1.1G      0.0                    
     ├─ nvidia.com/mig-1g.10gb               1.0      1.0                    
     └─ pods                                 2.0      2.0                    
  kube-system                                                                
  ├─ asmodeus                                                                
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 5.0      5.0                    
  ├─ beholder01                                                              
  │  ├─ cpu                               900.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 6.0      6.0                    
  ├─ beholder02                                                              
  │  ├─ cpu                               900.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 6.0      6.0                    
  ├─ beholder03                                                              
  │  ├─ cpu                                  1.0   100.0m                    
  │  ├─ memory                           420.0Mi  270.0Mi                    
  │  └─ pods                                 7.0      7.0                    
  ├─ belial                                                                  
  │  ├─ cpu                               450.0m   100.0m                    
  │  ├─ memory                           420.0Mi  270.0Mi                    
  │  └─ pods                                 6.0      6.0                    
  ├─ demogorgon                                                              
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 5.0      5.0                    
  ├─ fierna                                                                  
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 5.0      5.0                    
  ├─ kiaransalee                                                             
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 5.0      5.0                    
  ├─ mindflayer01                                                            
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 4.0      4.0                    
  ├─ mindflayer02                                                            
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 4.0      4.0                    
  ├─ mindflayer03                                                            
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 4.0      4.0                    
  ├─ tiamat                                                                  
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 5.0      5.0                    
  ├─ vecna                                                                   
  │  ├─ cpu                               350.0m   100.0m                    
  │  ├─ memory                           350.0Mi  100.0Mi                    
  │  └─ pods                                 5.0      5.0                    
  └─ zariel                                                                  
     ├─ cpu                               350.0m   100.0m                    
     ├─ memory                           350.0Mi  100.0Mi                    
     └─ pods                                 5.0      5.0                    
  kubevirt                                                                   
  ├─ beholder02                                                              
  │  ├─ cpu                                25.0m      0.0                    
  │  ├─ memory                             1.2Gi      0.0                    
  │  └─ pods                                 3.0      3.0                    
  ├─ beholder03                                                              
  │  ├─ cpu                                25.0m      0.0                    
  │  ├─ memory                             1.2Gi      0.0                    
  │  └─ pods                                 3.0      3.0                    
  ├─ belial                                                                  
  │  ├─ cpu                                10.0m      0.0                    
  │  ├─ memory                           325.0Mi      0.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ fierna                                                                  
  │  ├─ cpu                                10.0m      0.0                    
  │  ├─ memory                           325.0Mi      0.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ mindflayer01                                                            
  │  ├─ cpu                                10.0m      0.0                    
  │  ├─ memory                           325.0Mi      0.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer03                                                            
     ├─ cpu                                10.0m      0.0                    
     ├─ memory                           325.0Mi      0.0                    
     └─ pods                                 1.0      1.0                    
  local-path-storage                         1.0      1.0                    
  └─ mindflayer02                            1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  monitoring                                                                 
  ├─ belial                                                                  
  │  ├─ cpu                               500.0m      1.0                    
  │  ├─ memory                            500.0M    1.0Gi                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer02                            1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  node-feature-discovery                                                     
  ├─ beholder03                                                              
  │  ├─ cpu                               100.0m   300.0m                    
  │  ├─ memory                           128.0Mi    4.0Gi                    
  │  └─ pods                                 1.0      1.0                    
  └─ belial                                                                  
     ├─ cpu                                10.0m    20.0m                    
     ├─ memory                           128.0Mi    1.0Gi                    
     └─ pods                                 1.0      1.0                    
  registry                                                                   
  ├─ belial                                  1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer02                                                            
     ├─ cpu                               400.0m      0.0                    
     └─ pods                                 3.0      3.0                    
  traefik                                    2.0      2.0                    
  ├─ fierna                                  1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer02                            1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-alex-chan                                                             
  └─ belial                                                                  
     ├─ cpu                                  2.0      2.0                    
     ├─ memory                            32.0Gi   32.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-andri-rutschmann                                                      
  └─ kiaransalee                                                             
     ├─ cpu                                 12.0     12.0                    
     ├─ memory                            16.0Gi   16.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-angela-albi                                                           
  ├─ mindflayer03                                                            
  │  ├─ cpu                                 10.0     10.0                    
  │  ├─ memory                            10.0Gi   10.0Gi                    
  │  └─ pods                                 1.0      1.0                    
  └─ tiamat                                                                  
     ├─ cpu                                 10.0     16.0                    
     ├─ memory                            30.0Gi   30.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-bastian-goldluecke                                                    
  └─ beholder02                                                              
     ├─ cpu                                  2.0      2.0                    
     ├─ memory                             1.0Gi    1.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-celine-angonin                                                        
  └─ demogorgon                                                              
     ├─ cpu                                 12.0     12.0                    
     ├─ memory                           300.0Gi  300.0Gi                    
     ├─ nvidia.com/gpu                       3.0      3.0                    
     └─ pods                                 1.0      1.0                    
  user-christoph-hanselka                                                    
  └─ belial                                                                  
     ├─ cpu                                  2.0      2.0                    
     ├─ memory                            44.0Gi   44.0Gi                    
     ├─ nvidia.com/gpu                       3.0      3.0                    
     └─ pods                                 2.0      2.0                    
  user-eduard-buss                                                           
  └─ demogorgon                                                              
     ├─ cpu                                  2.0     64.0                    
     ├─ ephemeral-storage                 30.0Gi   60.0Gi                    
     ├─ memory                            92.0Gi  100.0Gi                    
     ├─ nvidia.com/gpu                       2.0      2.0                    
     └─ pods                                 2.0      2.0                    
  user-felix-petersen                                                        
  ├─ demogorgon                                                              
  │  ├─ cpu                                  6.0      8.0                    
  │  ├─ memory                            50.0Gi  100.0Gi                    
  │  ├─ nvidia.com/gpu                       1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ vecna                                                                   
     ├─ cpu                                  4.0      4.0                    
     ├─ memory                            50.0Gi   50.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-giovanna-ratini                                                       
  └─ mindflayer02                                                            
     ├─ cpu                               100.0m      1.0                    
     ├─ memory                           100.0Mi    1.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-isaac-breinyn                                                         
  └─ mindflayer01                                                            
     ├─ cpu                               200.0m      1.0                    
     ├─ memory                           256.0Mi    1.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-julian-jandeleit                                                      
  └─ demogorgon                                                              
     ├─ cpu                                 48.0     48.0                    
     ├─ memory                            80.0Gi   80.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-julian-zimmermann                                                     
  ├─ demogorgon                                                              
  │  ├─ cpu                                  8.0      8.0                    
  │  ├─ memory                           256.0Gi  256.0Gi                    
  │  ├─ nvidia.com/gpu                       1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ zariel                                                                  
     ├─ cpu                                250.0    250.0                    
     ├─ memory                             1.9Ti    1.9Ti                    
     ├─ nvidia.com/gpu                       8.0      8.0                    
     └─ pods                                 1.0      1.0                    
  user-mattia-montanari                                                      
  └─ mindflayer01                                                            
     ├─ cpu                               100.0m      1.0                    
     ├─ memory                           512.0Mi    4.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-mike-battistella                                                      
  └─ fierna                                                                  
     ├─ cpu                                  5.0      6.0                    
     ├─ ephemeral-storage                256.0Mi    1.0Gi                    
     ├─ memory                            64.0Gi   64.0Gi                    
     ├─ nvidia.com/gpu                       2.0      2.0                    
     └─ pods                                 2.0      2.0                    
  user-mohsen-jenadeleh                                                      
  └─ asmodeus                                                                
     ├─ cpu                                 60.0     60.0                    
     ├─ memory                           200.0Gi  200.0Gi                    
     ├─ nvidia.com/gpu                       2.0      2.0                    
     └─ pods                                 2.0      2.0                    
  user-onur-oender                                                           
  └─ vecna                                                                   
     ├─ cpu                                 16.0     32.0                    
     ├─ memory                            64.0Gi  128.0Gi                    
     ├─ nvidia.com/gpu                       4.0      4.0                    
     └─ pods                                 4.0      4.0                    
  user-segun-aroyehun                                                        
  └─ kiaransalee                                                             
     ├─ cpu                                 25.0     50.0                    
     ├─ ephemeral-storage                 20.0Gi   40.0Gi                    
     ├─ memory                           100.0Gi  150.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-sifei-li                                                              
  └─ asmodeus                                                                
     ├─ cpu                                 32.0     32.0                    
     ├─ memory                           256.0Gi  284.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-till-aust                                                             
  ├─ belial                                                                  
  │  ├─ cpu                                 65.0     65.0                    
  │  ├─ memory                           210.0Gi  250.0Gi                    
  │  ├─ nvidia.com/gpu                       2.0      2.0                    
  │  └─ pods                                 3.0      3.0                    
  ├─ fierna                                                                  
  │  ├─ cpu                                 64.0     64.0                    
  │  ├─ memory                           200.0Gi  240.0Gi                    
  │  ├─ nvidia.com/gpu                       2.0      2.0                    
  │  └─ pods                                 2.0      2.0                    
  ├─ mindflayer01                                                            
  │  ├─ cpu                                 20.0     21.0                    
  │  ├─ memory                           100.0Gi  120.0Gi                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer03                                                            
     ├─ cpu                                 20.0     20.0                    
     ├─ memory                           100.0Gi  120.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-ulas-bingoel                                                          
  └─ asmodeus                                                                
     ├─ cpu                                  8.0      8.0                    
     ├─ memory                            80.0Gi   80.0Gi                    
     ├─ nvidia.com/gpu                       1.0      1.0                    
     └─ pods                                 1.0      1.0                    
  user-urs-waldmann                                                          
  ├─ mindflayer01                                                            
  │  ├─ cpu                                 16.0     16.0                    
  │  ├─ memory                            64.0Gi  160.0Gi                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer03                                                            
     ├─ cpu                                 16.0     16.0                    
     ├─ memory                            64.0Gi  160.0Gi                    
     └─ pods                                 1.0      1.0                    
  user-valentin-schmuker                                                     
  └─ mindflayer01                                                            
     ├─ cpu                                  2.0      2.0                    
     ├─ memory                             2.0Gi    2.0Gi                    
     └─ pods                                 1.0      1.0                    
  web                                                                        
  ├─ beholder01                              1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  ├─ beholder02                                                              
  │  ├─ cpu                               250.0m      1.0                    
  │  └─ pods                                 2.0      2.0                    
  ├─ beholder03                              1.0      1.0                    
  │  └─ pods                                 1.0      1.0                    
  └─ mindflayer02                                                            
     ├─ cpu                               500.0m      0.0                    
     └─ pods                                 5.0      5.0                    




Ceph file system report

  cluster:
    id:     3fee6f38-ba9f-11ec-9328-e188936dcafd
    health: HEALTH_OK
 
  services:
    mon: 5 daemons, quorum beholder03,beholder01,beholder02,mindflayer02,mindflayer03 (age 3w) [leader: beholder03]
    mgr: mindflayer02.ympgrs(active, since 4w), standbys: beholder03.nprqzk, mindflayer03.rzdvrr, beholder02.akktmp, mindflayer01.mkuopd, beholder01.verxwn
    mds: 4/4 daemons up, 2 standby
    osd: 24 osds: 24 up (since 4w), 24 in (since 10w)
 
  data:
    volumes: 1/1 healthy
    pools:   3 pools, 545 pgs
    objects: 82.12M objects, 102 TiB
    usage:   205 TiB used, 77 TiB / 282 TiB avail
    pgs:     543 active+clean
             2   active+clean+scrubbing+deep
 
  io:
    client:   1.4 MiB/s rd, 561 KiB/s wr, 1 op/s rd, 9 op/s wr
 
HEALTH_OK




Etcd cluster

+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|     ENDPOINT     |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.1.1:4252 | 7126e7a3a9cc42ca |  3.4.30 |  156 MB |     false |      false |    349248 |  733198401 |          733198401 |        |
| 192.168.1.2:4252 | 39d72894bf6c7600 |  3.4.30 |  156 MB |      true |      false |    349248 |  733198401 |          733198401 |        |
| 192.168.1.3:4252 | bbf4a2b99c3fd692 |  3.4.30 |  156 MB |     false |      false |    349248 |  733198401 |          733198401 |        |
| 192.168.2.1:4252 | 5cb9997dd1c2246b |  3.4.30 |  156 MB |     false |      false |    349248 |  733198401 |          733198401 |        |
| 192.168.2.3:4252 |  cbc1cf89959ea4e |  3.4.30 |  156 MB |     false |      false |    349248 |  733198402 |          733198402 |        |
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+




Detailed network health

API and web servers

beholder01
SSH port open yes
Report available yes
External interface up ok
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs
local raid mounted /raid
Test pod responding dnsutils-beholder01
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
beholder02
SSH port open yes
Report available yes
External interface up ok
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs
local raid mounted /raid
Test pod responding dnsutils-beholder02
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
beholder03
SSH port open yes
Report available yes
External interface up ok
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs
local raid mounted /raid
Test pod responding dnsutils-beholder03
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes




Ceph osd nodes

mindflayer01
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs
local raid mounted /raid
Test pod responding dnsutils-mindflayer01
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
mindflayer02
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs
local raid mounted /raid
Test pod responding dnsutils-mindflayer02
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
mindflayer03
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs
local raid mounted /raid
Test pod responding dnsutils-mindflayer03
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes




Compute nodes

vecna
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-vecna
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
kiaransalee
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-kiaransalee
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
belial
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-belial
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
fierna
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-fierna
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
demogorgon
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-demogorgon
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
tiamat
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-tiamat
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
asmodeus
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-asmodeus
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes
zariel
SSH port open yes
Report available yes
Infiniband interface up ok
API servers reachable 1 2 3 4
Ceph monitors reachable 1 2 3
cephfs mounted /cephfs/abyss
local raid mounted /raid
Test pod responding dnsutils-zariel
Can reach kube-dns 10.96.0.10
Pod can reach kube-dns yes
Pod can reach internet yes




nVidia driver and GPU status

dretch

belial

Sat Mar 21 12:42:50 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Quadro RTX 6000                Off |   00000000:1B:00.0 Off |                  Off |
| 33%   33C    P8              4W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  Quadro RTX 6000                Off |   00000000:1C:00.0 Off |                  Off |
| 33%   32C    P8             12W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  Quadro RTX 6000                Off |   00000000:1D:00.0 Off |                  Off |
| 33%   33C    P8              5W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  Quadro RTX 6000                Off |   00000000:1E:00.0 Off |                  Off |
| 33%   33C    P8              5W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  Quadro RTX 6000                Off |   00000000:3D:00.0 Off |                  Off |
| 33%   32C    P8             13W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  Quadro RTX 6000                Off |   00000000:3F:00.0 Off |                  Off |
| 33%   32C    P8              6W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  Quadro RTX 6000                Off |   00000000:40:00.0 Off |                  Off |
| 33%   32C    P8              4W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  Quadro RTX 6000                Off |   00000000:41:00.0 Off |                  Off |
| 33%   33C    P8              4W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

fierna

Sat Mar 21 12:42:52 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Quadro RTX 6000                On  |   00000000:1B:00.0 Off |                  Off |
| 33%   33C    P8             16W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  Quadro RTX 6000                On  |   00000000:1C:00.0 Off |                  Off |
| 33%   31C    P8             16W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  Quadro RTX 6000                On  |   00000000:1D:00.0 Off |                  Off |
| 33%   34C    P8              4W /  260W |    4128MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  Quadro RTX 6000                On  |   00000000:1E:00.0 Off |                  Off |
| 33%   32C    P8              4W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  Quadro RTX 6000                On  |   00000000:3D:00.0 Off |                  Off |
| 33%   31C    P8             15W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  Quadro RTX 6000                On  |   00000000:3F:00.0 Off |                  Off |
| 33%   32C    P8              4W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  Quadro RTX 6000                On  |   00000000:40:00.0 Off |                  Off |
| 33%   32C    P8             12W /  260W |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  Quadro RTX 6000                On  |   00000000:41:00.0 Off |                  Off |
| 33%   32C    P8             16W /  260W |     786MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    2   N/A  N/A         2012068      C   python                                 4124MiB |
|    7   N/A  N/A          801557      C   python                                  782MiB |
+-----------------------------------------------------------------------------------------+

tiamat

Sat Mar 21 12:42:54 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100-SXM4-40GB          On  |   00000000:01:00.0 Off |                    0 |
| N/A   28C    P0             50W /  400W |       0MiB /  40960MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A100-SXM4-40GB          On  |   00000000:41:00.0 Off |                    0 |
| N/A   30C    P0             53W /  400W |       0MiB /  40960MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A100-SXM4-40GB          On  |   00000000:81:00.0 Off |                    0 |
| N/A   33C    P0             69W /  400W |   31635MiB /  40960MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A100-SXM4-40GB          On  |   00000000:C1:00.0 Off |                    0 |
| N/A   26C    P0             50W /  400W |       0MiB /  40960MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    2   N/A  N/A         1832489      C   python                                31624MiB |
+-----------------------------------------------------------------------------------------+

vecna

Sat Mar 21 13:42:58 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla V100-SXM3-32GB           Off |   00000000:34:00.0 Off |                    0 |
| N/A   32C    P0             48W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  Tesla V100-SXM3-32GB           Off |   00000000:36:00.0 Off |                    0 |
| N/A   32C    P0             47W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  Tesla V100-SXM3-32GB           Off |   00000000:39:00.0 Off |                    0 |
| N/A   55C    P0            283W /  350W |    8838MiB /  32768MiB |    100%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  Tesla V100-SXM3-32GB           Off |   00000000:3B:00.0 Off |                    0 |
| N/A   34C    P0             50W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  Tesla V100-SXM3-32GB           Off |   00000000:57:00.0 Off |                    0 |
| N/A   32C    P0             48W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  Tesla V100-SXM3-32GB           Off |   00000000:59:00.0 Off |                    0 |
| N/A   36C    P0             52W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  Tesla V100-SXM3-32GB           Off |   00000000:5C:00.0 Off |                    0 |
| N/A   58C    P0            341W /  350W |    9020MiB /  32768MiB |     99%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  Tesla V100-SXM3-32GB           Off |   00000000:5E:00.0 Off |                    0 |
| N/A   71C    P0            358W /  350W |    8728MiB /  32768MiB |    100%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   8  Tesla V100-SXM3-32GB           Off |   00000000:B7:00.0 Off |                    0 |
| N/A   34C    P0             50W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   9  Tesla V100-SXM3-32GB           Off |   00000000:B9:00.0 Off |                    0 |
| N/A   33C    P0             49W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|  10  Tesla V100-SXM3-32GB           Off |   00000000:BC:00.0 Off |                    0 |
| N/A   36C    P0             51W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|  11  Tesla V100-SXM3-32GB           Off |   00000000:BE:00.0 Off |                    0 |
| N/A   38C    P0             49W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|  12  Tesla V100-SXM3-32GB           Off |   00000000:E0:00.0 Off |                    0 |
| N/A   63C    P0            329W /  350W |    9006MiB /  32768MiB |     99%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|  13  Tesla V100-SXM3-32GB           Off |   00000000:E2:00.0 Off |                    0 |
| N/A   37C    P0             49W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|  14  Tesla V100-SXM3-32GB           Off |   00000000:E5:00.0 Off |                    0 |
| N/A   54C    P0             56W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|  15  Tesla V100-SXM3-32GB           Off |   00000000:E7:00.0 Off |                    0 |
| N/A   39C    P0             49W /  350W |       0MiB /  32768MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    2   N/A  N/A         1847074      C   python                                 8834MiB |
|    6   N/A  N/A         1105071      C   python                                 9016MiB |
|    7   N/A  N/A         1116970      C   python                                 8724MiB |
|   12   N/A  N/A         1854113      C   python                                 9002MiB |
+-----------------------------------------------------------------------------------------+

asmodeus

Sat Mar 21 12:43:02 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100-SXM4-80GB          On  |   00000000:01:00.0 Off |                    0 |
| N/A   28C    P0             61W /  500W |       5MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A100-SXM4-80GB          On  |   00000000:41:00.0 Off |                    0 |
| N/A   30C    P0             70W /  500W |     143MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A100-SXM4-80GB          On  |   00000000:81:00.0 Off |                    0 |
| N/A   27C    P0             59W /  500W |       5MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A100-SXM4-80GB          On  |   00000000:C1:00.0 Off |                    0 |
| N/A   27C    P0             60W /  500W |       5MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    1   N/A  N/A          388245      C   python                                   12MiB |
|    1   N/A  N/A          388252      C   python                                   12MiB |
+-----------------------------------------------------------------------------------------+

zariel

Sat Mar 21 13:43:06 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100-SXM4-40GB          On  |   00000000:07:00.0 Off |                    0 |
| N/A   57C    P0            317W /  400W |   38547MiB /  40960MiB |     99%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A100-SXM4-40GB          On  |   00000000:0F:00.0 Off |                    0 |
| N/A   51C    P0            309W /  400W |   39165MiB /  40960MiB |     79%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A100-SXM4-40GB          On  |   00000000:47:00.0 Off |                    0 |
| N/A   50C    P0            296W /  400W |   39049MiB /  40960MiB |     87%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A100-SXM4-40GB          On  |   00000000:4E:00.0 Off |                    0 |
| N/A   52C    P0            307W /  400W |   39263MiB /  40960MiB |     77%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA A100-SXM4-40GB          On  |   00000000:87:00.0 Off |                    0 |
| N/A   68C    P0            318W /  400W |   39039MiB /  40960MiB |     99%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA A100-SXM4-40GB          On  |   00000000:90:00.0 Off |                    0 |
| N/A   65C    P0            328W /  400W |   39071MiB /  40960MiB |     78%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA A100-SXM4-40GB          On  |   00000000:B7:00.0 Off |                    0 |
| N/A   64C    P0            271W /  400W |   39083MiB /  40960MiB |     79%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA A100-SXM4-40GB          On  |   00000000:BD:00.0 Off |                    0 |
| N/A   62C    P0            313W /  400W |   38871MiB /  40960MiB |     99%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A         2080449      C   /usr/bin/python3                      38538MiB |
|    1   N/A  N/A         2080450      C   /usr/bin/python3                      39156MiB |
|    2   N/A  N/A         2080451      C   /usr/bin/python3                      39040MiB |
|    3   N/A  N/A         2080452      C   /usr/bin/python3                      39254MiB |
|    4   N/A  N/A         2080453      C   /usr/bin/python3                      39030MiB |
|    5   N/A  N/A         2080456      C   /usr/bin/python3                      39062MiB |
|    6   N/A  N/A         2080458      C   /usr/bin/python3                      39074MiB |
|    7   N/A  N/A         2080460      C   /usr/bin/python3                      38862MiB |
+-----------------------------------------------------------------------------------------+

demogorgon

Sat Mar 21 12:43:10 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A40                     Off |   00000000:01:00.0 Off |                    0 |
|  0%   30C    P8             24W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA A40                     Off |   00000000:25:00.0 Off |                    0 |
|  0%   30C    P8             24W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA A40                     Off |   00000000:41:00.0 Off |                    0 |
|  0%   29C    P8             24W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA A40                     Off |   00000000:61:00.0 Off |                    0 |
|  0%   28C    P8             24W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA A40                     Off |   00000000:81:00.0 Off |                    0 |
|  0%   26C    P8             16W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA A40                     Off |   00000000:A1:00.0 Off |                    0 |
|  0%   27C    P8             24W /  300W |       0MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA A40                     Off |   00000000:C1:00.0 Off |                    0 |
|  0%   76C    P0            298W /  300W |    7555MiB /  46068MiB |    100%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA A40                     Off |   00000000:E1:00.0 Off |                    0 |
|  0%   74C    P0            296W /  300W |   16803MiB /  46068MiB |     99%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    6   N/A  N/A          149195      C   python                                 7546MiB |
|    7   N/A  N/A         1057243      C   python                                16794MiB |
+-----------------------------------------------------------------------------------------+

kiaransalee

Sat Mar 21 12:43:13 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.08             Driver Version: 550.127.08     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:26:00.0 Off |                    0 |
| N/A   65C    P0            630W /  700W |   64404MiB /  81559MiB |     89%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H100 80GB HBM3          On  |   00000000:2F:00.0 Off |                    0 |
| N/A   33C    P0             92W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA H100 80GB HBM3          On  |   00000000:46:00.0 Off |                    0 |
| N/A   43C    P0             83W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA H100 80GB HBM3          On  |   00000000:54:00.0 Off |                    0 |
| N/A   31C    P0            100W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA H100 80GB HBM3          On  |   00000000:A6:00.0 Off |                    0 |
| N/A   47C    P0             85W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA H100 80GB HBM3          On  |   00000000:AF:00.0 Off |                    0 |
| N/A   32C    P0             76W /  700W |       1MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA H100 80GB HBM3          On  |   00000000:C6:00.0 Off |                   On |
| N/A   29C    P0             75W /  700W |      89MiB /  81559MiB |     N/A      Default |
|                                         |                        |              Enabled |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA H100 80GB HBM3          On  |   00000000:CF:00.0 Off |                   On |
| N/A   30C    P0            124W /  700W |     404MiB /  81559MiB |     N/A      Default |
|                                         |                        |              Enabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| MIG devices:                                                                            |
+------------------+----------------------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |                     Memory-Usage |        Vol|      Shared           |
|      ID  ID  Dev |                       BAR1-Usage | SM     Unc| CE ENC DEC OFA JPG    |
|                  |                                  |        ECC|                       |
|==================+==================================+===========+=======================|
|  6    1   0   0  |              51MiB / 40320MiB    | 64      0 |  4   0    4    0    4 |
|                  |                 0MiB / 65535MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  6    2   0   1  |              38MiB / 40320MiB    | 60      0 |  3   0    3    0    3 |
|                  |                 0MiB / 65535MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  7    7   0   0  |              13MiB /  9984MiB    | 16      0 |  1   0    1    0    1 |
|                  |                 0MiB / 16383MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  7    8   0   1  |              13MiB /  9984MiB    | 16      0 |  1   0    1    0    1 |
|                  |                 0MiB / 16383MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  7    9   0   2  |              13MiB /  9984MiB    | 16      0 |  1   0    1    0    1 |
|                  |                 0MiB / 16383MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  7   11   0   3  |              13MiB /  9984MiB    | 16      0 |  1   0    1    0    1 |
|                  |                 0MiB / 16383MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  7   12   0   4  |             327MiB /  9984MiB    | 16      0 |  1   0    1    0    1 |
|                  |                 2MiB / 16383MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  7   13   0   5  |              13MiB /  9984MiB    | 16      0 |  1   0    1    0    1 |
|                  |                 0MiB / 16383MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
|  7   14   0   6  |              13MiB /  9984MiB    | 16      0 |  1   0    1    0    1 |
|                  |                 0MiB / 16383MiB  |           |                       |
+------------------+----------------------------------+-----------+-----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   3518301      C   /opt/conda/bin/python                       64394MiB |
|    7   12    0    3751193      C   /home/jovyan/.drl/bin/python                  306MiB |
+-----------------------------------------------------------------------------------------+