CLI Demo
This notebook illustrates the basic usage of Vis4D. We run the provided Faster R-CNN toy config to run training and inference on COCO images.
Inference
Run inference on the validation set.
!vis4d test --config faster_rcnn_example.py --config.params.num_epochs 1
[08/07 03:11:01 Vis4D]: Environment info: PyTorch version: 2.7.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: macOS 15.6 (arm64)
GCC version: Could not collect
Clang version: 17.0.0 (clang-1700.0.13.5)
CMake version: Could not collect
Libc version: N/A
Python version: 3.12.9 | packaged by Anaconda, Inc. | (main, Feb 6 2025, 12:55:12) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-15.6-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Apple M4
Versions of relevant libraries:
[pip3] mypy_extensions==1.1.0
[pip3] numpy==1.26.4
[pip3] pytorch-lightning==2.5.2
[pip3] torch==2.7.1
[pip3] torchmetrics==1.7.4
[pip3] torchvision==0.22.1
[conda] numpy 1.26.4 pypi_0 pypi
[conda] pytorch-lightning 2.5.2 pypi_0 pypi
[conda] torch 2.7.1 pypi_0 pypi
[conda] torchmetrics 1.7.4 pypi_0 pypi
[conda] torchvision 0.22.1 pypi_0 pypi
GPU available: True (mps), used: False
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
/Users/royyang/anaconda3/envs/vis4d/lib/python3.12/site-packages/lightning/pytorch/trainer/setup.py:177: GPU available but not used. You can set it by doing `Trainer(accelerator='gpu')`.
[08/07 03:11:01 Vis4D]: Load checkpoint from http path: https://download.pytorch.org/models/resnet50-0676ba61.pth
Downloading: "https://download.pytorch.org/models/resnet50-0676ba61.pth" to /var/folders/xd/tqr32hx17vz95rkzbx76ngsh0000gn/T/tmpok5awk2d/.cache/torch/hub/checkpoints/resnet50-0676ba61.pth
100%|███████████████████████████████████████| 97.8M/97.8M [00:00<00:00, 166MB/s]
/Users/royyang/Workspace/vis4d/vis4d/common/ckpt.py:377: UserWarning: The model and loaded state dict do not match exactly
unexpected key in source state_dict: fc.weight, fc.bias
rank_zero_warn(err_msg)
[08/07 03:11:02 Vis4D]: Load checkpoint from http path: https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
Downloading: "https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth" to /var/folders/xd/tqr32hx17vz95rkzbx76ngsh0000gn/T/tmpok5awk2d/.cache/torch/hub/checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
100%|████████████████████████████████████████| 160M/160M [00:04<00:00, 34.1MB/s]
[08/07 03:11:07 Vis4D]: Generating COCODataset(root=data/coco_test, split=train, use_pascal_voc_cats=False) data mapping...
[08/07 03:11:07 Vis4D]: Loading COCODataset(root=data/coco_test, split=train, use_pascal_voc_cats=False) takes 0.00 seconds.
/Users/royyang/anaconda3/envs/vis4d/lib/python3.12/site-packages/lightning/pytorch/trainer/connectors/data_connector.py:425: The 'test_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=9` in the `DataLoader` to improve performance.
[08/07 03:11:08 Vis4D]: Testing: 1/2, ETA: 0:00:01, 1.14s/it
[08/07 03:11:09 Vis4D]: Testing: 2/2, ETA: 0:00:00, 1.01s/it
[08/07 03:11:09 Vis4D]: Running evaluator CocoEvaluator(annotation_path=data/coco_test/annotations/instances_train.json) with Det metric...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
[08/07 03:11:09 Vis4D]: Det/AP: 0.3974
[08/07 03:11:09 Vis4D]: Det/AP50: 0.5497
[08/07 03:11:09 Vis4D]: Det/AP75: 0.4414
[08/07 03:11:09 Vis4D]: Det/APs: 0.2868
[08/07 03:11:09 Vis4D]: Det/APm: 0.6932
[08/07 03:11:09 Vis4D]: Det/APl: 0.6000
[08/07 03:11:09 Vis4D]: Showing results for metric: Det
[08/07 03:11:09 Vis4D]:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.397
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.550
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.441
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.287
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.693
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.600
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.278
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.429
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.454
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.348
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.693
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.633
Training
Run the training for 1 epoch and inference on the validation set.
!vis4d fit --config faster_rcnn_example.py --config.params.num_epochs 1
[08/07 03:11:20 Vis4D]: Environment info: PyTorch version: 2.7.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: macOS 15.6 (arm64)
GCC version: Could not collect
Clang version: 17.0.0 (clang-1700.0.13.5)
CMake version: Could not collect
Libc version: N/A
Python version: 3.12.9 | packaged by Anaconda, Inc. | (main, Feb 6 2025, 12:55:12) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-15.6-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Apple M4
Versions of relevant libraries:
[pip3] mypy_extensions==1.1.0
[pip3] numpy==1.26.4
[pip3] pytorch-lightning==2.5.2
[pip3] torch==2.7.1
[pip3] torchmetrics==1.7.4
[pip3] torchvision==0.22.1
[conda] numpy 1.26.4 pypi_0 pypi
[conda] pytorch-lightning 2.5.2 pypi_0 pypi
[conda] torch 2.7.1 pypi_0 pypi
[conda] torchmetrics 1.7.4 pypi_0 pypi
[conda] torchvision 0.22.1 pypi_0 pypi
GPU available: True (mps), used: False
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
/Users/royyang/anaconda3/envs/vis4d/lib/python3.12/site-packages/lightning/pytorch/trainer/setup.py:177: GPU available but not used. You can set it by doing `Trainer(accelerator='gpu')`.
Seed set to 798931529
[08/07 03:11:20 Vis4D]: Global seed set to 798931529
[08/07 03:11:20 Vis4D]: Load checkpoint from http path: https://download.pytorch.org/models/resnet50-0676ba61.pth
Downloading: "https://download.pytorch.org/models/resnet50-0676ba61.pth" to /var/folders/xd/tqr32hx17vz95rkzbx76ngsh0000gn/T/tmpx4ximh4c/.cache/torch/hub/checkpoints/resnet50-0676ba61.pth
100%|███████████████████████████████████████| 97.8M/97.8M [00:00<00:00, 164MB/s]
/Users/royyang/Workspace/vis4d/vis4d/common/ckpt.py:377: UserWarning: The model and loaded state dict do not match exactly
unexpected key in source state_dict: fc.weight, fc.bias
rank_zero_warn(err_msg)
[08/07 03:11:21 Vis4D]: Load checkpoint from http path: https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
Downloading: "https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth" to /var/folders/xd/tqr32hx17vz95rkzbx76ngsh0000gn/T/tmpx4ximh4c/.cache/torch/hub/checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
100%|████████████████████████████████████████| 160M/160M [00:04<00:00, 33.8MB/s]
[08/07 03:11:26 Vis4D]: Generating COCODataset(root=data/coco_test, split=train, use_pascal_voc_cats=False) data mapping...
[08/07 03:11:26 Vis4D]: Loading COCODataset(root=data/coco_test, split=train, use_pascal_voc_cats=False) takes 0.00 seconds.
/Users/royyang/anaconda3/envs/vis4d/lib/python3.12/site-packages/lightning/pytorch/trainer/connectors/data_connector.py:425: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=9` in the `DataLoader` to improve performance.
/Users/royyang/anaconda3/envs/vis4d/lib/python3.12/site-packages/torch/utils/data/dataloader.py:683: UserWarning: 'pin_memory' argument is set as true but not supported on MPS now, then device pinned memory won't be used.
warnings.warn(warn_msg)
/Users/royyang/anaconda3/envs/vis4d/lib/python3.12/site-packages/lightning/pytorch/loops/fit_loop.py:310: The number of training batches (2) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
| Name | Type | Params | Mode
---------------------------------------------------
0 | loss_module | LossModule | 0 | train
1 | model | FasterRCNN | 41.8 M | train
---------------------------------------------------
41.5 M Trainable params
225 K Non-trainable params
41.8 M Total params
167.012 Total estimated model params size (MB)
161 Modules in train mode
29 Modules in eval mode
[08/07 03:11:26 Vis4D]: Generating COCODataset(root=data/coco_test, split=train, use_pascal_voc_cats=False) data mapping...
[08/07 03:11:26 Vis4D]: Loading COCODataset(root=data/coco_test, split=train, use_pascal_voc_cats=False) takes 0.00 seconds.
/Users/royyang/anaconda3/envs/vis4d/lib/python3.12/site-packages/lightning/pytorch/trainer/connectors/data_connector.py:425: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=9` in the `DataLoader` to improve performance.
[08/07 03:11:28 Vis4D]: Epoch 1: 1/2, ETA: 0:00:02, 2.72s/it, loss: 1.4686, RPNLoss.loss_cls: 0.4833, RPNLoss.loss_bbox: 0.2011, RCNNLoss.rcnn_loss_cls: 0.5931, RCNNLoss.rcnn_loss_bbox: 0.1911
[08/07 03:11:31 Vis4D]: Epoch 1: 2/2, ETA: 0:00:00, 2.65s/it, loss: 1.5583, RPNLoss.loss_cls: 0.1662, RPNLoss.loss_bbox: 0.0978, RCNNLoss.rcnn_loss_cls: 0.8646, RCNNLoss.rcnn_loss_bbox: 0.4296
[08/07 03:11:32 Vis4D]: Validation: 1/2, ETA: 0:00:00, 1.26it/s
[08/07 03:11:33 Vis4D]: Validation: 2/2, ETA: 0:00:00, 1.26it/s
[08/07 03:11:33 Vis4D]: Running evaluator CocoEvaluator(annotation_path=data/coco_test/annotations/instances_train.json) with Det metric...
Loading and preparing results...
DONE (t=0.00s)
creating index...
index created!
[08/07 03:11:33 Vis4D]: Det/AP: 0.3194
[08/07 03:11:33 Vis4D]: Det/AP50: 0.4682
[08/07 03:11:33 Vis4D]: Det/AP75: 0.3778
[08/07 03:11:33 Vis4D]: Det/APs: 0.2240
[08/07 03:11:33 Vis4D]: Det/APm: 0.5861
[08/07 03:11:33 Vis4D]: Det/APl: 0.3133
[08/07 03:11:33 Vis4D]: Showing results for metric: Det
[08/07 03:11:33 Vis4D]:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.319
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.468
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.378
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.224
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.586
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.313
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.162
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.396
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.396
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.315
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.621
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.367
`Trainer.fit` stopped: `max_epochs=1` reached.