```
import matplotlib.pyplot as plt
import torch
from diffdrr.drr import DRR
from diffdrr.data import load_example_ct
from diffdrr.visualization import plot_drr
# Read in the volume and get its origin and spacing in world coordinates
= load_example_ct()
subject
# Initialize the DRR module for generating synthetic X-rays
= torch.device("cuda" if torch.cuda.is_available() else "cpu")
device = DRR(
drr # An object storing the CT volume, origin, and voxel spacing
subject, =1020.0, # Source-to-detector distance (i.e., focal length)
sdd=200, # Image height (if width is not provided, the generated DRR is square)
height=2.0, # Pixel spacing (in mm).
delx
).to(device)
# Set the camera pose with rotations (yaw, pitch, roll) and translations (x, y, z)
= torch.tensor([[0.0, 0.0, 0.0]], device=device)
rotations = torch.tensor([[0.0, 850.0, 0.0]], device=device)
translations
# 📸 Also note that DiffDRR can take many representations of SO(3) 📸
# For example, quaternions, rotation matrix, axis-angle, etc...
= drr(rotations, translations, parameterization="euler_angles", convention="ZXY")
img =False)
plot_drr(img, ticks plt.show()
```

# DiffDRR

Auto-differentiable DRR rendering and optimization in PyTorch

`DiffDRR`

is a PyTorch-based digitally reconstructed radiograph (DRR) generator that provides

- Differentiable X-ray rendering
- GPU-accelerated synthesis and optimization
- A pure Python implementation

Most importantly, `DiffDRR`

implements DRR rendering as a PyTorch module, making it interoperable in deep learning pipelines.

## Install

`pip install diffdrr`

## Hello, World!

The following minimal example specifies the geometry of the projectional radiograph imaging system and traces rays through a CT volume:

On a single NVIDIA RTX 2080 Ti GPU, producing such an image takes

`25.2 ms ± 10.5 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)`

The full example is available at `introduction.ipynb`

.

## Usage

### Rendering

The physics-based pipeline in `DiffDRR`

renders photorealistic X-rays. For example, compare a real X-ray to a synthetic X-ray rendered from a CT of the same patient using `DiffDRR`

(X-rays and CTs from the DeepFluoro dataset):

### 2D/3D Registration

The impotus for developing `DiffDRR`

was to solve 2D/3D registration problems with gradient-based optimization. Here, we demonstrate `DiffDRR`

’s capabilities by generating two DRRs:

- A fixed DRR from a set of ground truth parameters
- A moving DRR from randomly initialized parameters

To align the two images, we use gradient descent to maximize an image similarity metric between the two DRRs. This produces optimization runs like this:

The full example is available at `optimizers.ipynb`

*🆕 Examples on Real-World Data 🆕*

For examples running `DiffDRR`

on real surgical datasets, check out our latest work, `DiffPose`

:

This work includes a lot of real-world usecases of `DiffDRR`

including

- Using
`DiffDRR`

as a layer in a deep learning architecture - Alignment of real X-rays and rendered DRRs
- Achieving sub-millimeter registration accuracy very quickly

### Segmentation

`DiffDRR`

can project 3D labelmaps into 2D simply using perspective geometry, helping identify particular structures in simulated X-rays (these labels come from the TotalSegmentator v2 dataset):

### Volume Reconstruction

`DiffDRR`

is differentiable with respect to the 3D volume as well as camera poses. Therefore, it could (in theory) be used for volume reconstruction via differentiable rendering. However, this feature has not been robustly tested and is currently under active development (see `reconstruction.ipynb`

)!

## Development

`DiffDRR`

source code, docs, and CI are all built using `nbdev`

. To get set up with `nbdev`

, install the following

```
mamba install jupyterlab nbdev -c fastai -c conda-forge
nbdev_install_quarto # To build docs
nbdev_install_hooks # Make notebooks git-friendly
```

Running `nbdev_help`

will give you the full list of options. The most important ones are

```
nbdev_preview # Render docs locally and inspect in browser
nbdev_clean # NECESSARY BEFORE PUSHING
nbdev_test # tests notebooks
nbdev_export # builds package and builds docs
```

For more details, follow this in-depth tutorial.

## How does `DiffDRR`

work?

`DiffDRR`

reformulates Siddon’s method,^{1} the canonical algorithm for calculating the radiologic path of an X-ray through a volume, as a series of vectorized tensor operations. This version of the algorithm is easily implemented in tensor algebra libraries like PyTorch to achieve a fast auto-differentiable DRR generator.

## Citing `DiffDRR`

If you find `DiffDRR`

useful in your work, please cite our paper (or the freely accessible arXiv version):

```
@inproceedings{gopalakrishnanDiffDRR2022,
author = {Gopalakrishnan, Vivek and Golland, Polina},
title = {Fast Auto-Differentiable Digitally Reconstructed Radiographs for Solving Inverse Problems in Intraoperative Imaging},
year = {2022},
booktitle = {Clinical Image-based Procedures: 11th International Workshop, CLIP 2022, Held in Conjunction with MICCAI 2022, Singapore, Proceedings},
series = {Lecture Notes in Computer Science},
publisher = {Springer},
doi = {https://doi.org/10.1007/978-3-031-23179-7_1},
}
```