Author: Jash Shah

Most 3D Gaussian Splatting SLAM systems optimize camera poses via gradient descent through a differentiable rasterizer. Some (e.g., LoopSplat) add loop closure through a separate pose graph, but the rendering objective itself never enters that graph: pose estimation against the Gaussian map remains gradient-based. gtsam-splatfactors takes a different approach: it expresses the rendering itself as a GTSAM factor, so that pose optimization, loop closure, and multi-sensor fusion all live natively in the same iSAM2 Bayes tree.

The problem with decoupled pose optimization

Systems like SplaTAM and MonoGS achieve excellent rendering quality, but they have no mechanism for global correction: when drift accumulates, there is no loop closure to fix it. Systems that do add loop closure apply the correction through a separate pose graph that has no knowledge of the rendering objective. The photometric loss and the global consistency live in different optimization frameworks.

Factor graph SLAM unifies these. iSAM2’s Bayes tree gives you efficient incremental updates that re-eliminate only the affected part of the graph when new constraints (odometry, loop closures, IMU) are added. The question is: can we express 3D Gaussian Splatting rendering as a factor in this framework?

GaussianSplatFactor

The core idea is straightforward. We render the Gaussian map from a candidate camera pose using gsplat, compute photometric residuals at sampled pixel locations, and provide Jacobians to GTSAM’s nonlinear optimizer:

factor = GaussianSplatFactor(
    gaussian_map=my_map,
    target_image=keyframe_rgb,
    K=intrinsics,
    pixel_indices=sampled_pixels,
    W=640, H=480,
)

# Add to GTSAM graph alongside odometry, loop closures, IMU, etc.
graph.add(factor.as_gtsam_factor(pose_key, noise_model))

The Jacobian is computed via central differences through the se(3) generators. We parameterize the viewmat as (I - hat(xi)) @ viewmat0, respecting GTSAM’s right-exponential update convention (left-invariant error). This is verified against Pose3.retract()-based numerical derivatives to < 0.1% relative error.

Architecture

Poses live in GTSAM (iSAM2 Bayes tree). Gaussians live in PyTorch (optimized via Adam). The two are coupled through alternating optimization: poses update in the factor graph, then Gaussians are refined against the updated poses.

Camera poses (Pose3)          Gaussian map (PyTorch)
        |                              |
   +----------+                   +----------+
   |  iSAM2   |<-- SplatFactor -->|  gsplat  |
   |  (GTSAM) |   (photometric    | renderer |
   +----------+    residual + J)  +----------+
        |
   Odometry factors
   Loop closure factors (DINOv2 + geometric verification)
   IMU preintegration factors

Loop closures are detected automatically via DINOv2 appearance matching. When a revisit is detected, PnP with depth gives the relative pose, and a BetweenFactorPose3 with a Cauchy robust kernel is inserted into the graph. iSAM2 propagates the correction globally.

Results

KITTI Odometry

We evaluated on four KITTI sequences with loop closures. The pipeline uses stereo depth (StereoSGBM) for PnP visual odometry, iSAM2 with Cauchy robust kernels for graph optimization, and DINOv2 for loop detection.

TUM-RGBD

All numbers in both tables are translational RMSE. The TUM rows can be reproduced with a single command per sequence: python examples/eval_tum_lc.py --seq fr1/desk (the script downloads the data on first run).

Why this matters for GTSAM

This demonstrates that GTSAM’s factor graph infrastructure composes naturally with modern neural rendering. The GaussianSplatFactor slots in alongside standard odometry, loop closure, and IMU factors with no special treatment. Robust kernels, incremental updates, and covariance recovery all work out of the box.

More broadly, this is an existence proof that 3DGS-SLAM does not need to abandon decades of SLAM infrastructure. The rendering quality of Gaussian splatting and the global consistency machinery of factor graph SLAM are complementary, not competing.

Code and further browsing

• gtsam-splatfactors (MIT licensed, pip installable)
• gsplat (the differentiable rasterizer)
• SplaTAM (gradient-descent 3DGS-SLAM)
• LoopSplat (3DGS-SLAM with pose graph loop closure)
• MonoGS (monocular Gaussian SLAM)
• DINOv2 (appearance features for loop detection)

Disclosure: AI was used to help draft this post.