4DGS
A note for Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting.
Preliminary
3DGS
We all know 3DGS uses a series of Gaussian Points to simulation the scene. The scene rendering is a differentiable rasterization process, aiming at getting the color of a certain pixel \((u,v)\) in the direction of observation \(d\). This can be formulated as,
\[I(u,v)=\sum_{i=1}^Np_i(u,v;\mu_i^{2d},\Sigma_i^{2d})\alpha_ic_i(d_i)\prod_{j=1}^{i-1}(1-p_j(u,v;\mu_i^{2d},\Sigma_i^{2d})\alpha_j)\]4DGS
Problem formulation
Back to 4DGS, the naive idea is to add the time \(t\) directly into the above 3DGS rendering formula. Then, we can obtain,
\[I(u,v,t)=\sum_{i=1}^Np_i(u,v,t)\alpha_ic_i(d_i)\prod_{j=1}^{i-1}(1-p_j(u,v,t)\alpha_j)\]if we assume time and space are independent, we can further use joint probability to decompose the probability density. And we can transfer the naive 4DGS formula into,
\[I(u,v,t)=\sum_{i=1}^Np_i(t)p_i(u,v|t)\alpha_ic_i(d_i)\prod_{j=1}^{i-1}(1-p_j(t)p_j(u,v|t)\alpha_j)\]Now, we decompose the color contribution \(p_i(u,v,t)\) of each Gaussian into the margin probability \(p_i(t)\) at time t, and the conditional space probability \(p_i(u,v\|t)\) at time \(t\). However, this assumption view time and space as independent, which is bound to fail to capture the dynamic nature of the scene.
4DGS representation
The following is the representation of 4DGS proposed by this article. We need to treat time and space dimensions equally as a integrated 4D Gaussian model. From 3DGS, the covariance matrix can use eigenvalue decomposition to obtain scale and rotation matrix, which is matter for 3DGS transformation. Then, in 4DGS, the 4D covariance matrix is first need to be created.
We first add time dimention into scale matrix. Due to scale matrix is diagonal matrix, we can add the time into the last position \(S={\rm diag}(s_x,s_y,s_z,s_t)\). As for 4D rotation matrix, we can represent by two isotropic rotations \(R=L(q_l)R(q_r)\). The first 3 items of the mean in 4D Gaussian distribution correspond to the position in 3D space, and the last one corresponds to the time \(\mu = (\mu_x,\mu_y,\mu_z,\mu_t)\).
Finally, according to the properties of multivariate Gaussian distribution, 3D Gaussians can be derived from 4D Gaussians for rasterized rendering by marginalizing the temporal dimension. Compute a 3D representation of the scene at any given time t can be formualted as,
\[\mu_{xyz|t}=\mu_{1:3}+\Sigma_{1:3,4}\Sigma_{4,4}^{-1}(t-\mu_t)\] \[\Sigma_{xyz|t}=\Sigma_{1:3,1:3}+\Sigma_{1:3,4}\Sigma_{4,4}^{-1}\Sigma_{4,1:3}\] \[p(t)=N(t;\mu_4,\Sigma_{4,4})\]Establishing 4DGS, we can simulate dynamics in terms of spatiotemporal consistency. And this is the key that the finally fps can up to 140 compared to papers utilize transformation field to simulation dynamic.
4DSH
The next part is about 4D spherindrical harmonics. Right, this noun is both familiar and unfamiliar. You must hear spherical harmonics is 3DGS. In the end, spherical harmonics is just a set of basis functions. With higher rank, the better for representation. Then, for adding time dimension, we need to use 4D spherindrical harmonics, just merging one dimension basis function into the 3DSH. As for 1D time SH, this paper use the Fourier series to represent. Thus, we can formualte 4DSH into,
\[Z_{nl}^m(t,\theta,\phi)={\rm cos}(\frac {2\pi n} T t)Y_l^m(\theta,\phi)\]Among, \(Y_l^m(\theta,\phi)\) is known to us as 3DSH. And \(l\) denotes the order of the spherical harmonic function, \(m\) is the corresponding suborder, and \(\theta\) and \(\phi\) are the spherical coordinates of the view Angle. \(n\) denotes the Fourier series. In the training, we enhance the spatio-temporal expressivity by gradually increasing \(l\) and \(n\).
Trick in training
Yep! When I reproduce this paper, the finally psnr can not reach the reported one. I tried two instance on Plenoptic Video dataset, coffee_martini and cook_spinach. The psnr results are 2 point lower than the author. The most reason is densitification iteration steps.
Densification
Densification in 3DGS has two main method, split and clone. Using this, the representation of scene during training can be better, due to splitting big Gaussians is important to allow good reconstruction of the background, while cloning the small Gaussians instead of splitting them allows for a better and faster convergence especially on thin structures.
In spatio-temporal densification, the main purpose is to dynamically add Gaussian points during the training process to better represent the scene details. The practice is to evaluate the Gaussian density in the scene considering both the average gradient in the temporal dimension and the gradient in the average pilot spatial location. From the ablation result in this paper, densification is matter for the last scene rendering.
Reference and useful links
Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting, code
3D Gaussian Splatting for Real-Time Radiance Field Rendering
Spherical Harmonics Introduction