Almost six months ago, my advisors suggested that we write a survey in this field. Here, this note comes.
At first, I conducted a literature review and collected papers in this field to understand the volume of existing work. Following my advisor’s suggestion, I created a table to record the papers. Don’t worry about just few papers—over time, they will accumulate linearly.
I tried to classify the papers, but it was difficult to establish a clear classification system. Then, I quited, and drafted an outline for the survey architecture. The most efficient way to draft an outline is to find a reference.
In the beginning, it was very hard for me to write anything meaningful. I have no idea where to start. Then, my advisor suggested, “Maybe you can start by adding one paper you’ve read.” Don’t worry about the connections, just add it.
So, we set a goal that every week, read, summarize, and add around three papers in this area. This is a long process, please hang in there, and be regular. You will have fun.
At this stage, the survey still couldn’t be considered a proper article because there was no connection between different parts, especially between papers.
Luckily, you will have developed some ideas on how to make a classification system, and how to organize each parts. It’s amazing, but it’s the truth.
I first created a relatively mature classification system. Actually, I have modified several version before settling on one.
First of all, the biggest problem lies in organizing different sections. If you jump straight into content without transition sentences, the flow will feel unnatural. Take some time in organization the transition sentence.
Second, after proposing a problem, always provide corresponding solutions or discuss existing progress.
Finally, a good visualization or table can leave a strong impression.
Grounding Creativity in Physics: A Brief Survey of Physical Priors in AIGC
See you soon👋!
]]>There is a form to record the maximum supported GCC version for your CUDA version: | CUDA | max supported GCC version | | :—-: | :—-: | | 12.4, 12.5 | 13.2| | 12.1, 12.2, 12.3 | 12.2| | 12 | 12.1| | 11.4.1+, 11.5, 11.6, 11.7, 11.8 | 11| | 11.1, 11.2, 11.3, 11.4.0 | 10| | 11 |9| | 10.1, 10.2 |8| | 9.2, 10.0 |7| | 9.0, 9.1 |6| | 8 |5.3| | 7 |4.9| | 5.5, 6 |4.8| | 4.2, 5 |4.6| | 4.1 |4.5| | 4.0 |4.4|
Fast GCC version alternate:
sudo update-alternatives --config gcc
We all know 3DGS uses a series of Gaussian Points to simulation the scene. The scene rendering is a differentiable rasterization process, aiming at getting the color of a certain pixel \((u,v)\) in the direction of observation \(d\). This can be formulated as,
\[I(u,v)=\sum_{i=1}^Np_i(u,v;\mu_i^{2d},\Sigma_i^{2d})\alpha_ic_i(d_i)\prod_{j=1}^{i-1}(1-p_j(u,v;\mu_i^{2d},\Sigma_i^{2d})\alpha_j)\]Back to 4DGS, the naive idea is to add the time \(t\) directly into the above 3DGS rendering formula. Then, we can obtain,
\[I(u,v,t)=\sum_{i=1}^Np_i(u,v,t)\alpha_ic_i(d_i)\prod_{j=1}^{i-1}(1-p_j(u,v,t)\alpha_j)\]if we assume time and space are independent, we can further use joint probability to decompose the probability density. And we can transfer the naive 4DGS formula into,
\[I(u,v,t)=\sum_{i=1}^Np_i(t)p_i(u,v|t)\alpha_ic_i(d_i)\prod_{j=1}^{i-1}(1-p_j(t)p_j(u,v|t)\alpha_j)\]Now, we decompose the color contribution \(p_i(u,v,t)\) of each Gaussian into the margin probability \(p_i(t)\) at time t, and the conditional space probability \(p_i(u,v\|t)\) at time \(t\). However, this assumption view time and space as independent, which is bound to fail to capture the dynamic nature of the scene.
The following is the representation of 4DGS proposed by this article. We need to treat time and space dimensions equally as a integrated 4D Gaussian model. From 3DGS, the covariance matrix can use eigenvalue decomposition to obtain scale and rotation matrix, which is matter for 3DGS transformation. Then, in 4DGS, the 4D covariance matrix is first need to be created.
We first add time dimention into scale matrix. Due to scale matrix is diagonal matrix, we can add the time into the last position \(S={\rm diag}(s_x,s_y,s_z,s_t)\). As for 4D rotation matrix, we can represent by two isotropic rotations \(R=L(q_l)R(q_r)\). The first 3 items of the mean in 4D Gaussian distribution correspond to the position in 3D space, and the last one corresponds to the time \(\mu = (\mu_x,\mu_y,\mu_z,\mu_t)\).
Finally, according to the properties of multivariate Gaussian distribution, 3D Gaussians can be derived from 4D Gaussians for rasterized rendering by marginalizing the temporal dimension. Compute a 3D representation of the scene at any given time t can be formualted as,
\[\mu_{xyz|t}=\mu_{1:3}+\Sigma_{1:3,4}\Sigma_{4,4}^{-1}(t-\mu_t)\] \[\Sigma_{xyz|t}=\Sigma_{1:3,1:3}+\Sigma_{1:3,4}\Sigma_{4,4}^{-1}\Sigma_{4,1:3}\] \[p(t)=N(t;\mu_4,\Sigma_{4,4})\]Establishing 4DGS, we can simulate dynamics in terms of spatiotemporal consistency. And this is the key that the finally fps can up to 140 compared to papers utilize transformation field to simulation dynamic.
The next part is about 4D spherindrical harmonics. Right, this noun is both familiar and unfamiliar. You must hear spherical harmonics is 3DGS. In the end, spherical harmonics is just a set of basis functions. With higher rank, the better for representation. Then, for adding time dimension, we need to use 4D spherindrical harmonics, just merging one dimension basis function into the 3DSH. As for 1D time SH, this paper use the Fourier series to represent. Thus, we can formualte 4DSH into,
\[Z_{nl}^m(t,\theta,\phi)={\rm cos}(\frac {2\pi n} T t)Y_l^m(\theta,\phi)\]Among, \(Y_l^m(\theta,\phi)\) is known to us as 3DSH. And \(l\) denotes the order of the spherical harmonic function, \(m\) is the corresponding suborder, and \(\theta\) and \(\phi\) are the spherical coordinates of the view Angle. \(n\) denotes the Fourier series. In the training, we enhance the spatio-temporal expressivity by gradually increasing \(l\) and \(n\).
Yep! When I reproduce this paper, the finally psnr can not reach the reported one. I tried two instance on Plenoptic Video dataset, coffee_martini and cook_spinach. The psnr results are 2 point lower than the author. The most reason is densitification iteration steps.
Densification in 3DGS has two main method, split and clone. Using this, the representation of scene during training can be better, due to splitting big Gaussians is important to allow good reconstruction of the background, while cloning the small Gaussians instead of splitting them allows for a better and faster convergence especially on thin structures.
In spatio-temporal densification, the main purpose is to dynamically add Gaussian points during the training process to better represent the scene details. The practice is to evaluate the Gaussian density in the scene considering both the average gradient in the temporal dimension and the gradient in the average pilot spatial location. From the ablation result in this paper, densification is matter for the last scene rendering.
Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting, code
3D Gaussian Splatting for Real-Time Radiance Field Rendering
]]>