Skip to content

Commit 813e941

Browse files
EmGarrfacebook-github-bot
EmGarr
authored and
facebook-github-bot
committed
Add the OverfitModel
Summary: Introduces the OverfitModel for NeRF-style training with overfitting to one scene. It is a specific case of GenericModel. It has been disentangle to ease usage. ## General modification 1. Modularize a minimum GenericModel to introduce OverfitModel 2. Introduce OverfitModel and ensure through unit testing that it behaves like GenericModel. ## Modularization The following methods have been extracted from GenericModel to allow modularity with ManyViewModel: - get_objective is now a call to weighted_sum_losses - log_loss_weights - prepare_inputs The generic methods have been moved to an utils.py file. Simplify the code to introduce OverfitModel. Private methods like chunk_generator are now public and can now be used by ManyViewModel. Reviewed By: shapovalov Differential Revision: D43771992 fbshipit-source-id: 6102aeb21c7fdd56aa2ff9cd1dd23fd9fbf26315
1 parent 7d8b029 commit 813e941

File tree

16 files changed

+2012
-201
lines changed

16 files changed

+2012
-201
lines changed

‎projects/implicitron_trainer/README.md‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -248,7 +248,7 @@ The main object for this trainer loop is `Experiment`. It has four top-level rep
248248
* `data_source`: This is a `DataSourceBase` which defaults to `ImplicitronDataSource`.
249249
It constructs the data sets and dataloaders.
250250
* `model_factory`: This is a `ModelFactoryBase` which defaults to `ImplicitronModelFactory`.
251-
It constructs the model, which is usually an instance of implicitron's main `GenericModel` class, and can load its weights from a checkpoint.
251+
It constructs the model, which is usually an instance of `OverfitModel` (for NeRF-style training with overfitting to one scene) or `GenericModel` (that is able to generalize to multiple scenes by NeRFormer-style conditioning on other scene views), and can load its weights from a checkpoint.
252252
* `optimizer_factory`: This is an `OptimizerFactoryBase` which defaults to `ImplicitronOptimizerFactory`.
253253
It constructs the optimizer and can load its weights from a checkpoint.
254254
* `training_loop`: This is a `TrainingLoopBase` which defaults to `ImplicitronTrainingLoop` and defines the main training loop.
@@ -292,6 +292,43 @@ model_GenericModel_args: GenericModel
292292
╘== ReductionFeatureAggregator
293293
```
294294
295+
Here is the class structure of OverfitModel:
296+
297+
```
298+
model_OverfitModel_args: OverfitModel
299+
└-- raysampler_*_args: RaySampler
300+
╘== AdaptiveRaysampler
301+
╘== NearFarRaysampler
302+
└-- renderer_*_args: BaseRenderer
303+
╘== MultiPassEmissionAbsorptionRenderer
304+
╘== LSTMRenderer
305+
╘== SignedDistanceFunctionRenderer
306+
└-- ray_tracer_args: RayTracing
307+
└-- ray_normal_coloring_network_args: RayNormalColoringNetwork
308+
└-- implicit_function_*_args: ImplicitFunctionBase
309+
╘== NeuralRadianceFieldImplicitFunction
310+
╘== SRNImplicitFunction
311+
└-- raymarch_function_args: SRNRaymarchFunction
312+
└-- pixel_generator_args: SRNPixelGenerator
313+
╘== SRNHyperNetImplicitFunction
314+
└-- hypernet_args: SRNRaymarchHyperNet
315+
└-- pixel_generator_args: SRNPixelGenerator
316+
╘== IdrFeatureField
317+
└-- coarse_implicit_function_*_args: ImplicitFunctionBase
318+
╘== NeuralRadianceFieldImplicitFunction
319+
╘== SRNImplicitFunction
320+
└-- raymarch_function_args: SRNRaymarchFunction
321+
└-- pixel_generator_args: SRNPixelGenerator
322+
╘== SRNHyperNetImplicitFunction
323+
└-- hypernet_args: SRNRaymarchHyperNet
324+
└-- pixel_generator_args: SRNPixelGenerator
325+
╘== IdrFeatureField
326+
```
327+
328+
OverfitModel has been introduced to create a simple class to disantagle Nerfs which the overfit pattern
329+
from the GenericModel.
330+
331+
295332
Please look at the annotations of the respective classes or functions for the lists of hyperparameters.
296333
`tests/experiment.yaml` shows every possible option if you have no user-defined classes.
297334

‎projects/implicitron_trainer/configs/overfit_base.yaml‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
defaults:
2+
- default_config
3+
- _self_
4+
exp_dir: ./data/exps/overfit_base/
5+
training_loop_ImplicitronTrainingLoop_args:
6+
visdom_port: 8097
7+
visualize_interval: 0
8+
max_epochs: 1000
9+
data_source_ImplicitronDataSource_args:
10+
data_loader_map_provider_class_type: SequenceDataLoaderMapProvider
11+
dataset_map_provider_class_type: JsonIndexDatasetMapProvider
12+
data_loader_map_provider_SequenceDataLoaderMapProvider_args:
13+
dataset_length_train: 1000
14+
dataset_length_val: 1
15+
num_workers: 8
16+
dataset_map_provider_JsonIndexDatasetMapProvider_args:
17+
dataset_root: ${oc.env:CO3D_DATASET_ROOT}
18+
n_frames_per_sequence: -1
19+
test_on_train: true
20+
test_restrict_sequence_id: 0
21+
dataset_JsonIndexDataset_args:
22+
load_point_clouds: false
23+
mask_depths: false
24+
mask_images: false
25+
model_factory_ImplicitronModelFactory_args:
26+
model_class_type: "OverfitModel"
27+
model_OverfitModel_args:
28+
loss_weights:
29+
loss_mask_bce: 1.0
30+
loss_prev_stage_mask_bce: 1.0
31+
loss_autodecoder_norm: 0.01
32+
loss_rgb_mse: 1.0
33+
loss_prev_stage_rgb_mse: 1.0
34+
output_rasterized_mc: false
35+
chunk_size_grid: 102400
36+
render_image_height: 400
37+
render_image_width: 400
38+
share_implicit_function_across_passes: false
39+
implicit_function_class_type: "NeuralRadianceFieldImplicitFunction"
40+
implicit_function_NeuralRadianceFieldImplicitFunction_args:
41+
n_harmonic_functions_xyz: 10
42+
n_harmonic_functions_dir: 4
43+
n_hidden_neurons_xyz: 256
44+
n_hidden_neurons_dir: 128
45+
n_layers_xyz: 8
46+
append_xyz:
47+
- 5
48+
coarse_implicit_function_class_type: "NeuralRadianceFieldImplicitFunction"
49+
coarse_implicit_function_NeuralRadianceFieldImplicitFunction_args:
50+
n_harmonic_functions_xyz: 10
51+
n_harmonic_functions_dir: 4
52+
n_hidden_neurons_xyz: 256
53+
n_hidden_neurons_dir: 128
54+
n_layers_xyz: 8
55+
append_xyz:
56+
- 5
57+
raysampler_AdaptiveRaySampler_args:
58+
n_rays_per_image_sampled_from_mask: 1024
59+
scene_extent: 8.0
60+
n_pts_per_ray_training: 64
61+
n_pts_per_ray_evaluation: 64
62+
stratified_point_sampling_training: true
63+
stratified_point_sampling_evaluation: false
64+
renderer_MultiPassEmissionAbsorptionRenderer_args:
65+
n_pts_per_ray_fine_training: 64
66+
n_pts_per_ray_fine_evaluation: 64
67+
append_coarse_samples_to_fine: true
68+
density_noise_std_train: 1.0
69+
optimizer_factory_ImplicitronOptimizerFactory_args:
70+
breed: Adam
71+
weight_decay: 0.0
72+
lr_policy: MultiStepLR
73+
multistep_lr_milestones: []
74+
lr: 0.0005
75+
gamma: 0.1
76+
momentum: 0.9
77+
betas:
78+
- 0.9
79+
- 0.999

‎projects/implicitron_trainer/configs/overfit_singleseq_base.yaml‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
defaults:
2+
- overfit_base
3+
- _self_
4+
data_source_ImplicitronDataSource_args:
5+
data_loader_map_provider_SequenceDataLoaderMapProvider_args:
6+
batch_size: 1
7+
dataset_length_train: 1000
8+
dataset_length_val: 1
9+
num_workers: 8
10+
dataset_map_provider_JsonIndexDatasetMapProvider_args:
11+
assert_single_seq: true
12+
n_frames_per_sequence: -1
13+
test_restrict_sequence_id: 0
14+
test_on_train: false
15+
model_factory_ImplicitronModelFactory_args:
16+
model_class_type: "OverfitModel"
17+
model_OverfitModel_args:
18+
render_image_height: 800
19+
render_image_width: 800
20+
log_vars:
21+
- loss_rgb_psnr_fg
22+
- loss_rgb_psnr
23+
- loss_eikonal
24+
- loss_prev_stage_rgb_psnr
25+
- loss_mask_bce
26+
- loss_prev_stage_mask_bce
27+
- loss_rgb_mse
28+
- loss_prev_stage_rgb_mse
29+
- loss_depth_abs
30+
- loss_depth_abs_fg
31+
- loss_kl
32+
- loss_mask_neg_iou
33+
- objective
34+
- epoch
35+
- sec/it
36+
optimizer_factory_ImplicitronOptimizerFactory_args:
37+
lr: 0.0005
38+
multistep_lr_milestones:
39+
- 200
40+
- 300
41+
training_loop_ImplicitronTrainingLoop_args:
42+
max_epochs: 400

‎projects/implicitron_trainer/configs/overfit_singleseq_nerf_blender.yaml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
defaults:
2+
- overfit_singleseq_base
3+
- _self_
4+
exp_dir: "./data/overfit_nerf_blender_repro/${oc.env:BLENDER_SINGLESEQ_CLASS}"
5+
data_source_ImplicitronDataSource_args:
6+
data_loader_map_provider_SequenceDataLoaderMapProvider_args:
7+
dataset_length_train: 100
8+
dataset_map_provider_class_type: BlenderDatasetMapProvider
9+
dataset_map_provider_BlenderDatasetMapProvider_args:
10+
base_dir: ${oc.env:BLENDER_DATASET_ROOT}/${oc.env:BLENDER_SINGLESEQ_CLASS}
11+
n_known_frames_for_test: null
12+
object_name: ${oc.env:BLENDER_SINGLESEQ_CLASS}
13+
path_manager_factory_class_type: PathManagerFactory
14+
path_manager_factory_PathManagerFactory_args:
15+
silence_logs: true
16+
17+
model_factory_ImplicitronModelFactory_args:
18+
model_class_type: "OverfitModel"
19+
model_OverfitModel_args:
20+
mask_images: false
21+
raysampler_class_type: AdaptiveRaySampler
22+
raysampler_AdaptiveRaySampler_args:
23+
n_pts_per_ray_training: 64
24+
n_pts_per_ray_evaluation: 64
25+
n_rays_per_image_sampled_from_mask: 4096
26+
stratified_point_sampling_training: true
27+
stratified_point_sampling_evaluation: false
28+
scene_extent: 2.0
29+
scene_center:
30+
- 0.0
31+
- 0.0
32+
- 0.0
33+
renderer_MultiPassEmissionAbsorptionRenderer_args:
34+
density_noise_std_train: 0.0
35+
n_pts_per_ray_fine_training: 128
36+
n_pts_per_ray_fine_evaluation: 128
37+
raymarcher_EmissionAbsorptionRaymarcher_args:
38+
blend_output: false
39+
loss_weights:
40+
loss_rgb_mse: 1.0
41+
loss_prev_stage_rgb_mse: 1.0
42+
loss_mask_bce: 0.0
43+
loss_prev_stage_mask_bce: 0.0
44+
loss_autodecoder_norm: 0.00
45+
46+
optimizer_factory_ImplicitronOptimizerFactory_args:
47+
exponential_lr_step_size: 3001
48+
lr_policy: LinearExponential
49+
linear_exponential_lr_milestone: 200
50+
51+
training_loop_ImplicitronTrainingLoop_args:
52+
max_epochs: 6000
53+
metric_print_interval: 10
54+
store_checkpoints_purge: 3
55+
test_when_finished: true
56+
validation_interval: 100

‎projects/implicitron_trainer/experiment.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@
5959
DataSourceBase,
6060
ImplicitronDataSource,
6161
)
62-
from pytorch3d.implicitron.models.generic_model import ImplicitronModelBase
62+
from pytorch3d.implicitron.models.base_model import ImplicitronModelBase
6363

6464
from pytorch3d.implicitron.models.renderer.multipass_ea import (
6565
MultiPassEmissionAbsorptionRenderer,

0 commit comments

Comments
 (0)