Questions on supervised traning part code and performance

[Kin-Zhang]

Thanks for your work and open source,

When I read the code following, I'm wondering why set a assert here for z index must be 0? since Voxelization is 3D, z should not be 0?

zeroflow/models/heads/fast_flow_decoder.py

Lines 15 to 32 in 58a93f7

	def forward_single(self, before_pseudoimage: torch.Tensor,
	after_pseudoimage: torch.Tensor,
	point_offsets: torch.Tensor,
	voxel_coords: torch.Tensor) -> torch.Tensor:
	voxel_coords = voxel_coords.long()
	assert (voxel_coords[:, 0] == 0).all(), "Z index must be 0"
	# Voxel coords are Z, Y, X, and the pseudoimage is Channel, Y, X
	# I have confirmed via visualization that these coordinates are correct.
	after_voxel_vectors = after_pseudoimage[:, voxel_coords[:, 1],
	voxel_coords[:, 2]].T
	before_voxel_vectors = before_pseudoimage[:, voxel_coords[:, 1],
	voxel_coords[:, 2]].T
	concatenated_vectors = torch.cat(
	[before_voxel_vectors, after_voxel_vectors, point_offsets], dim=1)
	flow = self.decoder(concatenated_vectors)
	return flow

[Kin-Zhang]

I see, saw your config to set z voxel size to the range, so for the config, it will be 0 [len=1] but why for that? 2D grid is better?

[kylevedder]

The student is the FastFlow3D model, which uses a PointPillars feature encoder that turns everything into a 2D birds eye view pseudoimage.

The voxelization function that I am using is provided by MMCV, which is more general and can be used for 3D voxelization (e.g. SECOND / VoxelNet). I set the minimum and maximum point height in the config as you referenced, and the point clouds should be chopped accordingly, so everything should be in a single very tall voxel (a point pillar, hence the name PointPillars) to form the pseudoimage. I added the referenced assert to validate this assumption when doing the voxilization for FastFlow3D; if something had a Z index other than zero, it means that the assumption is being violated and that assert should trigger.

[Kin-Zhang]

Thanks for your reply. 🥰

[kylevedder]

Download the pre-trained model weights and run the evaluation on those to ensure that everything else is setup correctly.

If you're able to reproduce those test numbers, then there's something going on with the training run that we can dig into further.

[Kin-Zhang]

Thanks for your help! Appreciate.

By the way, I saw there is new eval on zeroflow in leaderboard

1. what's mean on XL?
2. is it still zeroflow pipeline [but it's kind of weird since it is over the teacher NSFP?] or fastflow3d with gt flow to supervise for the result in the leaderboard?

[kylevedder]

ZeroFlow XL is the ZeroFlow pipeline with two changes:

1. We use twice as much data, pulling unlabeled point clouds from the Argoverse 2 LiDAR dataset (I updated GETTING_STARTED.md with details)
2. We use an enlarged student model; we quadruple the pseudoimage area (512x512 to 1024x1024 by making the pillars twice as small in each dimension), we double the size of the point embedding vector (thereby making each layer of the UNet twice as fat), and we add another layer to the UNet. To contextualize the size of the model change, the normal student model weights are 79MB, the XL model weights are 1.3GB. I've pushed the new UNet backbone to the repo.

To be clear, like with ZeroFlow, ZeroFlow XL is using zero human labels. Our results are simply because we used more unlabeled data and added more parameters! We are able to beat the teacher model's performance and achieve state-of-the-art on the AV2 test set because our model has seen enough diverse data and is expressive enough to learn to distinguish noise from signal in the teacher labels.

On Sunday when we got this result, I tweeted this cool updated graph showing we are doing better than our scaling laws predicted on the normal student model:

To further drive home this point, here are the raw results from our submissions to the AV2 Scene Flow test split:

• Base ZeroFlow results; Threeway EPE of 0.0814
• NSFP results; Threeway EPE of 0.0684
• ZeroFlow XL results; Threeway EPE of 0.0578

If you look at the linked results, our XL model outperforms the teacher across all three categories of the Threeway EPE, but makes particularly large gains in the static foreground category. This means our model has learned to recognize (a lack of motion) better than NSFP is able to represent because it's seen enough data to know that, in expectation, static objects should have zero flow, even if there's a bit of noise in the teacher labels. This is while, in expectation, also extracting what correct movement vectors look like for moving objects.

I also have more good news! I'm an idiot and forgot to run the XL model with our Speed Scaling feature enabled (Equation 5 from the paper), and so I stopped training this model after only 5 epochs (this is akin to seeing ~10 epochs worth of frames). This means that the XL model is undertrained, and it's missing a feature that provides free Foreground Dynamic EPE improvements (which substantially improves Threeway EPE). We are training a new XL model with these features enabled, and for more epochs, so we should hopefully get even better performance from our new student model.

[Kin-Zhang]

thank you so much for sharing these! looking forward to your updates.

[Kin-Zhang]

one more question 😊, XL dataset means also needs zeroflow paper pipeline which is NSFP to produce pseudo label first on the LiDAR dataset? or a new pipeline to do that?

[kylevedder]

That's correct, I used NSFP to pseudolabel the Argoverse 2 LiDAR dataset data subset. We have a large SLURM cluster with a bunch of old 2080tis so the pseudolabeling only took a few days because I could parallelize across them all.

I used the data_prep_scripts/split_nsfp_jobs_sbatch.py to setup and launch these jobs for both the Sensor and LiDAR subset pseudolabeling.

[Kin-Zhang]

@kylevedder sorry to bother you again, but could you specify which three models here to get these three results? or maybe one Base Zeroflow results?

Base ZeroFlow results; Threeway EPE of 0.0814
NSFP results; Threeway EPE of 0.0684
ZeroFlow XL results; Threeway EPE of 0.0578

[kylevedder]

As we discuss in the paper, our reported Threeway EPE for ZeroFlow is an average of three runs.

These weights are the ones highlighted in the weight repo README:

https://github.com/kylevedder/zeroflow_weights/tree/master/argo/nsfp_distilatation_speed_scaled_updated

https://github.com/kylevedder/zeroflow_weights/tree/master/argo/nsfp_distilatation_speed_scaled_updated_run2

https://github.com/kylevedder/zeroflow_weights/tree/master/argo/nsfp_distilatation_speed_scaled_updated_run3

NSFP doesn't have trained weights, it's a test time optimization method.

We have not uploaded the ZeroFlow XL weights, they are too large (1.2GB) and require me to setup git LFS.