Skip to content

[MM] speed up OPs using hf models (clip, ...)#199

Description

@drcege

Currently, when set np=28, clip of vit-base-p32 takes over 1h to compute similarities for 558k dataset, and tens of hours for vit-large-p14-336.

Image

Image

Image

Perhaps the following can help:

  1. loading on GPU (implemented)
  2. using batched computing (not easy to implement, as batching is closely related to the internal logic of operators)

Metadata

Metadata

Assignees

Labels

dj:multimodalissues/PRs about multimodal data processingenhancementNew feature or request

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions