indeed it would be great if the authors did so. I personally found some non-official implementations:

• https://github.com/kyegomez/PALI
• https://github.com/ahmdtaha/distributed_sigmoid_loss

• SigLIP
• PaLI
• PaLI-X

IIRC DeTr generate a sequence to predict boxes of objects. I think this paradigm can be applied to such models. “Think before you locate” could be a new path to explore.

https://github.com/FudanDISC/weakly-supervised-mVLP/tree/master

Related links:

• https://github.com/lucidrains/MaMMUT-pytorch
• https://ai.googleblog.com/2023/05/mammut-simple-vision-encoder-text.html

Github link

https://github.com/Jamie-Stirling/RetNet non-official implementation

Relevant links:

• https://github.com/unlearning-challenge/starting-kit
• https://arxiv.org/abs/2209.02299

I know we are moving away from Reddit. However, if I don’t link, I feel like we may miss out good threads on r/machinelearning. Moreover, the authors don’t only post arxiv links, they post other sutff such as Summary, Key points, … (e.g this).

So can I at least put them in the posts instead of posting in a comment?

Reddit thread: https://www.reddit.com/r/MachineLearning/comments/14pq5mq/r_hardwiring_vit_patch_selectivity_into_cnns/

The idea is similar to BLIP-2. Both papers use learnable tokens as queries for a transformer decoder. This decoder query from vision space base on the trainable queries and prompt.

I also want to share some resources.
For Pytorch,

• https://pytorch.org/tutorials/ their basic tutorials are fundamental but some more advanced tutorials might be outdated.
• https://www.learnpytorch.io/ the author guides mostly in computer vision but he gives the overview from research to production.

For TPU,

• https://github.com/ayaka14732/tpu-starter full guideline using TPUs with Jax