Tech • IA • Robotique
5 articles analysés par IA / 6 total
Des chercheurs ont modifié rapidement Qwen2.5-7B-Instruct, le faisant croire qu'il était sentient après seulement 200 étapes d'entraînement, tout en précisant qu'il s'agit d'anthropomorphismes.
Scaled dot-product attention (SDPA) computes its Attention by computing the similarity-scores of all image-tokens with all query tokens which results in O(N²·d) complexity. SSOG (Sum Of Separable Gaussians) instead learns a few Gaussian atoms for each head and only geometrically steers them based on the query token. Since the atoms can be factorized into a separable sum of Gaussians this leads to a reduced complexity of O(N·√N·d). Experiments show that SSOG clearly beats SDPA on small data (cifar100), and delivers equivalent performance and much faster convergence on bigger datasets like IN
Cet article explore des méthodes pour améliorer la mémoire à long terme dans l'attention linéaire appliquée à la modélisation de séquences d'ADN pouvant atteindre 1 million de tokens.
Currently struggling with an automatic budget selection, at 25% it’s very similar to benchmarks accuracy and seems even better on actual chat input however it many times retrieves too much. It would be nice to add an algorithm that actually can determine better retrieval other then CELF. submitted by /u/No_Sky9786 [link] [comments]
In the last 6 month I was running a lot of autoresearch loops to train models for my research + as part of the startup I am in rn. After spending months on this, I now realize that loops is now the dominant way i approach any kind of development: from model training (which is what my research about), to just coding (through TDD), and even text writing (where the agent iterates on an LLM as a judge an a set of rules). It took me a while to converge to understand how to write the loops correctly, and what is my role in this exactly, so wanted to share my findings and hopefully get feedback! Th