Transformator (model mašinskog učenja)

Transformator je arhitektura dubokog učenja koju je razvio Gugl i zasnovana na algoritmu mehanizma višestruke pažnje, predloženom u radu objavljenom 2017. godine, pod naslovom „Pažnja je sve što vam je potrebno“.^[1] On nema rekurentne jedinice i stoga zahteva manje vremena za obuku od prethodnih rekurentnih neuronskih arhitektura, kao što je dugotrajna kratkoročna memorija (LSTM),^[2] i njegova kasnija varijacija je pretežno usvojena za obuku velikih jezičkih modela (LLM) na velikim (jezičkim) skupovima podataka, kao što su korpus Vikipedije i Komon Krol.^[3] Tekst se konvertuje u numeričke reprezentacije koje se nazivaju tokeni, a svaki token se konvertuje u vektor traženjem iz tabele ugrađinih reči.^[1] Na svakom sloju, svaki token se zatim kontekstualizuje unutar okvira kontekstnog prozora sa drugim (nemaskiranim) tokenima preko paralelnog mehanizma pažnje sa više glava koji omogućava da se pojača signal za ključne tokene i umanji značaj manje važnih tokena. Publikacija o transformatoru, objavljena 2017. godine, zasnovana je na softmaks mehanizmu pažnje koji su predložili Bahdanau et. al. 2014. za mašinsko prevođenje,^[4]^[5] i brzom kontroleru težine, sličnom transformatoru, predloženom 1992. godine.^[6]^[7]^[8]

Ova arhitektura se sada koristi ne samo u obradi prirodnog jezika i kompjuterskoj viziji,^[9] već i u audio^[10] i multimodalnoj obradi. To je takođe dovelo do razvoja unapred obučenih sistema, kao što su generativni prethodno obučeni transformatori (GPT)^[11] i BERT^[12] (bidirekciona inkoderska reprezentacija za transformatore).

Reference

^ ^а ^б Vaswani, Ashish; Shazeer, Noam; Parmar, Niki; Uszkoreit, Jakob; Jones, Llion; Gomez, Aidan N; Kaiser, Łukasz; Polosukhin, Illia (2017). „Attention is All you Need” (PDF). Advances in Neural Information Processing Systems. Curran Associates, Inc. 30.
^ Hochreiter, Sepp; Schmidhuber, Jürgen (1. 11. 1997). „Long Short-Term Memory”. Neural Computation. 9 (8): 1735—1780. ISSN 0899-7667. PMID 9377276. S2CID 1915014. doi:10.1162/neco.1997.9.8.1735.
^ „Better Language Models and Their Implications”. OpenAI. 2019-02-14. Архивирано из оригинала 2020-12-19. г. Приступљено 2019-08-25.
^ Bahdanau; Cho, Kyunghyun; Bengio, Yoshua (1. 9. 2014). „Neural Machine Translation by Jointly Learning to Align and Translate”. arXiv:1409.0473  [cs.CL].
^ Luong, Minh-Thang; Pham, Hieu; Manning, Christopher D. (17. 8. 2015). „Effective Approaches to Attention-based Neural Machine Translation”. arXiv:1508.04025  [cs.CL].
^ Schmidhuber, Jürgen (1992). „Learning to control fast-weight memories: an alternative to recurrent nets.”. Neural Computation. 4 (1): 131—139. S2CID 16683347. doi:10.1162/neco.1992.4.1.131.
^ Schlag, Imanol; Irie, Kazuki; Schmidhuber, Jürgen (2021). „Linear Transformers Are Secretly Fast Weight Programmers”. ICML 2021. Springer. стр. 9355—9366.
^ Katharopoulos, Angelos; Vyas, Apoorv; Pappas, Nikolaos; Fleuret, François (2020). „Transformers are RNNs: Fast autoregressive Transformers with linear attention”. ICML 2020. PMLR. стр. 5156—5165.
^ He, Cheng (31. 12. 2021). „Transformer in CV”. Transformer in CV. Towards Data Science. Архивирано из оригинала 16. 4. 2023. г. Приступљено 19. 6. 2021.
^ Radford, Alec; Jong Wook Kim; Xu, Tao; Brockman, Greg; McLeavey, Christine; Sutskever, Ilya (2022). „Robust Speech Recognition via Large-Scale Weak Supervision”. arXiv:2212.04356  [eess.AS].
^ Wolf, Thomas; Debut, Lysandre; Sanh, Victor; Chaumond, Julien; Delangue, Clement; Moi, Anthony; Cistac, Pierric; Rault, Tim; Louf, Remi; Funtowicz, Morgan; Davison, Joe; Shleifer, Sam; von Platen, Patrick; Ma, Clara; Jernite, Yacine; Plu, Julien; Xu, Canwen; Le Scao, Teven; Gugger, Sylvain; Drame, Mariama; Lhoest, Quentin; Rush, Alexander (2020). „Transformers: State-of-the-Art Natural Language Processing”. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. стр. 38—45. S2CID 208117506. doi:10.18653/v1/2020.emnlp-demos.6.
^ „Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing”. Google AI Blog. 2. 11. 2018. Архивирано из оригинала 2021-01-13. г. Приступљено 2019-08-25.

Literatura

Hubert Ramsauer et al. (2020), "Hopfield Networks is All You Need" Архивирано 2021-09-18 на сајту Wayback Machine, preprint submitted for ICLR 2021. arXiv:2008.02217; see also authors' blog Архивирано 2021-09-18 на сајту Wayback Machine

– Discussion of the effect of a transformer layer as equivalent to a Hopfield update, bringing the input closer to one of the fixed points (representable patterns) of a continuous-valued Hopfield network

Alexander Rush, The Annotated transformer Архивирано 2021-09-22 на сајту Wayback Machine, Harvard NLP group, 3 April 2018
Phuong, Mary; Hutter, Marcus (2022), Formal Algorithms for Transformers, arXiv:2207.09238

[2017_Attention_Is_All_You_Need-1] а ^б Vaswani, Ashish; Shazeer, Noam; Parmar, Niki; Uszkoreit, Jakob; Jones, Llion; Gomez, Aidan N; Kaiser, Łukasz; Polosukhin, Illia (2017). „Attention is All you Need” (PDF). Advances in Neural Information Processing Systems. Curran Associates, Inc. 30.

[lstm1997-2] Hochreiter, Sepp; Schmidhuber, Jürgen (1. 11. 1997). „Long Short-Term Memory”. Neural Computation. 9 (8): 1735—1780. ISSN 0899-7667. PMID 9377276. S2CID 1915014. doi:10.1162/neco.1997.9.8.1735.

[:7-3] „Better Language Models and Their Implications”. OpenAI. 2019-02-14. Архивирано из оригинала 2020-12-19. г. Приступљено 2019-08-25.

[inventors-4] Bahdanau; Cho, Kyunghyun; Bengio, Yoshua (1. 9. 2014). „Neural Machine Translation by Jointly Learning to Align and Translate”. arXiv:1409.0473  [cs.CL].

[inventconfirm-5] Luong, Minh-Thang; Pham, Hieu; Manning, Christopher D. (17. 8. 2015). „Effective Approaches to Attention-based Neural Machine Translation”. arXiv:1508.04025  [cs.CL].

[transform1992-6] Schmidhuber, Jürgen (1992). „Learning to control fast-weight memories: an alternative to recurrent nets.”. Neural Computation. 4 (1): 131—139. S2CID 16683347. doi:10.1162/neco.1992.4.1.131.

[schlag2021-7] Schlag, Imanol; Irie, Kazuki; Schmidhuber, Jürgen (2021). „Linear Transformers Are Secretly Fast Weight Programmers”. ICML 2021. Springer. стр. 9355—9366.

[fastlinear2020-8] Katharopoulos, Angelos; Vyas, Apoorv; Pappas, Nikolaos; Fleuret, François (2020). „Transformers are RNNs: Fast autoregressive Transformers with linear attention”. ICML 2020. PMLR. стр. 5156—5165.

[9] He, Cheng (31. 12. 2021). „Transformer in CV”. Transformer in CV. Towards Data Science. Архивирано из оригинала 16. 4. 2023. г. Приступљено 19. 6. 2021.

[Robust_Speech_Recognition_via_Large-Scale_Weak_Supervision-10] Radford, Alec; Jong Wook Kim; Xu, Tao; Brockman, Greg; McLeavey, Christine; Sutskever, Ilya (2022). „Robust Speech Recognition via Large-Scale Weak Supervision”. arXiv:2212.04356  [eess.AS].

[wolf2020-11] Wolf, Thomas; Debut, Lysandre; Sanh, Victor; Chaumond, Julien; Delangue, Clement; Moi, Anthony; Cistac, Pierric; Rault, Tim; Louf, Remi; Funtowicz, Morgan; Davison, Joe; Shleifer, Sam; von Platen, Patrick; Ma, Clara; Jernite, Yacine; Plu, Julien; Xu, Canwen; Le Scao, Teven; Gugger, Sylvain; Drame, Mariama; Lhoest, Quentin; Rush, Alexander (2020). „Transformers: State-of-the-Art Natural Language Processing”. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. стр. 38—45. S2CID 208117506. doi:10.18653/v1/2020.emnlp-demos.6.

[:6-12] „Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing”. Google AI Blog. 2. 11. 2018. Архивирано из оригинала 2021-01-13. г. Приступљено 2019-08-25.

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]