SUI

https://arxiv.org/abs/2305.19466 The Impact of Positional Encoding on Length Generalization in Transformers Length generalization, the ability to generalize from small training context sizes to larger ones, is a critical challenge in the development of Transformer-based language models. Positional encoding (PE) has been identified as a major factor influencing l arxiv.org 설명: https://twitter.com..

https://arxiv.org/abs/2306.00323 Thought Cloning: Learning to Think while Acting by Imitating Human Thinking Language is often considered a key aspect of human thinking, providing us with exceptional abilities to generalize, explore, plan, replan, and adapt to new situations. However, Reinforcement Learning (RL) agents are far from human-level performance in any arxiv.org https://twitter.com/jef..

abs: https://arxiv.org/abs/2306.00739 SQL-PaLM: Improved Large Language ModelAdaptation for Text-to-SQL One impressive emergent capability of large language models (LLMs) is generation of code, including Structured Query Language (SQL) for databases. For the task of converting natural language text to SQL queries, Text-to-SQL, adaptation of LLMs is of paramo arxiv.org 1. 대형 언어 모델(LLM)의 놀라운 능력 중 ..

abs: https://arxiv.org/abs/2306.00008 Brainformers: Trading Simplicity for Efficiency Transformers are central to recent successes in natural language processing and computer vision. Transformers have a mostly uniform backbone where layers alternate between feed-forward and self-attention in order to build a deep network. Here we investigat arxiv.org 1. 트랜스포머는 자연어 처리와 컴퓨터 비전의 최근 성공에 중심 역할을 하는데, ..

https://arxiv.org/abs/2305.19452 Bigger, Better, Faster: Human-level Atari with human-level efficiency We introduce a value-based RL agent, which we call BBF, that achieves super-human performance in the Atari 100K benchmark. BBF relies on scaling the neural networks used for value estimation, as well as a number of other design choices that enable this sca arxiv.org https://github.com/google-re..

https://arxiv.org/abs/2305.20010 Human or Not? A Gamified Approach to the Turing Test We present "Human or Not?", an online game inspired by the Turing test, that measures the capability of AI chatbots to mimic humans in dialog, and of humans to tell bots from other humans. Over the course of a month, the game was played by over 1.5 million arxiv.org 1. 우리는 튜링 테스트에 영감을 받은 온라인 게임인 "Human or Not?"..

Chinchilla 스케일링 법칙 확장 설명: https://twitter.com/Muennighoff/status/1661895337248686081 트위터에서 즐기는 Niklas Muennighoff “How to keep scaling Large Language Models when data runs out? 🎢 We train 400 models with up to 9B params & 900B tokens to create an extension of Chinchilla scaling laws for repeated data. Results are interesting… 🧐 📜: https://t.co/586bWwvpba twitter.com 1. 이 연구에서는 데이터 제한 조건에서 언어 모델을..

abs: https://arxiv.org/abs/2305.19476 Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration A promising technique for exploration is to maximize the entropy of visited state distribution, i.e., state entropy, by encouraging uniform coverage of visited state space. While it has been effective for an unsupervised setup, it tends to struggle in a su arxiv.org code: ht..

abs: https://arxiv.org/abs/2305.19370 Blockwise Parallel Transformer for Long Context Large Models Transformers have emerged as the cornerstone of state-of-the-art natural language processing models, showcasing exceptional performance across a wide range of AI applications. However, the memory demands posed by the self-attention mechanism and the large arxiv.org 1. 트랜스포머 모델은 다양한 AI 응용 분야에서 최첨단 자..

Sam Altman에 따른 OpenAI의 계획 지난 주 저는 Sam Altman 및 20명의 다른 개발자와 함께 OpenAI의 API 및 제품 계획에 대해 논의할 수 있는 특권을 가졌습니다. Sam은 놀라울 정도로 개방적이었습니다. OpenAI의 사명과 AI의 사회적 영향과 관련된 더 큰 그림 질문뿐만 아니라 실용적인 개발자 문제에 대한 토론이 이루어졌습니다. 주요 내용은 다음과 같습니다. 1. OpenAI는 현재 GPU에 크게 제한되어 있습니다. 토론 내내 공통된 주제는 현재 OpenAI가 극도로 GPU 제한적이며 이로 인해 많은 단기 계획이 지연되고 있다는 것입니다. 가장 큰 고객 불만은 API의 안정성과 속도에 대한 것이었습니다. Sam은 그들의 우려를 인정하고 대부분의 문제가 GPU 부족의 결과라..