活动报名｜如何使用70万预算从头训练千亿语言大模型

以GPT系列为代表的语言大模型已经取得了显著的成功，但是其高昂的成本限制了大模型进一步的快速发展。同时，这也给学术界和工业界带来了新的机遇和挑战。为了进一步降低模型成本，我们采用了生长策略，成功地将千亿稠密大模型的成本降低到70万。

此外，为了更加全面合理地评估大模型，在目前已有的知识类评估的基础上，借鉴IQ测试的概念，提出了大模型的IQ测试方案。实验显示，70万训练成功的千亿大模型表现了非常好的能力。我们相信生长策略可以为突破单体稠密万亿模型带来全新的可能性。

Large language models (LLMs) have achieved remarkable success in NLP and multimodal tasks. However, their high costs constrain the further development of LLMs, which also brings both opportunities and challenges for academia and industry. To break down this barrier, FLM-101B employs a growth strategy and successfully lowers the cost of training a 100B-level dense model down to ￥700,000 CNY. Additionally, in order to evaluate LLMs systematically and more rationally, besides existing knowledge-based assessments, the IQ test in LLMs, whose concept is partially borrowed from psychology, is proposed. Experimental results show that the model trained with a budget of ￥700K, achieves comparable performance to powerful and well-known models and demonstrates impressive capabilities. We believe that the growth strategy offers new possibilities for breakthroughs in training 1T+ dense models.

活动时间：9月21日（周四）14:30-15:30