Grok-1 is an open source release by xAI, featuring the base model weights and network architecture of their large language model (LLM). With 314 billion parameters, Grok-1 is a powerful Mixture-of-Experts model that was trained from scratch by xAI.
This open source release provides access to the raw base model checkpoint from the Grok-1 pre-training phase, which was concluded in October 2023. It’s important to note that the model has not been fine-tuned for any specific application, such as dialogue.
Under the Apache 2.0 license, the weights and architecture of Grok-1 are now available for use. Developers interested in utilizing the model can find instructions on how to get started at github.com/xai-org/grok.
Grok-1 is a remarkable language model, trained on a substantial amount of text data. It is a 314 billion parameter Mixture-of-Experts model, with 25% of the weights active on a given token. The model was trained from scratch by xAI, using a custom training stack built on top of JAX and Rust in October 2023.
For more information, you can visit the Open Release of Grok-1 on x.ai’s blog.