Announcing Stable Code Alpha

Stability AI has just announced the release of Stable Code Alpha 3B, its very first LLM generative AI product for coding. This product is designed to assist programmers with their daily work while providing a great learning tool for new developers ready to take their skills to the next level.

 “a colorful parrot with glasses typing away at a computer, flat icon, vector” – SDXL 0.9

Stable Code Alpha offers a unique way for developers to become more efficient by using three different models to help in their coding. The base model was first trained on a diverse set of programming languages from the stack-dataset (v1.2) from BigCode and then trained further with popular languages like Python, Go, Java, Javascript, C, markdown, and C++.  In total, we trained our models on 560B tokens of code on our HPC cluster. 

After the base model had been established, the instruction model was tuned for specific use cases to help solve complex programming tasks. ~120,000 code instruction/response pairs in Alpaca format were trained on the base model to achieve this result. 

Code for using StableCode Instruct to generate a response to a given instruction.

Stable Code Alpha is the ideal building block for those wanting to learn more about coding, and the long-context window model is the perfect assistant to ensure single and multiple-line autocomplete suggestions are available for the user. This model is built to handle a lot more code at once (2-4X more than previously released open models with a context window of 16,000 tokens), allowing the user to review or edit the equivalent of up to five average-sized Python files at the same time, making it the ideal learning tool for a beginner who wants to rise to bigger challenges.

Stable Code Alpha completing a relatively complex python file utilizing the Pytorch deep learning library (gray text shows StableCode’s prediction).

And here is how we compare to other models of a similar number of parameters and number of tokens trained on. We use the standard pass@1 and pass@10 metrics using the popular HumanEval benchmark.

benchmark scores of stablecode.

HumanEval Benchmark Comparison with models of similar size(3B).

Stability AI aims to make technology more accessible, and Stable Code Alpha is a significant step toward this goal. People of every background will soon be able to create code to solve their everyday problems and improve their lives using AI, and we’d like to help make this happen. We hope that Stable Code Alpha will help the next billion software developers learn to code while providing fairer access to technology all over the world.

Updated 16/01/24