Anthropic retrains Mythos with less compute, showing algorithmic progress

Algorithmic progress - what superexponential growth means

Superexponential growth refers to a rate of progress that speeds up faster than exponential growth. In AI, this means improvements in algorithms are happening at an extremely fast pace. Such growth indicates that each new development makes training more efficient than the last.

This kind of progress can lead to dramatic reductions in the compute needed to train large models. It also suggests that AI research is entering a phase of rapid, accelerating improvements. The growth pattern seen in the slide shows that algorithmic efficiency is improving not just quickly but at an increasing rate. This can make AI development more accessible and less costly over time.

Retraining Mythos - how less compute is now sufficient

Jacob Coxon, a former researcher at OpenAI and Anthropic, states that Anthropic can now retrain Mythos using only a fraction of the compute originally needed. This means that the training process has become much more efficient. Coxon points out that Mythos was part of a superexponential trend in AI progress.

This efficiency gain suggests that the improvements in algorithms are enabling faster retraining. Instead of requiring vast amounts of computing power, teams can now update models more quickly and cheaply. Such progress can help AI teams iterate faster and deploy new capabilities more often.

Implications for AI development - automation and competition

The rapid improvements in algorithmic efficiency are on the verge of being automated. Coxon believes that next year, much of the process of optimizing models could be handled by AI itself. This automation could make AI development more scalable and less dependent on human effort.

These advances could also intensify competition among AI companies. Faster retraining and better efficiency mean models can be improved more frequently. This could lead to a faster pace of innovation and new capabilities. However, Coxon warns that rushing AI development without proper oversight could pose risks.

Why it matters - impact on costs and speed

Lower compute requirements directly reduce the costs of training and updating AI models. Teams can retrain models more often without needing huge budgets. This can accelerate the deployment of new features and improvements.

Faster retraining also means AI models can adapt more quickly to new data or tasks. This improves the relevance and accuracy of AI systems. For managers and engineers, these developments can lead to more agile AI projects and lower barriers to experimentation.

How OpenSmartRoute helps - routing AI requests efficiently

OpenSmartRoute is an open-source router for AI requests. It directs each request to the best-fit model, agent, or tool from a catalog defined by the team. It scores candidates based on quality, cost, speed, and safety, with the team setting the priorities.

Using OpenSmartRoute allows teams to optimize how requests are handled. They can pin certain requests to specific models or regions, or set cost caps. The system learns from outcomes, giving more traffic to better-performing models. This helps teams control costs and improve response quality.

By routing requests efficiently, OpenSmartRoute helps teams take advantage of improvements like those in Mythos retraining. It ensures that AI requests are handled by the most suitable models, saving time and money. The platform also provides visibility into costs and performance, supporting better decision-making.

How it compares - what existed before, what this changes and what stays the same

Before, retraining large AI models required massive amounts of compute power. This made updates slow and expensive. Teams had to plan carefully for retraining, often waiting until they had enough resources.

Now, Coxon says that models like Mythos can be retrained with much less compute. This means that the process becomes cheaper and faster. Teams can update models more often and respond quickly to new data or tasks.

This change does not mean that all models are now easy to retrain. It applies specifically to models like Mythos, which Coxon says can be retrained with a fraction of the original compute. Larger, more complex models may still need more resources. The core idea is that algorithmic improvements are making retraining more efficient.

What stays the same is that large AI models still need data, careful tuning, and testing. The improvements in efficiency do not eliminate the need for good training data or evaluation. They simply reduce the amount of compute needed for retraining.

This shift could lead to a new way of developing AI. Instead of waiting for huge compute resources, teams can retrain models more frequently with smaller budgets. This might change how companies plan their AI projects and updates.

Questions this leaves open - what the source does not say and how a reader can check it

The source does not specify exactly how much less compute is needed to retrain Mythos. It mentions a "fraction" but does not give precise numbers or percentages. Without these details, it is hard to judge the scale of improvement.

It also does not explain how Coxon measured or estimated the compute savings. Did he base this on internal experiments, or is it an educated guess? Knowing this would help assess how reliable the claim is.

The source does not describe the specific algorithmic improvements that made retraining more efficient. Are these new training techniques, better data handling, or other innovations? More technical details would clarify how these gains are achieved.

Another open question is whether these improvements apply only to Mythos or to other models as well. Coxon suggests a trend, but it is unclear how widespread or general these efficiency gains are.

Readers interested in verifying these claims can look for published research or technical reports from Anthropic. They can also compare these statements with other industry sources or independent evaluations. Experimenting with smaller models or open-source alternatives might give practical insights into retraining costs.

Finally, it is not clear how these efficiency gains will evolve over time. Will future models require even less compute, or are there limits? Monitoring ongoing research and industry trends can help answer this question.

In summary, while Coxon’s claims point to significant improvements, more detailed data and independent confirmation are needed. Staying informed through technical publications and experiments can help teams understand the true potential of these advancements.

What to do - next steps for model teams and managers

Teams should monitor advances in algorithmic efficiency and retraining methods. They can explore ways to incorporate automated retraining into their workflows. Using tools like OpenSmartRoute can help manage and optimize model requests.

Managers should consider how faster retraining and lower costs impact their AI strategies. They can plan for more frequent updates and experiment with different models. Staying informed about progress in AI efficiency can help teams stay competitive and innovate faster.

Next, teams might set up routing systems that adapt to new models and capabilities as they become available. This approach ensures they can quickly leverage improvements like Mythos retraining with less compute. It also helps control costs and improve response quality across AI applications.

Source: https://the-decoder.com/anthropic-can-already-retrain-mythos-with-a-fraction-of-the-compute-originally-needed-says-former-researcher-coxon/