Answers from the documentation; checking which model is on duty
Hi - I answer from the OpenSmartRoute documentation: routing, the API, plans and quotas, self-hosting. Ask away, or open a support ticket if you need a person.
Grounded in the docs - follow a source before acting on it.
California pushes for mandatory independent AI risk evaluations - OpenSmartRoute
California pushes for mandatory independent AI risk evaluations
California's government plans to require independent risk assessments for frontier AI developers, replacing voluntary reviews. The move aims to improve safety oversight.
Key points
California proposes phased mandates based on evaluator capacity growth.
Legislation would require ongoing reviews by licensed independent evaluators.
Evaluators would assess models for catastrophic risk and safety outcomes.
The plan includes defining safety standards in law and updating testing methods.
Why it matters: Ensuring independent risk evaluations improves AI safety and reduces potential catastrophic harms.
By OpenSmartRoute editorial · written through the router by llm-small
From Unite.AI - “Fathom Urges Rapid Implementation of California’s AI Oversight Plan”
Announcement - California's government proposes new AI oversight laws
California's government has proposed new laws to oversee frontier artificial intelligence (AI) development. These laws aim to make safety evaluations mandatory instead of voluntary. The goal is to better prevent risks from advanced AI systems. The proposal reflects a shift toward stricter regulation of AI safety.
The proposed laws are part of California’s broader AI roadmap. They focus on requiring independent assessments of AI models for catastrophic risks. The state plans to license evaluators and set clear safety standards in law. These steps aim to improve oversight and accountability for AI developers.
Current voluntary approach - how AI developers are reviewed now
Today, most AI developers review their models voluntarily. They conduct internal tests and sometimes share results with regulators or the public. This approach relies on the developer’s judgment and willingness to report risks. It does not guarantee that risks are fully identified or managed.
Voluntary reviews can vary greatly in quality. Some developers may lack the expertise or motivation to thoroughly assess risks. As a result, safety oversight depends on individual companies’ commitments. This can leave gaps in safety measures for highly capable AI systems.
Proposed model - mandatory independent evaluations by licensed evaluators
The new proposal would require AI developers to undergo evaluations by independent licensed evaluators. These evaluators would be outside experts approved by the state. They would assess AI models for risks that could cause catastrophic harm.
Evaluators would follow technical testing methods defined and updated by the state. Developers would pay for these assessments, but the process would be independent. The goal is to ensure that evaluations are objective, rigorous, and transparent. This model aims to replace the current voluntary system with a mandatory one.
Evaluator licensing and oversight - how evaluators are approved and supervised
U.S. officials have not endorsed Jensen Huang's return as Nvidia CEO amid ongoing investigations into export-control failures and compliance issues.
The state would license and supervise the evaluators. It would set standards for expertise, independence, and rigor. The licensing process would include background checks and assessments of evaluator capabilities.
Once licensed, evaluators would be continuously supervised by the state. The state could revoke licenses if evaluators lack independence or fail to meet standards. Evaluators would submit unredacted assessment reports directly to the state. These reports would inform whether a model is safe to deploy.
Assessment process - how risk is measured and safety outcomes are defined
The assessment process would measure whether an AI model is unlikely to cause catastrophic harm. It would also compare the model’s benefits against residual risks. The law would define what safety outcomes are acceptable.
Evaluators would develop and update technical testing methods. These methods would test for risks like unintended behavior or recursive self-improvement. The process would be flexible enough to adapt as AI technology advances. The goal is to keep safety standards aligned with current risks and capabilities.
Handling recursive self-improvement - monitoring automated AI development
Recursive self-improvement (RSI) occurs when AI systems improve themselves automatically. This can lead to rapid and unpredictable development. The proposal includes measures to monitor and verify automated research and development metrics.
Evaluators would be responsible for assessing risks from RSI. They would check internal and external use of AI systems. When certain capability thresholds are reached, the state would be notified. Further internal scaling would be paused until reassessment occurs. This helps prevent uncontrolled growth of risky AI systems.
Phasing in mandates - how requirements will grow with evaluator capacity
The laws would be phased in gradually. Initially, the state would assign an evaluator from a pre-approved pool. Developers would pay for assessments at state-set rates. Over time, as evaluator capacity increases, the process would expand.
The state would have the authority to veto evaluator choices, require rotations, and limit concentration of evaluators. Payments would be based on time and effort, not the assessment outcome. This phased approach aims to build a sustainable verification system that attracts talent and capital.
Legal and enforcement details - state powers and evaluator responsibilities
The state would hold enforcement powers. It could seek injunctions and impose civil penalties for non-compliance. Evaluators would make findings, prescribe corrective actions, and flag imminent risks. They would not seek injunctions or enforce measures themselves.
Embedded evaluators would have access to necessary records, personnel, and systems. They could be physically present at decision points. Confidentiality and anti-retaliation protections would back these access rights. This structure ensures that evaluations are thorough and that developers cannot hide risks.
Why it matters - the importance of safety standards and oversight
As AI systems become more capable, the risks of catastrophic harm increase. Without proper oversight, these risks could materialize unexpectedly. Mandatory independent evaluations help ensure that safety standards are met before deployment.
This approach aims to prevent accidents and misuse of AI technology. It also builds public trust by making safety a legal requirement. Strong oversight can guide AI development toward safer and more responsible practices.
How OpenSmartRoute helps - what a team routing through OpenSmartRoute does with California pushes for mandatory independent AI risk evaluations
Teams using OpenSmartRoute route their AI requests through a system that scores models on quality, cost, speed, and safety. They define rules to keep personal data on local models and avoid region or cost limits. The system learns from outcomes, giving more traffic to better models.
With California’s push for mandatory evaluations, teams can use OpenSmartRoute to verify model safety. They can measure routing accuracy and ensure models meet safety standards. The platform’s catalog keeps track of model prices and rankings, helping teams select the best options for compliance and safety.
By integrating OpenSmartRoute, teams can also monitor costs and performance as they adapt to new safety requirements. The system’s ability to spot prompt injection and personal data helps enforce safety rules. It provides a clear view of how models perform in real traffic, supporting compliance with California’s new oversight laws.
How it compares - what existed before, what this changes and what stays the same
Before this proposal, AI developers mostly relied on voluntary safety reviews. These reviews were not mandatory and varied widely in quality. Developers often conducted internal testing or used third-party auditors without legal requirements.
This new plan introduces mandatory independent evaluations for large frontier AI models. These evaluations will be conducted by licensed organizations called IVOs. The state will oversee and license these evaluators, ensuring they meet strict standards. This shifts oversight from voluntary to legally required.
The evaluation process will focus on safety outcomes defined by law. It will also include technical testing methods that can evolve with technology. The goal is to make safety assessments more consistent and reliable. The plan also emphasizes phased implementation, so mandates start once evaluator capacity is sufficient.
Some parts of the current system stay the same. Developers still conduct internal testing and may choose their own evaluators initially. The plan aims to gradually increase oversight as the evaluator pool grows. It also keeps the focus on safety outcomes, not just technical measures.
This approach differs from earlier efforts by making oversight a legal requirement. It also emphasizes transparency, with unredacted assessment reports submitted to the state. The plan aims to build a system that can adapt to rapid AI advances while maintaining safety standards.
Questions this leaves open - what the source does not say and how a reader can check it
The source does not specify how many evaluators will be licensed initially. It also does not say how long the phased implementation will take. The timeline for building evaluator capacity remains unclear.
It is not clear what specific safety standards the law will define. The details of what constitutes an acceptable safety outcome are not provided. The process for updating technical testing methods is also not explained.
The source mentions unresolved questions about legislation, rules, and costs. It does not specify how much the evaluation process will cost developers or how it will be funded. The impact on AI development timelines and innovation is also not discussed.
It is unclear how the state will verify the independence and rigor of evaluators over time. The criteria for decertifying evaluators or addressing conflicts of interest are not detailed. The effectiveness of enforcement measures, such as penalties or injunctions, depends on future legal rules.
The source does not describe how evaluations will handle proprietary or sensitive information. Developers may worry about confidentiality and trade secrets. The process for resolving disputes between developers and evaluators is not outlined.
Readers interested in these questions can check the legislation once it is proposed. They can also review the detailed rules for licensing and oversight. Monitoring updates from California’s government agencies will reveal how these issues are addressed. Engaging with industry groups or legal experts can provide insights into implementation challenges.
In summary, while the plan sets a clear direction, many specifics remain to be decided. The success of the oversight system will depend on how these questions are answered in practice.
Anthropic restricted IPO deal teams to four bankers and imposed work restrictions with AI competitors, amid a tight listing process and late-stage changes.