EXPLORINEER
en

Archive — US — techcrunch

One of Google’s recent Gemini AI models scores worse on safety

Google's Gemini 2.5 Flash AI model has been found to perform worse on safety benchmarks compared to its predecessor, Gemini 2.0 Flash, particularly in text-to-text and image-to-text safety tests. Attendant issues arise as AI models become more permissive, leading to potential safety policy violations and a need for transparency in testing practices.

Key points

  • Gemini 2.5 Flash scores worse on safety tests compared to Gemini 2.0 Flash; regressed by 4.1% in text-to-text safety and 9.6% in image-to-text safety.
  • Both safety tests are automated; higher scores indicate adherence to guidelines.
  • The trend in AI development shows companies making models more permissive, raising concerns about user safety.
  • False positives are cited as one reason for the regressions in safety.
  • Google claims Gemini 2.5 Flash follows instructions better but admits it still generates violative content.
  • There is an ongoing tension between providing diverse opinions and maintaining safety.
  • Transparency in model testing has been questioned, with calls for more detail on violations from external analysts.

Why it matters

  • The trend of AI models becoming more permissive has parallels in prior controversies, such as OpenAI's ChatGPT enabling inappropriate content.
  • Similar safety concerns were raised during the development of earlier AI models that prioritized user freedom over guideline adherence.
  • As AI tools evolve, the balance between user engagement and safety measures is becoming a significant topic in the industry.

The struggle between permissiveness and safety in AI development continues to challenge tech companies, exemplified by Google's Gemini 2.5 Flash model. This highlights an urgent need for enhanced reporting practices and transparency to ensure responsible AI deployment.