A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis

What happened

A researcher showed 200 skin lesions to two advanced AI models and to a 10-year-old image model. Each lesion had a clinical photo and a dermoscopic photo (taken through a special magnifying lens). The old model was fine-tuned on public skin images and named the right diagnosis more often than either chatbot.

Key numbers

• The CNN got the right diagnosis for 125 of 200 lesions (62%).
• The two AI chatbots got 84 and 85 right (42%).
• The test had 6 possible diagnoses for each lesion.
• All 200 lesions were checked by biopsy (a tissue sample sent to a lab), including the harmless ones.
• With a second guess allowed, the CNN reached 173 of 200.

How the test worked

The lesions came from a public dataset called MILK10k. The chatbots saw only the two images and a short prompt. They were asked for the most likely diagnosis, a confidence level, a second choice, and whether the patient could be safely sent home with no treatment or follow-up. The CNN was trained on about 68,000 skin images on one powerful computer chip. The same images of each patient were kept out of its training, so it had never seen the test cases.

Why it matters

Skin cancer, such as melanoma, must be caught early. A wrong answer can mean a dangerous cancer is sent home. In one example, both chatbots called an invasive melanoma a harmless growth and said the patient could go home. The result shows that general-purpose AI is not automatically the best tool for every medical task. A small model built for one job can do better. The author also notes that the CNN is not the newest technology, so specialized tools may still be worth testing.

What to watch

• Whether other researchers repeat this test with more lesions and other datasets.
• Whether newer chatbots improve on skin cancer images.
• How doctors and regulators decide when AI tools are safe enough to support diagnosis.