Welcome back!

OpenAI's o1 model is dominating benchmarks, even scoring 120 on the Mensa IQ test. While IQ tests aren’t entirely meaningful for AI, what truly matters is how o1 is pushing the limits of current evaluations, prompting the need for newer, more challenging benchmarks. Here’s what we found…

Today’s Summary:

• 🎯 OpenAI’s o1 model dominates benchmarks

• 🎥 YouTube Shorts to integrate Veo AI generator

• 🌍 Alibaba Qwen2.5 new king of open-source AI

• 👓 Snap launches AI-powered VR Glasses

• 🎬 Runway AI partners with Lionsgate

• 🎥 Luma unveils Dream Machine video AI API

• 2 new tools

TOP STORY

OpenAI o1 model tops benchmarks

The Summary: OpenAI's new o1 model has achieved remarkable results across multiple benchmarks, including Lmsys Arena, where it ranked highly in technical areas like math and coding. Some have estimated its IQ at 120 using a Mensa IQ test, though such comparisons aren’t entirely relevant for AI. As AI capabilities grow, researchers are working to develop more challenging benchmarks to better evaluate progress.

Key details:

• o1-preview ranked #1 overall in Lmsys with 6K+ community votes

• o1-mini ranked #2 overall and #1 in technical areas

• o1 scored 120 IQ on the Norway Mensa test, past average human score

• Researchers are calling for tougher questions for a new "Humanity's Last Exam" benchmark, with $500,000 in prizes for accepted challenges across all fields

• Some user-reported examples of o1 capabilities:

  • o1 accomplished in 1 hour what took a user a year in their PhD

  • created a cancer treatment framework with innovative strategies

  • created a 3D version of Snake

  • built a fully functional chess game with an AI opponent

  • can reason through complex enterprise tasks, like determining contract dates by understanding embedded rules

Why it matters: The rapid progress of AI models like o1 is outpacing traditional evaluation methods. As these models saturate existing benchmarks, new ways to measure and understand AI's true capabilities are becoming essential.

Explore Lmsys Arena results

TOOLS

AI video generation coming to YouTube

The Summary: YouTube is set to integrate Google DeepMind's Veo video AI model into Shorts, allowing creators to generate high-quality video backgrounds and 6-second clips. This update is expected to roll out in late 2024.

Key details:

• Veo will enable creators to generate standalone clips for Shorts

• AI content will be watermarked with SynthID and labeled as AI-created

• Automatic dubbing will be expanded to support more languages

Why it matters: This integration positions YouTube at the forefront of AI-powered content creation, potentially transforming how creators produce and share videos. As AI tools become more widely accessible, YouTube will need to balance innovation with concerns about content authenticity and creator rights.

Learn about Veo

NEW MODELS

Alibaba Qwen2.5: the new king of open-source AI

The Summary: Alibaba has released Qwen2.5, now ranking as the best open-source AI model according to evaluations. The 72B-parameter version matches GPT-4 performance on key benchmarks, including coding and math tasks. This release is a major milestone for open-source AI.

Key details:

• Qwen2.5-72B achieves a 55.5 score on LiveCodeBench, nearing GPT-4 and surpassing 405B-Llama 3.1

• Trained on an extensive 18 trillion token dataset

• Qwen2.5 Coder outperforms the previous leader DeepSeek in most categories

• Qwen2-Math-72B-Instruct surpasses GPT-4o and Claude 3.5 in math tasks

• Includes QwenVL 72B visual language model

• Available on GitHub and HuggingFace

Why it matters: This advancement brings state-of-the-art AI capabilities to the open-source community, enabling developers and innovators to access free models that rival proprietary systems like GPT-4.

Try it on HuggingFace

QUICK NEWS

Quick news

• Snap launches AI VR Glasses

• Runway AI inks deal with major Hollywood studio Lionsgate

• Luma unveils Dream Machine video generation API

TOOLS

🥇 New tools

• Llamacoder - Generate an entire app from a prompt

• Supademo - Create interactive product demos