Today, an improved Claude 3.5 Sonnet and a new model, the Claude 3.5 Haiku, are being introduced. Though it was already at the top of the field, the revised Claude 3.5 Sonnet performs better than its predecessor in every aspect, especially in coding.
It is also introducing computer use, a ground-breaking new capability, in public beta. With the help of the now-available API, developers can teach Claude to use computers just like people do by pointing at a screen, moving a cursor, hitting buttons, and writing text. Claude 3.5 Sonnet is the first frontier AI model released for public beta testing. At this stage, it is still experimental and can be challenging and error-prone. Claude is exposing PC use early for developer feedback and expects the capacity to improve rapidly over time.Businesses that require dozens or even hundreds of phases, such as Asana, Canva, Cognition, DoorDash, Replit, and The Browser Company, have already begun to explore these possibilities. For example, Replit is using the computer use and UI navigation capabilities of Claude 3.5 Sonnet to develop an essential component for their Replit Agent product that evaluates apps as they are being developed.
The updated Claude 3.5 Sonnet is now available to all users. The computer-based beta is now available to developers on Amazon Bedrock, Google Cloud's Vertex AI, and the Anthropic API. The new Claude 3.5 Haiku will be on the market later this month.
Claude 3.5 Sonnet: renowned software engineering specialist
The improved Claude 3.5 Sonnet shows significant gains on jobs involving tool use and agentic coding, among other advances on industrial benchmarks. Its performance on SWE-bench Verified increased from 33.4% to 49.0%, surpassing all publicly accessible models in terms of coding, including reasoning models such as OpenAI o1-preview and specialized systems designed for agentic coding. Furthermore, it improves performance on the TAU-bench agentic tool usage task from 62.6% to 69.2% in the retail sector and from 36.0% to 46.0% in the more challenging airline domain. The new Claude 3.5 Sonnet offers these enhancements at the same price and speed as the original.The upgraded Claude 3.5 Sonnet represents a significant breakthrough in AI-powered coding, based on early user feedback. GitLab, which evaluated the model for DevSecOps tasks, found that it offered stronger reasoning (up to 10% across use cases) and enabled multi-step software development processes without adding extra latency. Cognition used the new Claude 3.5 Sonnet for autonomous AI evaluations and demonstrated notable improvements in coding, planning, and problem-solving abilities compared to the prior edition. When the Browser Company utilized Claude 3.5 Sonnet to automate web-based workflows, they found that it outperformed all other models they had attempted.
As part of its continuous effort to work with external experts, the US AI Safety Institute (US AISI) and the UK Safety Institute (UK AISI) collaboratively pre-deployed the updated Claude 3.5 Sonnet model.
Its evaluation of the improved Claude 3.5 Sonnet for catastrophic risks indicates that the ASL-2 Standard, as outlined in its Responsible Scaling Policy, is still appropriate for this model.
Claude 3.5 Haiku: Innovative, quick, and reasonably priced
Claude 3.5 Haiku is the subsequent version of Claude's fastest model. For the same price and speed as Claude 3 Haiku, Claude 3.5 Haiku improves across all skill sets and surpasses even Claude 3 Opus, the largest model in its previous generation, on the majority of intelligence benchmarks. Claude 3.5 Haiku does exceptionally well on coding tasks. For example, with a score of 40.6% on SWE-bench Verified, it beats many agents using publicly available state-of-the-art models, like the original Claude 3.5 Sonnet and GPT-4o.Claude 3.5 Haiku is perfect for user-facing products, specialized sub-agent tasks, and building personalized experiences from vast volumes of data, like pricing, inventory records, or purchase histories, because of its low latency, improved instruction following, and more accurate tool use.
Use cases
Because of its fast speeds, improved command following, and more accurate tool utilization, Claude 3.5 Haiku is perfect for user-facing products, specialized sub-agent tasks, and producing individualized experiences from vast volumes of data. Examples of typical use cases are:Code completions
Claude 3.5 Haiku provides accurate, quick code completions and suggestions, which expedites development processes. It's great for software teams looking to improve efficiency and optimize their coding process.Interactive chatbots
Claude 3.5 is more capable of speaking and reacting quickly. Haiku is great at enabling chatbots that can handle a lot of user interactions and are responsive. It will be incredibly helpful for e-commerce, customer service, and educational platforms that require scaled interaction.Data extraction and labeling
Claude 3.5 Haiku efficiently processes and categorizes information, making it helpful for rapid data extraction and automatic labeling tasks. This capability may be especially useful for organizations in the research, healthcare, and financial sectors that deal with large volumes of unstructured data.Real-time content moderation
Claude 3.5 Reliable, real-time content moderation is made possible by Haiku's improved reasoning and content comprehension abilities. It is therefore helpful to media firms, social media platforms, and online forums that must continuously deliver safe and suitable information.Cost and accessibility
Later this month, Google Cloud's Vertex AI and its first-party API, Amazon Bedrock, will make Claude 3.5 Haiku available as a text-only model with the ability to input images.Claude 3.5 Haiku offers up to 90% cost savings with rapid caching and 50% cost savings with the Message Batches API, with starting prices of $0.25 per million input tokens and $1.25 per million output tokens.
Claude is receiving instruction on how to use a computer responsibly.
Claude is using a computer to try something basically new. Instead of developing customized tools to help Claude with particular activities, it is teaching him general computer abilities that will allow it to use a range of traditional tools and software applications designed for humans. Developers can utilize this new capability to automate repetitive processes, build and test software, and perform open-ended tasks like research.
Claude created an API that enables him to view and interact with computer interfaces, enabling these general skills. To allow Claude to translate requests (such as "check a spreadsheet," "move the cursor," and "use data from my computer and online to fill out this form") into computer commands With the help of this API, developers can "open a web browser," "navigate to the relevant web pages," "fill out a form with the data from those pages," and more.
On OSWorld, a website that evaluates AI models' computer skills, Claude 3.5 Sonnet scored 14.9% in the screenshot-only category. This is much higher than the next-best AI system's score of 7.8%. When Claude was granted extra steps to complete the task, he scored 22.0%.
Claude's computer abilities are now limited, but he expects this capacity to rapidly develop in the following months. Because Claude currently struggles to do certain operations that people accomplish effortlessly, such as scrolling, dragging, and zooming, it suggests that developers begin their experiments with low-risk tasks. It actively encourages the safe use of computers since they might present a fresh opportunity for more well-known issues like spam, fraud, or misinformation. It has produced new classifiers that are able to identify when computer use is occurring and when harm is occurring. You may read more about the research process that led to this new capability and other safety measures in its article on developing computer use.
Taking the future into account
As we gain insight from the early applications of this still-emerging technology, we will be able to better understand the potential and implications of increasingly potent AI systems.
Amazon Bedrock offers the updated Claude 3.5 Sonnet from Anthropic (available now), Claude 3.5 Haiku (coming soon), and PC usage (public beta).
The upgraded Claude 3.5 Sonnet is now available on Amazon Bedrock in the US West (Oregon) AWS Region and costs the same as the original.
In addition to the enhanced intelligence of the model, developers may now include computer use (available in public beta) into their apps to enhance software testing processes, automate complex desktop activities, and create ever-more complex AI-powered applications.
Claude 3.5 Haiku will be released in the coming weeks, initially as a text-only model before adding visuals.
0 Comments