Looking to cash in on the AI craze, Cloudflare, the cloud services provider, is launching a new collection of products and apps aimed at helping customers build, deploy and run AI models at the network edge.
One of the new offerings, Workers AI, lets customers access physically nearby GPUs hosted by Cloudflare partners to run AI models on a pay-as-you-go basis. Another, Vectorize, provides a vector database to store vector embeddings — mathematical representations of data — generated by models from Workers AI. A third, AI Gateway, is designed to provide metrics to enable customers to better manage the costs of running AI apps.
According to Cloudflare CEO Matthew Prince, the launch of the new AI-focused product suite was motivated by a strong desire from Cloudflare customers for a simpler, easier-to-use AI management solution — one with a focus on cost savings.
“The offerings already on the market are still very complicated — they require stitching together lots of new vendors, and it gets expensive fast,” Prince told TechCrunch in an email interview. “There’s also very little insight currently available on how you’re spending money on AI; observability is a big challenge as AI spend skyrockets. We can help simplify all of these aspects for developers.”
To this end, Workers AI attempts to ensure AI inference always happens on GPUs close to users (from a geographic standpoint) to deliver a low-latency, AI-powered end-user experience. Leveraging ONNX, the Microsoft-backed intermediary machine learning toolkit used to convert between different AI frameworks, Workers AI allows AI models to run wherever processing makes the most sense in terms of bandwidth, latency, connectivity, processing and localization constraints.
Workers AI users can choose models from a catalog to get started, including large language models (LLMs) like Meta’s Llama 2, automatic speech recognition models, image classifiers and sentiment analysis models. With Workers AI, data stays in the server region where it originally resided. And any data used for inference — e.g. prompts fed to an LLM or image-generating model — aren’t used to train current or future AI models.
“Ideally, inference should happen near the user for a low-latency user experience. However, devices don’t always have the compute capacity or battery power required to execute large models such as LLMs,” Prince said. “Meanwhile, traditional centralized clouds are often geographically too far from the end user. These centralized clouds are also mostly based in the U.S., making it complicated for businesses around the world that prefer not to (or legally cannot) send data out of its home country. Cloudflare provides the best place to solve both these problems.”
Workers AI already has a major vendor partner: AI startup Hugging Face. Hugging Face will optimize generative AI models to run on Workers AI, Cloudflare says, while Cloudflare will become the first serverless GPU partner for deploying Hugging Face models.
Databricks is another. Databricks says that it’ll work to bring AI inference to Workers AI through MLflow, the open source platform for managing machine learning workflows, and Databricks’ marketplace for software. Cloudflare will join the MLflow project as an active contributor, and Databricks will roll out MLflow capabilities to developers actively building on the Workers AI platform.
Vectorize targets a different segment of customers: those needing to store vector embeddings for AI models in a database. Vector embeddings, the building blocks of machine learning algorithms used by applications ranging from search to AI assistants, are representations of training data that are more compact while preserving what’s meaningful about the data.
Models in Workers AI can be used to generate embeddings that can then be stored in Vectorize. Or, customers can keep embeddings generated by third-party models from vendors such as OpenAI and Cohere.
Now, vector databases are hardly new. Startups like Pinecone host them, as do public cloud incumbents like AWS, Azure and Google Cloud. But Prince asserts that Vectorize benefits from Cloudflare’s global network, allowing queries of the database to happen closer to users — leading to reduced latency and inference time.
“As a developer, getting started with AI today requires access to — and management of — infrastructure that’s inaccessible to most,” Prince said. “We can help make it a simpler experience from the get-go … We’re able to add this technology to our existing network, allowing us to leverage our existing infrastructure and pass on better performance, as well as better cost.”
The last component of the AI suite, AI Gateway, provides observability features to assist with tracking AI traffic. For example, AI Gateway keeps tabs on the number of model inferencing requests as well as the duration of those requests, the number of users using a model and the overall cost of running an AI app.
In addition, AI Gateway offers capabilities to reduce costs, including caching and rate limiting. With caching, customers can cache responses from LLMs to common questions, minimizing (but presumably not entirely eliminating) the need for an LLM to generate a new response. Rate limiting confers more control over how apps scale by mitigating malicious actors and heavy traffic.
Prince makes the claim that, with AI Gateway, Cloudflare is one of the few providers of its size that lets developers and companies only pay for the compute they use. That’s not completely true — third-party tools like GPTCache can replicate AI Gateway’s caching functionality on other providers, and providers including Vercel deliver rate limiting as a service — but he also argues that Cloudflare’s approach is more streamlined than the competition’s.
We’ll have to see if that’s the case.
“Currently, customers are paying for a lot of idle compute in the form of virtual machines and GPUs that go unused,” Prince said. “We see an opportunity to abstract away a lot of the toil and complexity that’s associated with machine learning operations today, and service developers’ machine learning workflows through a holistic solution.”