Featured Article

Glass rethinks the smartphone camera through an old-school cinema lens

Prepare to undergo anamorphosis

Comment

CG render of a phone with a prototype anamorphic lens.
Image Credits: Glass

Smartphone cameras have gotten quite good, but it’s getting harder and harder to improve them because we’ve pretty much reached the limit of what’s possible in the space of a cubic centimeter. Glass is a startup looking to fundamentally change how the camera works, using a much bigger sensor and an optical trick from the depths of filmmaking: anamorphic lenses.

It may not be obvious that cameras won’t get better, since we’ve seen such advances in recent generations of phones. But we’ve used up all the slack left in this line, as it were.

To improve the image, you need a bigger sensor, better lens or some kind of computational wizardry. Unfortunately, sensors can’t get much bigger because they’d need bigger lenses to match. And lenses can’t get bigger because there’s just no room for them in the phone body, even when you “fold” the camera. Meanwhile, computational photography is great, but there’s only so much it can do — stacking a few images to get better dynamic range or depth information is good, but you reach a point of diminishing returns pretty quickly.

“The limitations used to be about price, but now it’s size,” explained Glass co-founder and CEO Ziv Attar, who has worked in mobile imaging for over a decade, including at Apple. The other co-founder, Tom Bishop, also worked at Apple, the two of them working on creating Portrait Mode and likely chafing at the limitations of traditional camera design.

“Up to 5 years ago they just made the lens wider, then they started making the sensor bigger,” Attar said. “Then you throw algorithms at it to reduce noise, but even that is reaching its limits; pretty soon it will be pure hallucination [i.e. AI-generated imagery]. Night mode takes exposure stacking to extremes — it deals very nicely with the lack of photons, but if you zoom in it starts to look very weird and fake.”

“The phone screen kind of deceives us,” he continued. “If you let a regular person compare an iPhone 12 and 13, they won’t see the difference — but compared to a pro camera, anyone can tell. And if you can see the difference, there’s a lot of work to do.”

So what is that work, exactly? Attar has decided that of these various conundrums, the only one that makes sense to change is the lens. True, it can’t get any bigger — but only if you’re using a normal, symmetrical lens assembly. But why should we? They gave up on that constraint a century ago in cinema.

Anamorphic evolution

A CG image showing examples of anamorphic (top) and traditional symmetric lenses and the resulting internal image size..
A CG image showing examples of anamorphic (top) and traditional symmetric lenses and the resulting internal image size. Image Credits: Glass

Films weren’t always widescreen. Originally they were more likely to be approximately the shape of a 35mm film frame, for obvious reasons. If you matted out the top and bottom, you could project a widescreen image, which people liked — but you were basically just zooming in on a part of the film, which you paid for in detail. But a technique first tested in the ’20s soon solved the problem.

Anamorphic lenses squeeze a wide field of view from the sides so it fits in the film frame, and when projected using an anamorphic projector, the process was reversed — the image is stretched back out to the desired aspect ratio. There are a few interesting optical effects introduced but… if I describe them you’ll never be able to un-see them in content, so I’ll forbear.

The lens system proposed by Glass isn’t quite the same, but it uses similar principles and unusually shaped lenses. It started from the fundamental idea of how to add a larger sensor. Simply making a larger square would necessitate a larger lens, which we can’t do — but what if you made the sensor longer, as in a rectangle? Well, you’d need a longer, rectangular lens too. The anamorphic technique means you can capture and project a larger but distorted image, then convert it to the right aspect ratio in the image processor. (The process isn’t exactly analogous to the film technique but it uses the same principles.)

How much larger an image are you able to capture? Well, an iPhone 13’s main camera has a sensor about 7×5 millimeters, so 35 square mm total. Glass’s prototype uses a sensor that’s about 24×8 mm: about 192 square mm, 5-6 times larger, with a commensurate increase to megapixels. Here’s a little chart for casual reference:

Image Credits: Devin Coldewey / TechCrunch

Considering the fanfare that generally accompanies increasing a phone’s sensor size by 15 or 20 percent, that’s an enormous leap.

But Attar explained that the way they measure it, it’s even more. If you were to expand the image to the correct aspect ratio, it would actually be twice as tall: 24x16mm, just shy of the APS-C standard in DSLRs but well above the Micro Four Thirds and 1″ sensors also common (and highly performant) in mirrorless cameras. That leads to the company’s claim of having 11 times the “imaging area” of an iPhone. The evaluation of these metrics is a non-trivial process I’m not equipped to do, but truthfully either one would be a game-changing upgrade for a phone.

Bigger, brighter and a bit weirder

There are benefits and drawbacks to this process. The most important one is an immense increase in light gathering and resolving power. More light means better exposures in general and better shots in challenging conditions — no need for a fancy machine learning powered multi-exposure night mode if you can just… see things. And there is far, far more detail in images compared with those from ordinary smartphones.

Images from an iPhone 12 Max (left) and Glass prototype. Image Credits: Glass

Note that the limited example above is just that — it’s hard to do apples-to-apples comparisons when the focal lengths, image processing and output resolution are so different (not to mention my cropping and re-encoding), but at the very least you can see that a great deal of detail is added even in this non-optimal presentation. The full-size original images are available here: iPhone, Glass.

An example of a very low-light exposure — iPhone left, Glass right. Image Credits: Glass

Because of the larger sensor and the nature of the glass, you also get natural bokeh, or background blur. Portrait mode is of course a favorite among smartphone users, but even the best methods of simulating bokeh are far from perfect. The same effect Apple painstakingly simulated lenses to achieve occurs naturally on the Glass prototype, just as it would on a larger digital camera. And there’s no chance of the kind of weird mistake you see in the AI-segmented images, which often clip out hair and other details, or fail to achieve the depth effect in subtler ways.

Example image showing portrait mode on an iPhone (left) and the unprocessed Glass shot, which lacks the smoothing and artifacts of the manipulated one. Image Credits: Glass

While there would be no optical zoom, Attar pointed out that zooming in by cropping (i.e. digital zoom) on a Glass system would let you zoom in more than most optical zooms out there, and you’d still have more light and pixels than the competition. I’m not normally one to let “digital zoom is fine” claims live, but in this case the sheer size of the lens and sensor more than make up for it.

These benefits, though briefly stated, are more than considerable. The improvement to light and detail puts it way out in front of the best cameras out there. (And while the smallest details may escape your notice on a small screen, a bad exposure is noticeable at any size.)

Drawbacks are mainly to do with the complexities of operating a camera that’s totally optically different from a traditional one. The mechanisms for autofocus are different (anamorphic focus is notoriously complex) and there are plenty of distortions and aberrations that need to be corrected for — symmetrical lenses at this size also have distortion and degradation, but of a different type.

“[Distortions] are all constrained during design such that we know in advance that we can correct for them,” said Attar. “It’s an iterative process but we did kick start development of a custom dedicated software tool to co-optimize lens parameters and neural network variables.” In other words they didn’t design anything they couldn’t correct for.

One effect I find disorientating but perhaps others will decide is trivial is the shape of the bokeh. Normally out of focus highlights blur out into little translucent discs, but in the Glass system they resolve into a gradient of ovals and chubby crescents.

Example image showing background blur behind a plant stem.
Image Credits: Glass

To my neurotic eye that just isn’t right. It’s… unnatural. But I also can’t not notice vignetted bokeh due to french flags in film and TV (don’t look it it up — this too is everywhere and you can’t unsee it). And anyway films shot in anamorphic show similar bokeh distortion, so it’s actually quite common, just not in still images and smartphone shots.

I assumed there would be drawbacks due to the need to stretch the image digitally — that sort of thing if done poorly can lead to moiré and other unwanted artifacts. But Attar said it’s remarkably straightforward to train a model to do it so that no one can tell the difference except pixel peepers: “We trained networks to apply 1-D super-resolution based on information from the other axes. After we apply our algorithm it looks like it came from a full APS-C sensor, in field of view and resolution.”

That will all have to be verified by reviewers and camera experts when there’s a production version, but the theory seems sound and the early results are more than promising.

Right now the company has moved on from standalone prototypes to a third-generation phone factor device that shows how the tech will fit into pretty much any chassis on the market. There’s nothing exotic about it other than the optical qualities, Attar said, so although it won’t be as cheap to manufacture as today’s off-the-shelf camera and image processing units, it can be made just as easily. As he noted, price is hardly an option any more, and if one company can make a huge leap in camera quality they can capture a large chunk of the market.

“We have to convince a phone maker to basically ditch the old technology,” said Attar. “We’re seeing nice feedback. The only challenge is doing it in a reasonable time. I’m not saying there’s no risk. But a lot of us had good jobs at big companies — we didn’t leave our fancy salaries at Apple to work on some BS thing. We had a plan from the beginning.”

Even if an agreement was struck now with a big mobile manufacturer, it would take a year and half or two years to get to market. “But we have to start somewhere,” he concluded.

Glass has raised $2.2 million in seed funding, led by LDV Capital and a collection of angel investors. Of course that’s not meant to cover the cost of manufacturing, but now that the company is leaving the lab it will need operating cash to commercialize even should a major manufacturer make a commitment. Greg Gilley, formerly Apple’s VP of cameras and photos, and MIT Media lab’s Ramesh Raskar joined as advisors, rounding out a team investors are likely to have a lot of confidence in.

If the Glass approach catches on, expect to hear about other companies claiming to have invented it in a little less than two years.

More TechCrunch

The spam reached Bluesky by first crossing over two other decentralized networks: Mastodon and Nostr.

The ‘vote Trump’ spam that hit Bluesky in May came from decentralized rival Nostr

Welcome to TechCrunch Fintech! This week, we’re looking at the continued fallout from Synapse’s bankruptcy, how Layer wants to disrupt SMB accounting, and much more! To get a roundup of…

There’s a real appetite for a fintech alternative to QuickBooks

The company is hoping to produce electricity at $13 per megawatt hour, which would be more than 50% cheaper than traditional onshore wind.

Bill Gates-backed wind startup AirLoom is raising $12M, filings reveal

Generative AI makes stuff up. It can be biased. Sometimes it spits out toxic text. So can it be “safe”? Rick Caccia, the CEO of WitnessAI, believes it can. “Securing…

WitnessAI is building guardrails for generative AI models

It’s not often that you hear about a seed round above $10 million. H, a startup based in Paris and previously known as Holistic AI, has announced a $220 million…

French AI startup H raises $220M seed round

Hey there, Series A to B startups with $35 million or less in funding — we’ve got an exciting opportunity that’s tailor-made for your growth journey! If you’re looking to…

Boost your startup’s growth with a ScaleUp package at TC Disrupt 2024

TikTok is pulling out all the stops to prevent its impending ban in the United States. Aside from initiating legal challenges against the government, that means shaping up its public…

As a U.S. ban looms, TikTok announces a $1M program for socially driven creators

Microsoft wants to put its Copilot everywhere. It’s only a matter of time before Microsoft renames its annual Build developer conference to Microsoft Copilot. Hopefully, some of those upcoming events…

Microsoft’s Power Automate no-code platform adds AI flows

Build is Microsoft’s largest developer conference and of course, it’s all about AI this year. So it’s no surprise that GitHub’s Copilot, GitHub’s “AI pair programming tool,” is taking center…

GitHub Copilot gets extensions

Microsoft wants to make its brand of generative AI more useful for teams — specifically teams across corporations and large enterprise organizations. This morning at its annual Build dev conference,…

Microsoft intros a Copilot for teams

Microsoft’s big focus at this year’s Build conference is generative AI. And to that end, the tech giant announced a series of updates to its platforms for building generative AI-powered…

Microsoft upgrades its AI app-building platforms

The U.K.’s data protection watchdog has closed an almost year-long investigation of Snap’s AI chatbot, My AI — saying it’s satisfied the social media firm has addressed concerns about risks…

UK data protection watchdog ends privacy probe of Snap’s GenAI chatbot, but warns industry

U.S. cell carrier Patriot Mobile experienced a data breach that included subscribers’ personal information, including full names, email addresses, home ZIP codes and account PINs, TechCrunch has learned. Patriot Mobile,…

Conservative cell carrier Patriot Mobile hit by data breach

It’s been three years since Spotify acquired live audio startup Betty Labs, and yet the music streaming service isn’t leveraging the technology to its fullest potential — at least not…

Spotify’s ‘Listening Party’ feature falls short of expectations

Alchemist Accelerator has a new pile of AI-forward companies demoing their wares today, if you care to watch, and the program itself is making some international moves into Tokyo and…

Alchemist’s latest batch puts AI to work as accelerator expands to Tokyo, Doha

“Late Pledge” allows campaign creators to continue collecting money even after the campaign has closed.

Kickstarter now lets you pledge after a campaign closes

Stack AI’s co-founders, Antoni Rosinol and Bernardo Aceituno, were PhD students at MIT wrapping up their degrees in 2022 just as large language models were becoming more mainstream. ChatGPT would…

Stack AI wants to make it easier to build AI-fueled workflows

Pinecone, the vector database startup founded by Edo Liberty, the former head of Amazon’s AI Labs, has long been at the forefront of helping businesses augment large language models (LLMs)…

Pinecone launches its serverless vector database out of preview

Young geothermal energy wells can be like budding prodigies, each brimming with potential to outshine their peers. But like people, most decline with age. In California, for example, the amount…

Special mud helps XGS Energy get more power out of geothermal wells

Featured Article

Sonos finally made some headphones

The market play is clear from the outset: The $449 headphones are firmly targeted at an audience that would otherwise be purchasing the Bose QC Ultra or Apple AirPods Max.

5 hours ago
Sonos finally made some headphones

Adobe says the feature is up to the task, regardless of how complex of a background the object is set against.

Adobe brings Firefly AI-powered Generative Remove to Lightroom

All cars suffer when the mercury drops, but electric vehicles suffer more than most as heaters draw more power and batteries charge more slowly as the liquid electrolyte inside thickens.…

Porsche Ventures invests in battery startup South 8 to boost cold-weather EV performance

Scale AI has raised a $1 billion Series F round from a slew of big-name institutional and corporate investors including Amazon and Meta.

Data-labeling startup Scale AI raises $1B as valuation doubles to $13.8B

The new coalition, Tech Against Scams, will work together to find ways to fight back against the tools used by scammers and to better educate the public against financial scams.

Meta, Match, Coinbase and others team up to fight online fraud and crypto scams

It’s a wrap: European Union lawmakers have given the final approval to set up the bloc’s flagship, risk-based regulations for artificial intelligence.

EU Council gives final nod to set up risk-based regulations for AI

London-based fintech Vitesse has closed a $93 million Series C round of funding led by investment giant KKR.

Vitesse, a payments and treasury management platform for insurers, raises $93M to fuel US expansion

Zen Educate, an online marketplace that connects schools with teachers, has raised $37 million in a Series B round of funding. The raise comes amid a growing teacher shortage crisis…

Zen Educate raises $37M and acquires Aquinas Education as it tries to address the teacher shortage

“When I heard the released demo, I was shocked, angered and in disbelief that Mr. Altman would pursue a voice that sounded so eerily similar to mine.”

Scarlett Johansson says that OpenAI approached her to use her voice

A new self-driving truck — manufactured by Volvo and loaded with autonomous vehicle tech developed by Aurora Innovation — could be on public highways as early as this summer.  The…

Aurora and Volvo unveil self-driving truck designed for a driverless future

The European venture capital firm raised its fourth fund as fund as climate tech “comes of age.”

ETF Partners raises €285M for climate startups that will be effective quickly — not 20 years down the road