Zero-Shot Search: Search aerial imagery by meaning
)
Zero-Shot Search turns Nearmap aerial imagery into embeddings — compact numerical summaries that capture what’s in each image. That innovation opens up several ways to search and analyse imagery at scale.
Describe anything in plain language (e.g., a blue tarp on a roof, a cracked driveway, a solar array) and Zero-Shot Search finds it. No training or example images required.
The same embeddings do more than search. Compare properties side by side, group similar scenes together, or train a detector for a new feature using just a handful of examples.
Zero-Shot Search also powers tile search and heatmap scoring, enabling full property views and insights you don’t get from imagery alone.
What are embeddings?
A computer can understand an image, but not in the way you do. Your eyes take it in instantly. A computer has to work for it, spending real compute to process every pixel. Embeddings let a computer pay that cost once. A neural network generates a list of numbers representing the image, and that list is far cheaper and faster to work with than the raw image itself.
Similar inputs produce similar numbers. Two photos of a cat and the word “cat” land on the same embedding. A photo of a dog lands on a different one.
)
Every input, a photo or a phrase, comes out of the neural network as an embedding, drawn here as a line. The two cats and the word “cat” trace near-identical lines; the dog’s is clearly different. Lines are illustrative.
Picture each embedding as a point in space. Similar inputs are next to each other, while inputs that aren’t alike are far apart. “How similar are these two?” becomes a question you can measure. Apply this across a full collection of aerial imagery, and it organises itself. Pools cluster in one neighborhood, solar panels in another, and open fields in a third — with nothing telling them where to go.
)
33 real Zero-Shot Search tiles, arranged so that tiles with similar embeddings sit close together. The clusters (pools, solar panels, fields, rooftops, cars, and more) form on their own.
The real breakthrough is that a photo and a phrase can share the same space. Training makes this possible. An underlying network analyses millions of images paired with short text descriptions to learn how to place a picture near the words that describe it. We use a model from the SigLIP2 family, an open source set of models from Google that comes in several variants.
“Once words and images live in the same space, search gets simple. Drop a phrase in, and Zero-Shot Search finds what's nearby. Nobody has to teach the system that phrase in advance. That's what makes it zero-shot: any words work.”
Wikus Brink Director (ML Engineering) at NearmapTiles: Searching the archive
Zero-Shot Search embeds imagery ahead of time, not the moment you search. It cuts imagery into small square tiles — the same ones a web map loads as you pan and zoom — embeds each tile once, and stores it, making the images instantly searchable. This happens at six zoom levels, from a wide view down to fine detail, so the same location gets embedded at several scales.
At the finest level, a 256-pixel tile covers about five meters, roughly two centimeters per pixel. The scale adds up fast: 469 million tile embeddings so far, from 107 surveys, and we’re only just starting. The systems supports up to 10’s of billions of embeddings, and we’re planning to have several billion soon.
Search three ways:
- Type a phrase. Zero-Shot Search embeds the words and returns the tiles with the closest embeddings.
- Give an example image. Zero-Shot Search embeds your image and pulls back the tiles that look most like it.
- Point to a spot on the map. Zero-Shot Search embeds the imagery there and finds other places that look similar.
Choose your search method and Zero-Shot Search will turn that query into an embedding, compare it against hundreds of millions of stored tile embeddings, and drop the closest matches onto the map in seconds. Zero-Shot Search is built to find imagery by meaning, not coordinates.
This is where zero-shot earns its name. Search for almost anything, even something that barely exists from above, with no detector trained in advance. Ask for a solar farm, or a rainbow, and Zero-Shot Search returns the closest match the imagery has to offer.
The top tile Zero-Shot Search returns for each text search, or the closest it can find for rare ones like “rainbow.”
Heatmaps
Search returns individual tiles that match your query. A heatmap answers a different question. Instead of finding specific matches, it scores an entire area at once. Give Zero-Shot Search an area and a phrase, and it shades the whole area based on how well each part matches. Warm colours mean a strong match. Cool colours mean a weak one. The result is a single image showing you both where something exists and where it clusters. Heatmaps are zero-shot, just like search, so the phrase can be anything.
Featured below is a suburban block scored for swimming pools. The same works over a much larger area, and at the finest resolution Nearmap offers.
)
Every tile in the drawn area scored against “swimming pool” and shaded from cool to warm; the imagery outside the area is dimmed.
Properties: Many views of one place
Tiles and heatmaps both answer “where does this kind of scene show up?” But often what matters to you is a specific property. A property is more than a flat square. Nearmap captures from directly above (Vertical) and from several angles (Oblique, looking in from the north, south, east, west, and the corners). Properties get captured again and again as new surveys fly.
Zero-Shot Search takes property insights to another level. It produces a clean cut-out of a property from every view, embeds each cut-out, and bundles them together. A single property carries its Vertical capture, its Oblique capture, and multiple capture dates. A search can match on any of those views, and a result comes back as a whole property, not a stray tile.
)
One property, seen from straight above (centre) and from eight Oblique directions around it. Zero-Shot Search bundles all these views of the same place together as a single property.
Oblique angles catch facade detail the Vertical view misses. Search Obliques directly, just as zero-shot as everything else. Simply name the feature you’re after, and Zero-Shot Search finds it. Here’s the top Oblique match for each search, with the matched property’s boundary drawn on:
)
Top Oblique match for each search, with the matched property's boundary outlined.
“One idea powers all of this: turn imagery and the words that describe it into embeddings that sit in the same space. Once you have that, finding a tile, shading a neighborhood, and comparing a property across years all become the same question. No need to train anything. You just ask.”
Wikus Brink Director (ML Engineering) at NearmapDiscover more from Nearmap Labs
Curious what's next in property intelligence? Meet the team, follow the journey, and join the early access community.
Explore Labs