PlantAI
Photograph a sick houseplant, get a species ID, a diagnosis, and what to do about it
- 2
- chained vision models
- 5
- external APIs orchestrated
- 4
- surfaces sharing one core
What it does
Photograph a sick houseplant and PlantAI identifies the species, works out what is wrong with it, and writes a treatment plan. A second pipeline finds nearby nurseries and what they actually stock, so the diagnosis leads somewhere. It is a React Native app on Expo; the nursery pipeline runs standalone but is not wired into the app yet.
How it works
Get the photo off the device
React Native's WinterCG FormData rejects the blob shapes most upload examples assume. Images stream as expo-file-system File blobs instead.
Identify the species
PlantNet handles identification, since a purpose-built model beats a general one here. A 404 means the photo is not a plant, which is caught explicitly and surfaced as that rather than as a generic failure.
Diagnose from the species
The identified species becomes context for a second call to Claude or GPT, which writes the diagnosis and the treatment steps. Splitting the chain means the diagnosis model knows what it is looking at.
Find nearby nurseries
Google Places text search produces candidate businesses near the user.
Fingerprint each site
Each nursery site is classified as Shopify, WooCommerce, Wix or unknown, because each needs different handling. Results cache to disk so a site is only fingerprinted once.
Scrape, extract, then verify
Firecrawl first with Tavily as a fallback, narrowed to a price-focused excerpt, then a model pulls structured JSON and a second critic pass checks it before it is trusted.
One core, four consumers
A dependency-injection seam lets the mobile app, an offline dataset builder, a dashboard and a small HTTP server all sit on the same core.
Tech choices
The decisions that shaped it, and why they went that way.
- Two specialised models beat one general one
- Asking a single model to identify the species and diagnose the disease at once made both answers worse. Chaining them gives the second model a fact to work from.
- The critic pass exists because extraction hallucinated
- Single-pass JSON extraction invented prices often enough that a verification step was not optional.
- The DI seam is what makes it testable
- Because the core takes its dependencies rather than importing them, it can be exercised without network access. This is the one project here with a real unit test suite.
- Failure paths matter more than expected
- Users photograph the wrong thing constantly, so the not-a-plant case is a designed path rather than an error state.
Built with
- Mobile
- React Native 0.85Expo SDK 56React NavigationReanimatedreact-native-maps
- Backend
- Node.jsTypeScript
- AI & data
- PlantNetAnthropic ClaudeOpenAIFirecrawlTavilyGoogle Places
How the repo is laid out
Roughly the order you'd read it in: entry point first, then the parts doing the work.
App.tsxExpo entry: navigation stack, fonts, themesrc/Screens, camera diagnosis, nursery service clientsscraper/Places discovery, then scrape and extract stockserver/Node API serving the nearby nursery searchdashboard/Local dashboard for testing scraper queriesscripts/Offline job rebuilding the nursery datasetdesign-system/Tokens and layout rules for every screen
Where it falls short
The app currently ships a static nurseries.json, so the distances and photos you see in it are placeholder data. The live pipeline works standalone but takes 30 to 60 seconds, which is too slow to sit in front of a user without a rethink. Wiring it in properly is the next piece of work.