Researchers at Brown University built a robot that finds objects by combining spoken instructions with pointing gestures. The system is based on experiments showing how domestic dogs interpret human pointing, and it located the correct object in 89% of simulated trials.
The Brown University Study
- Title: “LEGS-POMDP: Language and Gesture-Guided Object Search in Partially Observable Environments”.
- Authors: Ivy Xiao He, Stefanie Tellex and Jason Xinyu Liu of Brown University.
- Preprint: posted to arXiv on March 5, 2026.
- Presentation: the ACM/IEEE International Conference on Human-Robot Interaction (HRI) in Edinburgh, Scotland, on March 17, 2026.
- Robot: a Boston Dynamics Spot quadruped with an arm.
- Funding: the US National Science Foundation and the Office of Naval Research.
How the Robot Searches
- POMDP: the system uses a “partially observable Markov decision process”, a planning framework for decisions under uncertainty.
- Two unknowns: the robot estimates both which object the person means and where that object is.
- Language: a vision-language model interprets spoken requests such as “bring me the red cup”.
- Gesture: pointing is modelled as a “cone of probability” along the line from the eye through the elbow to the wrist.
- Active viewing: when uncertain, the robot moves to get a better view before choosing an object.
Lead author Ivy He stated: “Humans use eye gaze to align with what they’re pointing to.”
Simulation Results
| Input | Success rate |
| Language and gesture combined | 89% |
| Language only | 71% |
| Gesture only | 61.8% |
| No input | 48.2% |
- Simulations used grid worlds of 5×5, 10×10 and 20×20 cells.
- Variants included “ambiguous” layouts with several similar objects.
- In the real-world demonstration, Spot chose among five objects, three of which were identical red cups.
What Dogs Contributed
The robot builds on a 2024 study, “Find it like a dog: Using Gesture to Improve Object Search”, presented at the Annual Meeting of the Cognitive Science Society:
- Participants: 6 dogs and 6 dog guardians.
- Trials: 72 dog-human trials and 36 human-human trials.
- Method: five ways of drawing a “pointing line” were compared, with dogs serving as a non-human baseline.
Accuracy of each pointing model:
- eye-to-wrist: 96.9%;
- nose-to-wrist: 95.7%;
- full arm: 94.0%;
- forearm: 92.5%;
- gaze alone: 51.8%.
The eye-to-wrist line became the basis of the LEGS-POMDP gesture model.
Dogs and Pointing: The Science
- Puppies vs wolves: A 2021 Duke University study in Current Biology tested 44 dog puppies and 37 wolf puppies aged 5 to 18 weeks. 17 of 31 dog puppies reliably chose the bowl a person pointed at; none of 26 wolf puppies performed better than chance.
- Early ability: dog puppies succeeded from 8 weeks of age.
- Genetics: A 2021 study from the Arizona Canine Cognition Center tested 375 eight-week-old service-dog puppies and found that genetics explained more than 40% of the variation in following human pointing.
The Cost of Lost Items
A survey by Pixie Technology of more than 1,700 Americans, released on May 2, 2017, measured the problem a fetching robot addresses:
- People spend 2.5 days per year searching for lost items.
- US households spend $2.7 billion per year replacing lost items.
- 45% lose the TV remote at least weekly.
- 33% lose their phone at least weekly.
- 28% lose car or house keys at least weekly.
Earlier Object-Finding Robots
In 2023, a University of Waterloo team led by Dr. Ali Ayub presented a Fetch mobile manipulator with “artificial memory” at HRI 2023. The robot logged where objects were last seen and was designed to help people with dementia.
Trust in Robot Helpers
Gesture-aware robots raise questions about how much people rely on automated systems. Experiments from the Wharton School show how much people trust the bot when its answers are wrong.
Short Domains for Robotics Projects
The .co extension is the country-code top-level domain of Colombia and opened to global registration in 2010. Its short, company-style suffix makes it common among startups; robotics teams that buy .co domain names use them for demos and product pre-launch pages.