Scene-Graphs

TL;DR for operators A robot can perceive more of its environment than its planner should necessarily receive. In the experiments summarized here, adding task-irrelevant objects to structured scene representations increases the burden on classical planners and can leave harder problems unsolved. The proposed response is not a new end-to-end planner. It is a learned relevance layer that decides which objects and relations should survive into the planning problem. ...

Furniture is not democratic. In a real room, the bed, sofa, dining table, and cabinet do not play the same role as the pillow, lamp, monitor, mug, or miniature ornament. Large furniture defines the room’s usable structure. Smaller objects depend on that structure. A chair can stand around a dining table; a book sits on a shelf; a lamp belongs near a bed or desk. The room has a hierarchy before the model begins to generate anything. ...

Sound is messy. Video is messy. Put them together in a real business environment—a factory floor, a training room, a retail aisle, a vehicle cabin—and the usual fantasy of clean perception quietly dies in a corner. A camera can see a person holding a tool. A microphone can hear a machine alarm. But the useful question is rarely “what objects exist?” or “what sound is present?” It is more awkward: which thing made the sound first? Where is the loudest source? Was the visible action actually producing the audio event, or merely happening near it? ...

A warehouse robot does not need poetry. It needs to know whether the box is behind the pallet, whether the cup is closer than the plate, and whether the object it is about to grab is actually reachable rather than merely visible. Small details. Very irritating when ignored. This is where many multimodal models still become strangely philosophical. They can describe an image fluently, infer intent, and produce a confident answer. Then they miss that one object is in front of another. Apparently, “seeing” and understanding space are not the same occupation. ...