CSAIL says kPAM or Keypoint Affordance Manipulation is more accurate than other similar technologies. After it detects all the coordinates on an object, it determines what it can do with it. For instance, if it sees a mug with a handle, it can hang it on a hook by its handle. If it determines that it’s looking at a pair of shoes, it can place the shoes on a rack. “Understanding just a little bit more about the object — the location of a few keypoints — is enough to enable a wide range of useful manipulation tasks. And this particular representation works magically well with today’s state-of-the-art machine learning perception and planning algorithms,” the study’s senior author, MIT professor Russ Tedrake, said.

The researchers are hoping to develop the system further until machines powered by kPAM can do bigger tasks, such as unloading the dishwasher and wiping down kitchens. They’re also hoping that the system can power factory robots part of a bigger manipulation machine in the future.