I think the thing people want to say in epistemology these days is that perceptual knowledge is non-inferential. This makes a good deal of phenomenological sense. Insofar as there's something distinctive that it feels like when we infer things from other things, it isn't going on in ordinary cases of perception. I don't look in the general direction of my desk, have a bunch of sense experiences, puzzle out what's going on with them, suddenly go 'Aha!' and arrive at the conclusion that that's my desk in front of me. Nor do I need to in order to be justified in believing that my desk is there.
There are lots of things for which we have this sort of noninferential phenomenology. It's not just the way things go with the size, color, and orientation of surfaces -- I can look at a sequence of letters and figure out what word it is in basically the same way. People extend this to all sorts of other stuff. Apparently experienced chess players can see who's winning a chess game in this sort of way. People talk about moral perception working this way too.
It's examples like the chess thing that make me less impressed with this point, however. That's something that probably started out with inferential phenomenology earlier in the players' chess careers, as people considered the positions and the values of the pieces and arrived at a decision about who was winning. They've just done it so many times that it's become second nature to them and there's no inferential phenomenology. Same with words -- you start out doing Hooked On Phonics or something and puzzling them out. Then after a while it all works automatically.
If I had to bet, I'd bet that perception of physical objects is the same way too. The puzzling-out part just gave way to automatic knowledge long ago when we were little babies, so we don't remember it. If people want to say that perceptual knowledge is noninferential for adults but inferential for really little kids, that's cool. But sometimes it sounds like they really want to make claims about this sort of knowledge as a general category, regardless of the age of the perceiver, and then I'd want to say that perceptual knowledge just involves an inference we've gotten real good at so we don't have to think about it.