True. Constructing a perspective with one-, two- or three-points works to some extent, but only on very narrow field of vision. Human field of vision, on the other hand, is much wider. Constructed perspectives also don't take the eye/head movement in account, so they are kinda limited.
I myself prefer trying to draw things the way I would see them.
There are probably many books, though I don't know/remember any. However, a good place to study perspective is your own room. Use your peripheral vision to look around yourself and see how things look anything but straight, although you know they are straight. For example, if the walls were truly straight in your vision, then the wall behind you would be larger in size than the wall in front of you. But because that's not of course true, the wall won't get larger, everything around your vision needs to distort and bend so that the wall behind you would match the size of the wall in front of you.
Next time you walk on floor with tiling, notice how spherical it looks if you look at your feet and use your peripheral vision. Also notice how it seems to constantly change as you walk forward.
I learned perspective because I'm working with CAD models and I have made 3D renderings. While I learned to see how the objects behave as I orbit the view around them, I also learned the limits of the mathematical perspectives. If you want to try easy 3D modelling, try Trimble Sketchup (used to be Google Sketchup). After a basic how-to tutorial, it's very straightforward and easy to make basic models and see them in perspective.