Jovo,
Over time, you'll find that when in Photofocus ... that distractions (eyes wandering ...) when Photoflipping (rapidly fanning of pages) will not be an issue as you've trained your eyes to automatically stay in the Photofocus state.
At this point I can maintain the Photofocus state at will and ever pause glance around if need be and get back into focus within seconds.
You too will experience this but give yourself time to develop.
As for is it good or bad for your eyes to wander ... it all depends upon what your eyes are wandering to! 
