Just start believing the conspiracy theories. Looking or not looking just changes the lighting system from ambient to raytracing, simple. Why spend so many resources rendering what no player is there to observe? Low level simulation on unloaded chunks.