Click VR button Slow

I could use a little help. I have created a scene with a sphere. The normals are reversed and an image is placed on it. I put the cardboard sdk in the center. I also made a world space canvas with a button to change the sphere material to another image when it is clicked. It works perfectly sort of. It is really slow to change the image usually taking 5 or 6 seconds. I remade the scene exactly the same except I unticked VR Mode Enabled under the cardboard.cs script. Like this the image...

Click VR button Slow