VR Camera offset, inaccurate collision position

My camera is offset from the rigidbody controller when moving around the real life room. This causes problems because my camera does not handle collisions or have gravity applied to it. How can I make my rigidbody stay where my camera is so that the position for the collision is accurate and not offset? (Oculus)

Thanks in advance