AR and VR did not only introduce a new class of output devices, but with tracked controllers and hands also the requirement for a new set of user interactions. This talk investigates solutions in existing implementations and points out how the classical UX model with keyboard and mouse translates to these new devices. The technical aspect of these requirements will also be highlighted. The audience will get an overview of the status of Open Source in VR and the opportunities for GNOME. We will propose an implementation that integrates VR in the GNOME desktop, featuring 3D window management and desktop input synthesis.