Since the introduction of immersive technologies to the mainstream market, beginning in earnest in 2016 with the release of HTC and Valve’s Vive room-scale virtual reality (VR) system, immersive technologies have offered incredible creative potential for new artistic and narrative works.
Immersive technologies now cover a range of technologies, such as room-scale, desktop, and inside-out tracking VR; phone– and headset-based augmented reality (AR); 360-degree video; and mixed reality. Additionally, within each of these technologies lies the capacity for both interactive and passive content.
Despite its cross-disciplinary applications, immersive technologies are not yet widely integrated into the practice of working artists and content producers. Barriers to adoption include the relatively high cost of production equipment, new and constantly evolving technology streams and a lack of appropriate training opportunities geared toward the arts.
Ontario-based trade association Interactive Ontario (IO) recognized this challenge. Together with its national collaborator, the Cultural Human Resources Council (CHRC), funding was secured through the Canada Council for the Arts to deliver a 5-day program for 20 participants in 2019. The program was delivered in Toronto by Dames Making Games (DMG) and Gamma Space Collaborative Studio, nonprofit artist-run centres based out of the Toronto Media Arts Centre (TMAC). It was made possible with the support of the Canadian Media Producers Association (CMPA) and Actra Fraternal Benefit Society (AFBS).
Since 2012, DMG has provided free immersive and interactive skills training for marginalized artists using a peer-led facilitative approach: Short workshops with a focus on hands-on practice in a variety of formats, direct professional and community support, and participant-determined balance of group and solo work. These intensives have seeded hundreds of video games, wearables, linear narrative works and new tools and platforms for social action.
This approach helps artists and producers across a range of disciplines learn new skills rapidly by applying their individual creative vision and existing artistic practice. This provides a practical foundation for taking advantage of the expanded opportunities for storytelling offered by immersive and interactive technology.
Two key factors in the success of a program to address the low uptake of immersive technologies by artists are: Selecting the right immersive technology and creating an accessible training framework delivered in a conducive learning environment. These issues are discussed in detail here, and provide a blueprint for future endeavours. The program relies on competent instruction and expert mentors to support participants through the learning stages from initial exposure to expertise. In light of the COVID-19 crisis, which is impacting all spheres of activity, consideration for a remote delivery of this learning program has been included.
IO and CHRC’s outreach revealed strong interest from filmmakers interested in exploring immersive technologies. Eighteen out of 20 participants in the program were involved in the film sector. All of the participants had experience with digital media production; two had worked with immersive technologies in a direct capacity, and one had ongoing involvement. The group represented a diverse mix of skills, resources, networks and experiences.
The program emphasized filling gaps in participants’ skill sets while allowing them to ground their projects’ content and expressive methods in their artistic practice. While a broad survey of technologies was presented during the program, the technologies selected by the coordinators for hands-on instruction needed to support:
Rapid iteration and testing. Waiting for projects to compile and load onto a device slows the learning process and can be distracting. For example, mobile platforms such as AR on Android and iOS, require multiple time-consuming steps to test changes.
Content production with existing skill sets. VR requires working with pre-made assets or learning how to create 3D models from scratch, a technical barrier.
Collaboration. The tools selected needed to be suitable for small groups to work with a single code base and library of digital assets.
Volumetric video meets all of the above criteria. Additionally, unique expertise exists within the DMG community, including artists and technical mentors with years of experience working with the technology.
Volumetric video involves the capture of a performance or location in three-dimensional space. The data captured contains spatial information, as well as video, and can be viewed on 3D displays, VR headsets and even traditional flat screens – albeit without the conveyance of depth information.
Some advantages of volumetric video:
Its potential to balance DMG’s facilitation priorities and IO and CHRC’s goal of supporting artistic creativity with immersive technologies made volumetric video the ideal technology choice.
Once the technology was selected, DMG customized the delivery plan and curriculum for IO, CHRC and participants.
The intensive was held over five days, from November 25 to 29, 2019, in DMG’s classroom and studio space. Each workshop day lasted eight hours, including break times. Each day started with instruction and presentations in the morning, followed by hands-on work and in-studio production in the afternoons.
After the first day of introduction, participants organized into groups of four after presenting narrative and interaction design ideas to each other. This supported DMG’s approach by making space for self-directed participation and peer learning, as individuals sought complementary skill sets and experiences in their groups.
Access to education varies greatly among intersections of marginalized groups. DMG has an explicit mandate to ensure diverse participation in all programs it delivers. An effective learning environment begins with a space that explicitly prioritizes the wellbeing and safety of participants. Participants with less experience, fewer resources and who have historically faced barriers to tech spaces based on their gender identity, race, orientation or cultural identity need to be provided with equitable access to instruction, materials, equipment and feedback in order to fully engage with and benefit from a learning environment.
IO and CHRC reached out to digital media producers through its industry network. DMG centers LGBTQ+ and gender-marginalized individuals, so its outreach was focused on those identity groups.
Providing equitable access means intentionally removing barriers marginalized groups frequently face: Child care, food security and transportation.
To reduce barriers to access for people experiencing socioeconomic challenges, transportation, travel and accommodation subsidies were offered to all participants. Meals and snacks were provided to all participants, and all dietary requirements were accommodated. Applicants were invited to request any other accommodations they required. To build confidence that this environment would be upheld throughout the program, this goal was made explicit in materials and presentations, and trusted instructors who support these values in their work and interactions were selected. Additionally, participants, facilitators, mentors, and speakers were regularly reminded of DMG’s code of conduct, which was posted in the physical space and made available online.
Situating the program within a creative hub allowed participants to engage with a broader community of artists with overlapping interests and specialties. Visitors, people attending workshops with other arts organizations, and professional members working in adjacent spaces provided ample opportunities for organic community participation, conversation and testing. Unhindered access to all spaces and relevant equipment within the hub was provided to provide participants with creative freedom and a sense of ownership in the learning experience.
A major barrier to participation in technology-intensive programs is the availability of specialty equipment.
TMAC has leveraged its members’ networks to develop relationships with equipment manufacturers, hardware and platform holders, and software makers to secure free equipment and software licenses. By accessing these relationships, a significant portion of the program budget could be allocated to human resources and costs associated with supporting participants directly.
For example, DMG was able to quickly secure two volumetric video capture workstations from AMD, two additional new VR headsets from Oculus, and Xbox Kinects by OCADU’s game:play lab.
Providing VR-capable workstations and volumetric video capture stations may be challenging for smaller groups as this hardware is not readily available to rent – or affordable to buy – in most areas of Canada.
The program was led by two main facilitators:
Lander and Faber have extensive knowledge in key areas of the curriculum, including:
DMG’s membership is a key source of experienced mentors. Through semi-formalized training and year-round programs that cultivate a culture of mutual mentorship, DMG supports a pool of several hundred highly skilled and engaged practitioners.
Six mentors supported the program in-person and via online communication tools – during the workshops as well as during studio time:
IO provided industry professionals from its network, who gave talks regarding their projects, success in the industry, as well as adjacent applications for immersive technologies. They complemented the facilitators’ knowledge base and helped to provide additional industry context.
Getting participants out of their comfort zones can be a challenge in any instructional program. For artists, learning how to use new technology that has been designed for industrial and commercial users and applications may seem especially difficult. However, artists are well-suited to experimentation and tinkering, and with the right framework, the possibilities of immersive technologies can lead to creative breakthroughs not possible with traditional media.
Facilitators with direct experience creating immersive and interactive work, an abundance of examples that participants can experience firsthand, and an environment that encourages exploration without fear will support the success of a program that seeks to ease participants through the learning process.
For CHRC-IO’s Arts Futures Program, that meant ensuring the following:
Facilitators also demonstrated troubleshooting techniques and methodical research to empower participants to educate themselves when direct support is not available:
A Slack channel was created within the existing DMG Slack team for all participants, facilitators and mentors to chat with each other and share resources while working. Participants were invited to the channel prior to the first day of the program to introduce themselves.
A separate private channel for mentors and facilitators was used for unobtrusive coordination during the program, timely administrative conversations and quick sharing of links and resources behind the scenes.
Basecamp was used by the facilitators to coordinate equipment prep, scheduling and other day-to-day requirements.
Instructors shared Google Slides presentations to avoid the need to swap computers connected to the projector, and reduce turnover and downtime. All presenters emailed or used Slack to send their materials to the coordinator a day before the program started to avoid distracting delays and issues with coordinating slides.
Games and immersive experiences were downloaded from Steam and the Oculus Store ahead of the first day of the program.
Desks were arranged in five groups of four during studio and production activities. During presentations, they were moved into rows or a U-shape to ensure all participants could see the presenter and screen.
Because the presentations included the display of code and a development environment, a bright projector was necessary for legibility. Working code was provided via Slack and by using a collaborative environment (e.g., Glitch or VS Code Live Share) to further support the visibility of content that may have been hard to follow on a distant screen.
Computers were pre-installed with four free software applications:
Unity Hub Unity Hub was installed on each machine with the latest version Unity. Hub allows older versions of Unity to be installed concurrently, which can be useful for accessing older or beta versions required by certain tutorials or cutting-edge features. While our workshop focused on the use of the Oculus Rift headset, it is recommended to install the latest Android tools when installing Unity for compatibility with Oculus Quest headsets.
Oculus SDK for Unity The Oculus Integration Package for Unity is downloadable from the Oculus site, and is also available as a plugin in the Unity Asset Store. Having this pre-installed for participants allows a quick look at the results of their work in VR in minutes.
Depthkit DepthKit is used for manipulating depth levels and video masking post-capture. Though primarily used in the studio setups, additional settings can be tweaked prior to import into Unity for fine-tuned results.
Custom Unity Project Starter Template An example scene was prepared using the Depthkit plugin, sample volumetric video and custom scripts to provide participants with an accessible starting point for making their projects. Participants were able to drop their newly captured media into this scene and instantly have it interactive in VR as well as be provided some simple introductions to scripting. The project can be downloaded from the GitHub link above and opened in Unity as-is.
Though it wasn’t necessary for participants to bring their own laptops, many did, which allowed them to write, edit audio and video, use Adobe Creative Cloud and Blender for 3D models if they were interested.
Instruction on the use of these tools was beyond the scope of the intensive, but many of the participants were comfortable in various applications that aided in the creation of their projects.
Encouraging participants to bring in the tools they were comfortable with helped them focus on the unfamiliar aspects of production.
Two different studios were created with three capture setups: Gamma Space’s windowless Gamma Gallery and TMAC’s Expansion Space multipurpose studio, with floor-to-ceiling windows on the east wall for natural light.
Three setups allowed instruction and creation to happen simultaneously as needed, while two different lighting options provided opportunities for both natural and studio lighting as required by the projects.
Morning: XR Technology Overview including examples from industry members
In the DMG classroom, facilitators kicked off the program by presenting an overview of current technologies, including VR, AR, and mixed reality. This included associated costs with production, audience size for each platform and accessibility considerations. Commercial and experimental examples of XR work were presented by:
Max Lander + Pyramid Attack - Night of the Living Dead VR
Sinking Ship - Dino Dana
Afternoon: Introduction to Volumetric Video
Participants were encouraged to try various demos provided by industry speakers, as well as games and experiences selected by the facilitators. While most of these demos are not available on a public platform, examples of volumetric video made with Depthkit can be found online:
During the afternoon session in the Gamma Gallery, Dr. Cindy Poremba from OCADU gave a hands-on demonstration of the principles of volumetric video capture and showcased works in VR that used the technology.
The afternoon closed with a demonstration of how volumetric video clips can easily be imported into Unity with the Depthkit plugin. Depthkit provides comprehensive documentation for setting up and capturing volumetric video, as well as importing that video into Unity using the Depthkit Unity Plugin: What is Volumetric Filmmaking?
Morning: Narrative design workshop
In the DMG classroom, participants were provided examples and learned techniques for developing branching narratives and spatial storytelling by artist Kylie Caraway. Key points that were covered included:
Participants were given an exercise to design a narrative and to plot that narrative out in three-dimensional interactive space called Storyboarding in Space. Storyboarding in Space is a tool for planning VR narratives, designed by Caraway, where artists take their main narrative elements and map them out in a 360 degree space relative to a central camera using post it notes. Each narrative element (sounds, objects of interest, characters) is given their own sticky notes (colour coordinated to other elements of the same type) and placed around the central camera. This map then provides the foundations for laying out a 3 dimensional scene in Unity.
Storyboarding in Space familiarizes participants with the idea that stories in VR happen spatially more than linearly. It was a particularly helpful framing as the participants were already familiar with many other storyboarding techniques. Participants then shared their ideas and formed groups around the ideas that resonated most with them. Four groups of five were formed.
Afternoon: Capture sessions
After lunch, the groups developed their ideas and developed a shot list. Those shot lists were then used as a guide in the studios for volumetric video capture for the rest of the afternoon.
Morning: Spatial audio design and implementation
Robby Duguay and Matt Miller from Level Curve, a games and VR audio production studio, provided details about the workflow of putting spatial audio into Unity, along with an overview of binaural audio recording and its properties.
Afternoon: Introduction to Unity
In the afternoon, Lander gave an overview of Unity that included basic concepts (e.g., transform, colliders, Rigidbodies) and navigating the user interface. Next, participants were shown how the Oculus VR plugin makes interacting with basic Unity concepts greatly simplified in VR through the use of the included elements such as grab and gaze.
OVRCameraRigand drag into the scene.
Steps for correctly setting up the Avatar SDK: Avatar Developer Guide - Rift For the purposes of this program, the instructor’s developer ID was used so that participants could access those features.
Finally, participants were introduced to scripting and showed how the scripts he prepared could be used as building blocks for more sophisticated interactions. These three pieces–basic Unity concepts, Oculus Integration for VR and intro to scripting–provided the participants with enough foundational knowledge to be able to navigate and manipulate the sample Unity project provided at the beginning of the program.
pointerTrigger.cs- When attached to the right hand, this script uses the trigger button to detect a playable Depthkit clip and plays it.
cycleClips.cs- When attached to a Depthkit clip gameobject, this script allows us to load in multiple different video clips that can be cycled through by pressing A on the Oculus Touch controller.
gazeToPlay.cs- When attached to the center-eye camera game object, this script functions similar to the pointerTrigger in that it uses raycasting to detect a playable Depthkit Clip and plays it when the player looks at it.
rotateToPlayer.cs- This script must be placed on the uppermost parent game object of a Depthkit clip and will keep the clip facing the player as they move through the environment.
pauseAtStart.cs- This script must be placed on each Depthkit clip in the scene that will be played with any of the above interaction scripts. It loads the correct clip thumbnail so the object displays correctly in the scene and then pauses it, in preparation for being played via the pointer or gaze triggers.
All day: Production time
Day 4 was entirely dedicated to production. Mentors were on hand to answer questions and provide technical expertise.
Facilitators helped participants develop a process for framing their narrative against a single point of polished interaction, helping them to develop their themes and intent through the completion of an action by the user.
The morning of the final day was an opportunity to explore distribution of immersive technologies. Arthur Yeung from Albedo Informatics discussed AR content distribution. Faber and David Plant from Trinity Square Video discussed VR distribution, as well as corporate and arts funding opportunities.
Afternoon: Final production time, wrap-up and presentations
During the afternoon, participants were given the opportunity to wrap up their projects with mentor support.
At the end of the afternoon, participants presented their work to each group and members of the public visiting the building. Guest artist Karen Vanderborght demonstrated their Grey Matter AR project.
In order to increase adoption of immersive technology by artists, educational initiatives – especially informal and community-based programs – must address two key challenges:
Narrowing the scope of curriculum to the most appropriate tools and platforms for the participant cohort. When selecting technology, it’s critical to consider the resources available to the program early on, because teaching immersive technology is inherently experiential.
Providing an equitable training framework and learning environment that addresses barriers to access. Explicitly acknowledging that the spaces and equipment that make experimenting with, creating for and pushing the boundaries of immersive tech are not readily available to all artists is necessary if we hope for more creative applications and approaches to VR, AR, and mixed reality. Consider:
With a considered and focused approach to designing curricula, seeking practical support and taking steps to ensure barriers to participation are reduced as much as possible, community-based informal learning initiatives can significantly contribute to the number of Canadian artists and content producers who utilize immersive technology in their work.
At the time of this document’s writing, the COVID-19 crisis has restricted in-person learning. Resource sharing is also challenging, but advancements in technology have allowed for peer-based programs like this to continue, though there are additional logistical and material considerations.
Remote group learning facilitation. Rapid and widespread adoption of virtual meeting and screen sharing tools – such as Zoom, Google Hangouts and Slack – has reduced barriers to access to expert technical, business and creative support in group and one-on-one format. The ubiquitousness of these tools has increased literacy around remote mentorship and informal learning techniques.
Broader access to industry speakers. Because the “new normal” is remote working, industry experts may be more likely to agree to participate from different locations around the world. This has the added benefit of reducing program travel budgets.
Streamlined VR equipment. The Oculus Quest inside-out, all-in-one headset is better suited to rapid development now as it can connect via highspeed USB cable to a VR capable computer. This allows for rapid iteration when connected to a game engine without the need to make a build and deploy it to the headset after every change.
VR Kits. A VR-capable laptop, Oculus Quest, Kinect (or other DepthKit compatible camera) can be assembled into an easily shippable hardcase that could be distributed to participants anywhere for under $2,000 CAD.
While it is unknown how long the pandemic will affect learning environments, the infrastructure and resources for delivering this program can be adapted to a fully remote format. If accessibility and inclusion of a variety of cultural experiences, geographies and socioeconomic situations is desired, it is worth considering offering hybrid in-person/remote programs even once it is possible to reconvene in person.
However, it’s important to recognize other barriers may emerge – such as access to high-speed internet and unlimited bandwidth, and physical constraints on at-home work environments.
Jennie Robinson Faber