Future of Filmmaking

About the project

• Capstone project with the team from Microsoft Mixed Reality at Work (MRW). The task was to design a Mixed Reality intervention for first-line workers. Our team started looking into the use case of Mixed Reality in the filmmaking industry.
• Timeline and duration: March 2019 - present (12 weeks, ongoing)
• Team members: Dolcie Dass, Javan Wang, and Surabhi Wadhwa.
• Mentors: Prof. Michael Smith, Prof. Daniela Rosner, Dan Osborn (Principal Creative Director), Ron Amador (Visual Design Director), Christian Sadak (Senior Designer), and Vibhuti Kanitkar (UX Designer).
• Deliverables: User research, competitive analysis, literature review, synthesis of data, visual design.

You can read the detailed research report we presented to Microsoft below. In this project, I talk about the process utilized to create this report. Next quarter (June - August 2019) I will be taking the learnings from this research to come up with concepts and prototypes.

Research report

Background

Technology has impacted the filmmaking industry in the past few years. From George Lucas to Robert Rodriguez, technology has been adopted to drive the industry in very different directions. Recently, there has been a huge interest from filmmakers in immersive technologies. Steven Spielberg used the VR headsets while building the VR cinematic universe of Ready Player One.

Steven Spielberg utilizing the HTC VIVE In The Filmmaking Process. More details here.

However, we stumbled across a tweet by Girish Balakrishnan, which spoke of the possible use case of Hololens in filmmaking. This kickstarted our research.

Filmmaking process

We started off by investigating the filmmaking process. Through literature review and expert interviews, we learned that it has 5 main stages.

• Development: The first stage in which the ideas for the film are created, rights to books/plays are bought, etc., and the screenplay is written.
• Pre-Production: Arrangements and preparations are made for the shoot, such as hiring cast and film crew selecting locations and constructing sets.
• Production: The raw footage and other elements for the film are recorded during the film shoot.
• Post-Production: The images, sound, and visual effects of the recorded film are edited and combined into a finished product.
• Production: The completed film is distributed, marketed, and screened in cinemas and/or released to home video.

However, our literature review and secondary research showed that tech innovations focus only in the post-production phase and improving the editing stage of films. The industry has seen a lot of advancements in the VFX process.

The immense amount of CGI work done for Avengers, Infinity Wars.

Seeing that all work has mostly been focused on post-production, we wanted to see how Mixed Reality applications could be utilized in pre-production and production phases.

Challenge

Based on early interviews with experts and secondary research, we found that filmmakers use various methods to visualize and externalize their ideas, shots, and sequences. We also identified gaps in collaboration that exists between different stakeholders in the entire filmmaking process.

Our design challenge while focusing on pre-production and production phases.

Research questions

We determined that it was important to build our intuition around both the industry as well as the technology. We wanted to identify opportunity spaces where mixed reality could be an appropriate response.

These questions would guide on what types of experts and participants we would be interested in recruiting, as well as the specific stages of filmmaking that we want to dedicate our focus on.

1. How does the director communicate envisioned ideas to different stakeholders?





2. What are some gaps in collaboration between different stakeholders in the filmmaking process?





3. Would Hololens be an appropriate design response given the context of use?

Literature review

Recently there has been some research around using Mixed Reality techniques for PreVisualization. In the pre-production stage, storyboards are been traditionally used to illustrate the director’s intention. Sometimes PreViz is also used to get stakeholders onto the same page. At the same time, MR has possible applications in designing Cinematic Lighting.

Virtual Reality Rehearsals for Acting with Visual Effects. More details here.

Expert interviews

To build off of our foundational knowledge developed in the secondary research phase, we sought out experts in a range of fields to shed light on filmmaking, mixed reality, and other new media technologies. A large number of our experts were constantly moving around due to changing filming locations. To accommodate this, over half of our interviews ended up being remote video calls.

8 experts were interviewed for 45 minutes each in a semi-structured format. Here are some of the profiles of experts I spoke with.

Me interviewing Ryan Woodward remotely.

Semi-structured interviews

The interviews were conducted on-site at the participant’s location or through a remote video call. The interview began with a set of general questions to establish the participant’s role in the filmmaking process. I then asked questions about how the participant communicates and collaborates with multiple stakeholders while working on a production. I also asked questions that focused on their process of decision-making and the factors that influence the final outcome.

Participant profiles

Diversification of participants was key for our project. We looked for triangulation across different stakeholders. We identified that it is important to speak to participants that work on the production of major motion picture films, as well as independent filmmakers such as art and documentary filmmakers.

We tried to advocate for inclusivity in our research by interviewing participants across gender, age, and ethnicities, however, all but one of our participants were men; all but two out of our 16 participants were white. We believe this to be a reflection of the existing filmmaking ecosystem.

Personal inventory

I asked and documented the tools and technologies the participants used to add value to their creative process and aid in collaboration with different stakeholders. This research method was used to understand the frequency and nature of interaction with the tools and technologies used by the participant. This method helped uncover the perceptions and values of the participant with regards to adopting new technologies.

Competitive analysis

We conducted a competitive analysis of a variety of products that currently help people envision their ideas as well as facilitate collaboration and feedback. We assessed each competitor, against the following design principles we hope to achieve with our design response:

• Flexibility: Can it be used in multiple stages of the filmmaking process?
• Customization: What is the level of customization available?
• Familiarity: What is the level of familiarity required?
• Externalization: In what ways does it help externalize creative ideas?
• Remote use: Can it be used remotely?
• Support: How does it support different roles?
• Feedback: How does it facilitate feedback?
• Sharing: How does it facilitate the sharing and consumption of information?

Products analyzed during the analysis.

Synthesis of data

To help us get a better grasp of our overwhelming amount of data, we each created our own synthesis models. These included making many process diagrams and system charts.

Process diagram for the pre-production phase of filmmaking.

We externalized all of our hard data from our interviews onto sticky notes, then placed them into different categories. Once in categories, we started looking more in-depth, finding four larger themes.

1. The production crew relies on written documents or 2D artifacts for blocking a scene in 3D, which results in gaps in communication of spatial information.

Blocking is the process of deciding where actors are going to be in the scene and how they’re going to move in relation to your camera. This also impacts the placement of camera and lighting equipment. The stakeholders involved are actors, director, and cinematographer.

Clip from Citizen Kane to explain what blocking is.

Currently, blocking is achieved through utilizing previously made storyboards, scripts, or previs. Since these artifacts do not communicate spatial information, which is vital for the positioning of actors and equipment, blocking can tend to be an inefficient process.

“It’s hard to verbally explain the geometrics, where’s everyone positioned. Just explaining the ideas to the DP takes a lot of time and effort.”
Brandon Crane ▪ Film Student, University of Washington

1. Production designers rely on physical 3D models for designing studio sets as photos and concept art do not provide spatial context.

Production designers find it hard to assess the cost and the amount of work that is required for furnishing and dressing the set when they don’t have access to it. They need to plan for renting, buying or creating props. They need to see how many sets they can fit in one location.

A short clip of the production design team which worked on 'Thor: Ragnarok'. Featured are Production Designer Dan Hennah, Conceptual Designer Director Ra Vincent, Head of Scenic Chris Williams.

"The analog style of having a physical model that everyone can look at, around the table is immensely helpful.”
Scott Baker ▪ Set designer, Black Panther

1. Set designers and decorators require access to filming locations in order to transform the set before the shoot. Reserving the location for long periods of time increases production costs.

The set decorator is responsible for furnishing on-location sets for film and television. This can, however, be a time-consuming process. They run on a tight budget based on the shooting schedule, rental periods, and purchases. And these iterations and experimentations on dressing the set tend to increase the budget.

1. Lower fidelity of computer-generated characters and objects allows for faster experimentation of its location, scale, and movement.

While exploring the position, scale, and movement of characters and cameras, low-fidelity representations prove to be adequate and effective. The lower fidelity of these artifacts helps in faster collaboration while also keeping the focus of conversations on only the intended aspects.

Low-fidelity visualization for “Meg Attacks Dock” sequence of The Meg

1. During pre-production, directors face difficulty experimenting with the movement, scale, and position of characters using current 2D pre-visualization tools.

Previsualization is the visualizing of complex scenes in a movie usually done during the pre-production phase. However, previs today is done using tools like Maya, Unity, and UnReal which are still 2D environments mapping 3D content. It doesn’t accurately represent the artistic spatial vision. Moreover, these tools have a steep learning curve which makes it often impossible for directors to play around with.

"Communicating to previs animators what kind of shots they [director] want to see can be a frustrating and a slow process. They wish for the control to let them create the shots themselves in an intuitive way without having to know how to operate 3D software.”
Marijn Eken ▪ Digital compositor, Captain America: Civil War

1. While shooting on set, actors and camera crew cannot envision computer-generated characters and objects.

While interviewing cameramen who work on sets involving computer-generated (CG) objects and characters, we discerned a repeating remark about the ambiguity in visualizing the position and scale of such objects. Our secondary research also helped us realize that this was an issue faced by actors as well.

Maisie Williams described the struggle she faced while acting with objects, not in the scene.

"They’re like ‘look out to the castle’ and you just think ‘well, how far away is it? Is it right here?’ and although you can ask all of those questions it never looks right.”
Maisie Williams ▪ Actor, Game of Thrones

1. Visual incoherence emerges from pre to post-production as a result of misaligned visions.

The filmmaking process from pre-production to the production phase requires many stakeholders. And every production team has their own artistic vision, their own style, and their process. These differences create a misalignment in the stylistic continuity.

1. Creative freedom during filmmaking is oftentimes constrained by time and budget.

In today’s world, the films produced tend to have a stronger push towards commercial success. As the production cost of movies has progressed exponentially in the past decade, the complete process of filmmaking leaves very little room for error.

1. Big productions are adopting and innovating new media technology. However, small productions have limited access to these technologies due to the lack of resources, knowledge, and best practices.

There is a noticeable inequality across the scale of production studios and their adoption of new media technologies. Access to time, knowledge, resources, and room for experimentation with this tech is directly correlated to the size of the budget. The use of this tech in filmmaking is not a standardized process across the industry.

You can read the detailed research report we presented to Microsoft below.

Research report

Next steps

Next quarter (June - August) we will be taking these insights, themes, and start working within different frameworks to come up with concepts. We will prototype four to five ideas that we feel encapsulate the essence of what we are trying to accomplish with our project to prototype. While working closely with the team from MRW, we will try to create viable design interventions which help to externalize creative ideas and collaboration.