Annotation Tuesday! Jenna Pirog on Virtual Reality in “The Displaced”

A good story transports its audience to the scene of the action. The classics of narrative nonfiction are memorable because they find the words that put the reader in the author’s shoes, witnessing the key moments as they unfold—think of the poolroom scene in Gay Talese’s “Frank Sinatra Has a Cold.” Similarly, one of the claims behind today’s surge in story-driven radio and podcasting is that audio forces listeners to paint pictures in their heads, making it the “most visual medium,” to quote Ira Glass.

But for fully immersive storytelling, there’s a new tool in the arsenal, one that may force writers and audio producers to work harder if they hope to keep up. It’s virtual reality filmmaking. [See Virtual Reality Lets the Audience Step into the Story.] VR’s arrival as a powerful documentary medium was underscored late last year by the November 5 release of “The Displaced.” The New York Times Magazine’s 11-minute VR film chronicles the global refugee crisis from the point of view of three displaced children and their families in Lebanon, South Sudan, and Ukraine.

This isn’t virtual reality in the original sense, i.e., the computer-generated, 3D, interactive environments first envisioned in the 1980s by sci-fi authors like Vernor Vinge, William Gibson, and Neal Stephenson. Rather, it’s full-motion filmmaking using multiple cameras, with the images digitally stitched into a 360-degree sphere in post-production. Audiences experience a VR film using either a Web-based viewer that lets them pan around the scene manually on a computer screen, or a more portable device such as a smartphone or a head-mounted display like the Oculus Rift; such gadgets are motion-sensitive and can pan and tilt the image automatically in response to the viewer’s head movements.

As a vehicle for “The Displaced,” the Times built a new VR app for iOS and Android phones and worked with Google to distribute hundreds of thousands of free Cardboard viewers, which are designed to turn smartphones into miniature VR headsets. Since November, the Times has released several additional VR movies that show off the medium’s capabilities, including the short fantasy “Take Flight,” a news report on the vigils following the terrorist attacks in Paris, and a look at the making of the magazine’s “Walking New York” cover.

Times readers have lapped it up, turning in their Twitter reviews to adjectives like “overwhelming,” “awesome,” and “mind-blowing.” What startles many VR newbies the most is the ability to steer the camera’s point of view­, perhaps to follow some new sound or action on the periphery. VR asks the viewer to take over the traditional role of the videographer or cinematographer, which means each viewing can be different from the last.

The reactions have been encouraging, says Jenna Pirog, the New York Times contributing editor who produces its VR films. With “The Displaced,” she says, “We really didn’t know what the reaction would be. There’s so much on-boarding associated with this film. You have to find your Cardboard viewer in your Sunday paper, and download the app, and download the film, and trust us and put your phone in the Cardboard viewer, and get your headphones on. But we were just blown away by the response.”

Downloads of the Times VR application on the weekend of the film’s debut broke all records for Times apps, and many viewers commented that they understood the refugee crisis differently after viewing the film, Pirog says. “A lot of parents, especially, drew parallels between what was happening to these kids, and their own. You could see how bright they are, and how limited their opportunities are going to be.”

New Territory

Shooting a VR documentary is such a new craft that the directors and editors of “The Displaced” had to learn what to do, and what not to do, as they went along. “You kind of have to throw out everything you know about traditional transitions and narrative flow,” Pirog says. “You can never determine where your viewer is looking, so you have to create a 360-degree environment. You can use audio to beckon your viewer’s attention toward certain things. But again, your viewer has autonomy. You have to just let go of your subjective view, and think more about, What environment can I place them in that will engage them?”

VR filmmakers also have to be careful not to overwhelm viewers with action or motion. The Times itself has covered efforts by Oculus and other hardware makers to head off the eyestrain and nausea experienced by many VR users, this reviewer included. The “nightmare scenario” for VR companies, Oculus chief technology officer John Carmack told the paper, is that “people like the demo, they take it home, and they start throwing up.” Pirog thinks the motion-sickness issue will soon abate, as hardware makers come up with ways to make the wraparound image track viewers’ head motions more quickly.

“Moving a VR camera can create nausea in the viewer’s mind, because they are seeing movement, but their brain can’t sense the movement,” she explains. “It’s the same way you’d get seasick on a boat. I think there are certain technology advances that are going to happen very, very quickly that will help with that. For instance, when you turn your head in VR, if there is a lag in the movement—if the film is even a millisecond behind your head movement—that will create motion sickness. That will continue to improve, but one thing you can do is to move a little more slowly when looking around than you would if you were standing around somewhere watching action.”

Technical kinks aside, the Times’ virtual reality unit is already demonstrating that VR has an unrivalled power to convey a sense of the space that subjects inhabit, and to prod the viewer into exploring that space. On top of that, Pirog says she’s convinced that VR “can be used to create a certain depth of empathy that photos or video just can’t do.” But whether tech-enabled immersion always flowers into an emotional connection between viewer and subject—on a par with the effects writers like Talese can summon in a handful of words—will likely remain a matter of storycraft.

I spoke with Pirog by phone about how The Displaced VR came about [for more, see our 5 Questions with Jake Silverstein].

Storyboard: What’s the job of the producer in a project like this, and how did you zero in on the three children featured in the film?

Pirog: Back in July, I started studying each of the refugee conflicts around the world and started looking at internally displaced conflicts as well. We decided to focus on South Sudan, Ukraine, and Lebanon in order to get a variety of conflicts in regional locations. But we were also working with various groups on the ground who were working to support these populations. We were trying to find children who were between the ages of eight and 12, before they start exhibiting adult-like characteristics, but are old enough to kind of understand what’s happening around them. And they also had to be children whose families and parents would allow them to work with us, and who had a certain charisma to them, to be filmed and to talk openly about their experiences. And they and their families had to be okay with us being around for a couple of weeks.

How much could you plan or storyboard the stories told in the film in advance, and how much was improvised in the field?

We had a lot of the logistics set up in advance….My role as producer was to research a lot and talk to as many people as I could find who were working out in the field, and figure out the best places to go to tell the story, and help with the logistics to get everybody there. We were working on the VR film and also the print portfolio for the print magazine. All those shots were done by Lynsey Addario, and she had to go separately so they could stay out of each others’ shots. It was a huge production to get everybody and their organizations working with the same families.

We knew where they were going. We had maybe a few children in mind that we would feature. But we left the choice to the two directors and the reporters we sent into the field. This film was made in partnership with a VR production company in Los Angeles called Vrse.works, but we also had our own New York Times journalist Ben Solomon in the field as co-director.

They were working hand-in-hand. Vrse.works is very knowledgeable and versed on the cinematography and technical aspects of filming in VR, but we also wanted to make sure we were telling the story in a New York Timesian way—that it adhered to all the strict guidelines that we have here at the Times. We also had reporters in the field interviewing the kids and sussing out the story with each of these kids, a small team of three to four people for each location.

Basically, they would arrive, they would have about a week, and at first they would interview the kids and their families and hear about their stories and their lives and what their daily activities were like. Then we really wanted to get a feeling of a day in the life of these kids. Ben would study their movements and see how the action of a day would play out, and then he would try to put the camera in the way of that action….They went into the field knowing that we wanted that day-in-the-life feeling, so we knew we wanted footage gathered in the morning, the afternoon, and the evening, as much as we could get of their daily quest, as we called it. Do they have to go to work, or fish, or whatever? We wanted to show what their lives were like and let people stay there with them.

We didn’t have a storyboard until we knew what they spent their days doing, what their actual lives were like…With Hana in Lebanon, they knew that the kids had to wake up at 4:00 in the morning or so, and be in that truck being driven to the field by 4:30, so they had to mount the camera on the side of the truck, and by the time [the kids] got there, the camera was just there. It’s an ubiquitous object in their lives. Then [the crew] would essentially hide. They’d set up the camera and hide, and let some sort of action happen in front of the camera.

When everybody came back, we edited the thing together. Jake Silverstein was really the driving vision behind this. He is an incredibly talented and amazing editor-in-chief, and had never-ending energy for this project. So we and the Vrse.works team came here to New York and sat down and edited the piece together to figure out the story with the interviews and how it would play out, how long it would be, everything like that.

Right now, with VR, the post-production is very time-consuming, to stitch together the pieces so that it flows in one sphere. We had to leave enough time for stitching and color correction and things like that. I’ve just kind of managed that entire process, working closely with all the other teams working on NYT VR, like the app team. We all met once a week and get together. Everybody working on this was from a different department. It was pretty cool to work on a project like that, where everybody from the Times who was working on this had another full-time job.

We are all learning as we go along. Every day, there is something new to learn about VR and the app. Now we’re looking ahead into 2016 and all the new projects we can work on.

How long did this all take? Were you working on the film from July right up to the day of the release on November 5?

Oh, absolutely. Right up to the last minute.

What types of stories or subjects lend themselves well to VR? And can you imagine stories where VR would be the wrong tool to use?

I think there are certain stories that lend themselves better to VR….Those will be ones where it will serve the viewer well to actually spend a few moments in that person’s shoes or in that place, where it will create a deeper understanding of what that place is like if they can actually go there for a few moments….Any place that you wish you could transport your reader to, any situation that you wish they could understand on a visceral level, those are the ones that are grabbing me the most for VR. If you have a really complex history, with context that you need to explain, it might be better suited for a more traditional film, or words or photos. I don’t really think that VR is going to replace any other medium. It’s just going to be another tool for us that we are going to use for very specific types of stories.

My questions are in red; her responses in blue.

To follow the time codes below, view the film in smartphone mode, without the Cardboard viewer, which obscures the time codes. Or visit the Web version.

 

There are a lot of very long shots in the film. We meet Chuol, the Sudanese boy, about 24 seconds into the film, at the beginning of a shot where he’s pushing off into the marsh, in his canoe. That shot lasts a leisurely 22 seconds, and seems to set the pace for the rest of the film.We knew we had to get long shots. With VR filmmaking you don’t really want to do quick cuts to take people from A to B, like you can with a traditional film. You need to give people time to look around and engage with what’s happening around them.

 

At 1 minute 30 seconds, we meet Oleg, the 11-year-old Ukrainian boy, who’s seemingly alone in a nearly demolished schoolroom. He’s clearly been asked not to move, just to stand at a certain distance from the camera and patiently stare. I bet some careful staging goes into a scene like that. The decisions are difficult, because you never want the action to be too far away from the camera, or too close. There is a sweet spot to let people feel they are interacting with the subjects and the people in the film. It can be, at times, a little frustrating for a traditional filmmaker who is used to moving around and getting the right angle or exactly the right distance.

 

A little later, from 2 minutes 8 seconds to 2 minutes 23 seconds, Oleg and his friends are playing on a rooftop and they’re dropping metal objects off the roof of the ruined building. What are those—are they spent shell casings?Yeah, those are shells, empty shells. That particular location was a place that they played, which is kind of crazy. Then there were other places like their school that was a place they visited, and their parents said we could go visit them.

 

Everyone who has written about “The Displaced” seems to agree that the food drop scene, starting at 5 minutes 20 seconds, is the one that brings home the power of VR most viscerally. Did you know right away, or even beforehand, that this scene would go into the finished film? Me too, I feel that way. We’re all very informed people, we know exactly what a food drop is, and we may have seen a ton of pictures or even video of it, but there was something so different about standing there and feeling the violent “thud” of the packages, and then the way the people were collecting the bags. They earn money based on how many pounds they collect, so there is a real scramble to get to the bags first, so you can earn a little bit of money or a little bit of food. It’s pretty intense. I think that was one of the last scenes that we shot. We filmed in South Sudan last, and I think at that point we had really started to hit our stride and to think about the space differently. By space I mean the 360-degree space of being in a VR film. We started to think, “Oh man, if you could be standing right here, you’d understand differently.” So when I heard from our contact at UNICEF that there was [going to be] a food drop, it was like, ‘Oh my god, we have to be there.’ I knew it would be a very different experience from that what I had seen before with photography. The photos that Lynsey Addario made of that same event are extraordinary, but it is such a visceral experience to hear the planes. If you think about Chuol’s world, to him, planes meant bombs, and the first time they heard them, he and his friends thought they were dropping bombs, but it was food. There is a unique poetry to that.

How do considerations of ambient sound, sound design, and music differ in VR? I think if you are going to be fully immersed in a place, being able to hear it is the most important thing. So I love the shots where you are just kind of listening to people talk around you, or hearing what the environment actually sounds like. That is incredibly powerful, to be able to look around and find the source yourself. Filmmakers are going to be using audio in completely different ways that we have never even thought of, in the next year or two. It’s going to be a really exciting time for audio and soundscapes. And who knows, but maybe music also. It’s important right now to provide a kind of flow through the whole film, but I don’t hear music as I’m going through my regular day. Does it actually create a different experience if you don’t have music? We are going to try a lot of different things… Now we are all looking at positional audio, so that you actually record the audio in 360 degrees as well, and you can beckon people’s attention. If they hear something in their right ear, they might turn and look.

 

What about graphics, captions, subtitles—are those decisions different also? There is not an established tradition for subtitles or any of that, which is kind of the best part. Reading in a VR space can be a little difficult. That’s something we discussed a lot—the tradeoff with that. Would we translate the kids’ words so you would hear them in English? There was something about being able to hear their voices in their languages that put you in the place that really worked. So we used subtitles. Again, I think the tech is going to advance pretty quickly on that. Right now, the subtitles are baked into the film in three places around the scene. [The first example comes at 1 minute 54 seconds.] But eventually there could be another solution for that, like having the titles follow our gaze.



There seems to be a taboo in VR filmmaking against having the camera operator appear in the shot. They tend to set up the camera, then hide. Why is this so important—and can you imagine a VR film where the camera operator stays in the shot? Wouldn't this be more true to life, in a way? Yeah, definitely, I totally agree. It’s funny because I’m traditionally a photo editor, and have worked on some video, where you have always had to consider it. You always have a place to stand when you’re shooting a photo. You can’t show yourself as involved in the scene. And this totally changes in VR. I think we will have to play with this. When it comes to doing more news-oriented pieces, like the Paris vigil, it just might not be as important to have the filmmakers be absent from the scene. For this one we thought it might be jarring, potentially, to see the filmmaker as part of the scene. But yeah, I don’t know—it’s something we talked about a ton here. We did decide in the end that it’s not different for the filmmaker to step behind a truck and hide than it is for them to stay behind the camera in a traditional documentary. We decided to go with it, but it might not become as important.

 

There’s a nice shot starting at 8 minutes 4 seconds where there’s a boy running with the VR cameras at the end of a stick, and if you spin around you can see him holding the stick, which gives some useful context. That kid was especially good at it. When the camera rotates in space, it can make people pretty seasick. He held it still. So he was a good cameraman, and they were having fun with it.



The atmosphere in all three locations is not one of unremitting conflict and violence, the kind of stuff we see on TV news. Instead it was one of daily survival, even punctuated with some lighthearted moments and beautiful landscapes. Was it a deliberate choice to capture the characters in these everyday situations, halfway between war and peace? Yeah, that was actually deliberate. We really wanted to show you what the life of a refugee feels like. It’s not as dramatic as you think it might be, after they have left the war-ravaged place or the conflict they are fleeing. What changes when they are in these places? People in refugee camps, they have nothing to do. They don’t have jobs. They might not have any fuel to cook their own food. They don’t have their communities, their family, their friends. Obviously, communities are created in these camps, but there is a general feeling that they are all just stuck in a time and a place, and they can’t move forward and they can’t move back. A lot of the kids, they don’t get to go to school, because there is no school. Time and again in all these interviews with these groups working with children, that’s all these kids want, is to go to school, and they can’t. So yeah, we really wanted to make the viewer feel what life is actually like as a refugee.