You’re standing in the middle of an eerily empty two-lane road. Cookie-cutter apartment complexes surround you. Broad-leaved trees line the street. It looks like an average American suburb, but something’s not right.
You look left, then right. Yellow police tape blocks off the street, and red and blue lights flash in the distance. You move forward a bit and notice the white outline of a body on the asphalt, sprawled with its left hand above its head. Glowing arrows beckon away from it. Following them, you end up at the passenger-side window of a police cruiser. You enter a flickering cylinder. It brightens, and a comic strip appears showing an illustration of a man wearing a baseball cap, looking down the road you just walked along.
This is a 3D rendering of Canfield Drive in Ferguson, Missouri, where Michael Brown was fatally shot by police officer Darren Wilson on August 9, 2014. Brown’s friend, Dorian Johnson, who was with Brown when he died and is the man in the baseball cap featured in the illustration, is just one character you meet in this virtual world, created by graphic journalist Dan Archer with the help of photographs, satellite imagery, and video game software.
Archer, through a fellowship at the Donald W. Reynolds Journalism Institute and a partnership with Fusion, where “The Michael Brown Shooting Visualized: Eyewitness Accounts” was published last December, reconstructed eight eyewitness accounts for users to explore, guided by arrows that lead to the locations where each person observed the event. He notes that users spend on average over 10 minutes with his Ferguson piece, “practically unheard of in the ADD [attention deficit disorder] world of online news,” he says. Those are promising stats at a time when media outlets are fiercely competing for users’ attention. Immersive journalistic experiences like this one could become a way to keep audiences engaged while offering reporters innovative new ways to tell stories.
The goal of journalism has always been to be immersive, to bring audiences as close to unfolding events as possible. New Journalism icons like Hunter S. Thompson, Tom Wolfe, Joan Didion, and Gay Talese practiced their own form of low-tech immersive journalism by inserting themselves into their stories in imaginative ways. The qualities that lend virtual reality its “wow” factor are the same ones inherent in any well-crafted tale, fiction or non. Though his digitally rendered version of Ferguson is far from seamless—the graphics are pixelated, and the mouse controls don’t always work the way you want them to—Archer believes immersive storytelling is the best model for presenting “complexities, ambiguities and all-out contradictions inherent in larger, longer-running stories.” Someday, he hopes, this interactive model could give the old-fashioned feature a run for its money.
Immersive journalism is picking up now in part because the necessary technology has gotten better, cheaper, and more portable. The smartphone’s ability to stream high-definition video as well as its increasing popularity—58% of Americans had one as of January 2014, according to Pew Research—have further accelerated adoption. Add to that the more widespread use of interactive data visualizations and advances in wearable computing and the stage may be set for a more robust adoption of virtual reality. “It’s not just the media coming to you,” says Dan Pacheco, professor of journalism innovation at the Newhouse School at Syracuse University. “You move into the media.”
[pq full]Immersive journalism is picking up now in part because the necessary technology has gotten better, cheaper, and more portable[/pq]
Immersive storytelling was on prominent display at the Tribeca Film Festival last spring in the form of “Use of Force” by former Newsweek correspondent Nonny de la Peña. When you strap on the headgear to experience the piece, the first thing you hear is crickets, then screaming. Two uniformed agents drag a man dressed in white to the ground and start kicking him. A dozen other officers stand quietly by as he screams. Other bystanders watch in horror. A man asks, “Why are you guys using excessive force?” A woman shouts, “He’s not resisting! He’s not resisting!”
This incident occurred in May of 2010 at the U.S.-Mexico border near San Diego. The death of Anastasio Hernandez-Rojas, the man in white, was ruled a homicide by the San Diego coroner’s office. Border agents maintain force was necessary because Hernandez-Rojas, who had methamphetamine in his system, was combative. It is a very disturbing few minutes to re-live. Though the computer-game feel of the graphics creates some cognitive distance between you and the action, you are confronted with the stark brutality of the beating in a way that feels more intimate than the documentary footage on which the virtual rendering is based. In the immersive version, you feel powerless to stop a violent act that feels like it’s happening before your very eyes.
De la peña also presented an immersive piece at this year’s Sundance Film Festival, “Project Syria.” It transports you to a lively intersection in Aleppo, Syria. You hear a girl sing in Arabic. Suddenly, a bomb explodes. People flee in every direction, and through the thick cloud of debris you see several bodies lying on the ground. A screeching fills your ears. When it stops, a narrator’s voice breaks in, “A third of all Syrians have been displaced by the war.” A child runs by as the narrator continues, “Reports indicate children have been specifically targeted in the violence.” The chaotic sounds crescendo and then cease as the scene fades to black.
You are then transported to a desert, with trailers and tents visible in the distance. Translucent white figures stand before you. More and more figures and more and more tents appear as the narrator continues, “There are now over a million refugee children.” For de la Peña, the virtue of virtual reality is that it puts “people inside the story so they can experience the action as it unfolds. [It] allows you to experience stories in a visceral way.”
Just a few years ago, head-mounted displays that simulate 3D environments cost tens or even hundreds of thousands of dollars. They were rarely found outside military research labs. Then, in 2012, inventor Palmer Luckey raised $2.4 million through a Kickstarter campaign for the Oculus Rift virtual reality system. He offered development kits to early adopters who wanted to make games for it. Facebook bought the company for $2 billion. Though the Oculus Rift has yet to reach the general public, anyone who wants to try it out can order a prototype. Other competitors have also launched consumer-grade devices.
Video game technology has attracted journalists looking to experiment with storytelling forms, including David Dufresne, who is a fellow at the Massachusetts Institute of Technology’s (MIT) Open Documentary Laboratory. Dufresne is convinced journalists can deepen users’ experiences by borrowing narrative and nonlinear techniques from filmmakers and gamers. “The video game industry has revolutionized the narrative,” he says, particularly by encouraging audiences to participate in the story.
In “Fort McMoney,” Dufresne put the fate of a real Canadian town in Alberta, Fort McMurray, into users’ hands. Fort McMurray sits atop a large oil sands reserve, and you get to decide its future in a Web-based documentary game. The experience opens cinematically. Driving down a snowy highway, you pass an overturned car. A woman’s voice narrates, “You have reached the end of the road, at the world’s edge.” You’re eventually left to explore the photorealistic setting.
There’s a woman facing you from the passenger seat of a Ford F250, door ajar. When you hover the mouse over her, everything around her blurs. When you click, titles appear on the screen identifying Marquesa Shore, a waitress and car saleswoman. She arrived in Fort McMurray two months ago and earns about $10,000 a month. “It’s good,” she says. “It’s good to be a woman here.”
Users are then offered two choices: Get Into Her Pickup Truck or Speak To Her Later. If you choose the latter, you can explore other characters. If you choose the former, you take a ride in her truck as she tells you about life in Fort McMurray. A few minutes later, she offers to drop you off at City Hall, where you can meet the mayor. Throughout the game, users refer to a dashboard that tracks their progress. For Dufresne, this kind of gamification can attract users to stories they might otherwise ignore. “Nobody wants to read a news report or watch a movie about environmental issues,” he observes. “What we saw with ‘Fort McMoney’ is a lot of people who came for the game stayed for the topic.”
[pq full]MIT’s David Dufresne aims to create a template for interactive documentaries[/pq]
One of the biggest challenges in constructing a story with so many different possible outcomes is thinking non-linearly. Rather than watching a documentary that unfolds scene by scene, the audience explores the people and places of Fort McMurray based on their preferences. If you don’t want to meet the mayor and hear her perspective on things, you can click away. If you’re interested in the real issues behind the game, you can debate them with other players. As part of his fellowship at MIT, Dufresne is building a tool that he hopes will make it easier for journalists to create interactive projects of their own, a kind of Microsoft Word or Final Draft for interactive documentaries. “We lose a lot of time explaining to each other what we are doing,” he says, and he hopes his software template will enable editorial teams to collaborate more efficiently.
There are potential downsides to immersive experiences. For starters, virtual reality technology can make some people feel sick. Simulators make you think you’re moving when you’re not, and some experience motion sickness. “You don’t always get a good match between what the sensors in the system are reporting compared to what the inner ear is experiencing,” says Douglas Maxwell, a project manager for the U.S. Army who has been studying virtual environments since the late 1990s. The Oculus Rift developers have tried to address the problem of motion sickness by reducing the lag that can occur between a user’s actions and the reaction of the program.
Simulations that supposedly portray real-world events also raise psychological questions, such as: Are you “you” when you enter these worlds or one of the individuals depicted? How do you distinguish your own views from whomever’s perspective is being displayed? And where does the journalist exist in these spaces? Karim Ben Khelifa provocatively plays with questions like this in “The Enemy,” an audio-video installation that puts users directly between two soldiers on opposing sides of a conflict.
Khelifa is collaborating with D. Fox Harrell, founder of MIT’s Imagination, Computation, and Expression Laboratory, to tailor the action to each individual visitor. “Your virtual representation shifts depending on how you interact with the soldiers,” Harrell explains. Are you looking the character in the eye, shifting to the side, or spending more time with one person than the other? The answers alter the narrative. That sort of advanced interaction creates a sense of intimate presence, says Harrell. You’re not just telling people a story; they’re participating in the creation of it.
“There are a lot of ethical questions, but it’s not unique to digital media or virtual reality,” says Harrell. “Even media that seem to be very direct are actually very subjective.” Harrell points out that a unique advantage of stories with game-like qualities is that people see different perspectives with each exposure to the story and can come to their own conclusions.
As virtual reality enters newsrooms, journalists will need to develop standards for working in the new form. One of the first to test them is The Des Moines Register with an immersive experience called “Harvest of Change.” It brings readers into the simulated world of the Dammanns, who have run their farm for the past century, to illustrate how climate change, new technologies, and cultural shifts are affecting agriculture. It combines a computer-generated world with 360-degree video to depict the nuances of farm life. In the video scenes, you feel like a voyeur, witnessing intimate moments like a father-son outing on a tractor.
The 360-degree video Dan Edge is shooting in Africa will give Frontline viewers a new perspective
A team at Frontline, in collaboration with the Tow Center for Digital Journalism at Columbia University and the Canadian company Secret Location, is taking users into the Ebola crisis. Director Dan Edge is filming 360-degree video in West Africa for the project, which includes a standard 2D, linear documentary as well as digital interaction. The project will present the Guinea village of Meliandou, where scientists have pinpointed what they believe is the first case of Ebola. In one planned scene, you can explore the inside of a towering, hollowed out tree, home to a colony of bats that scientists are studying to determine if they were the source of the outbreak.
While “Harvest of Change,” Dan Archer’s Ferguson piece, and the pioneering work by Nonny de la Peña rely heavily on computer graphics, the Ebola project replaces 3D modeling with 3D film footage. Raney Aronson, deputy executive producer at Frontline, envisions layering multimedia data visualizations into this 3D environment. “The dream would be that you could go inside the tree and then explore everything we know about it”—from inside, she says.
There are many questions to address as the team completes the project in the coming months. A big one, says Taylor Owen, research director at Tow Center, is: Where do you situate the journalist? What does narrative look like when cuts are no longer needed because the camera captures an entire room at once? In documentary interview scenes, for example, the subject is often the only person on screen. With 360-video, both interviewer and interviewee are captured and the viewer can look back and forth between them, just like they would in real life.
Though the technology is new, the ethical challenges facing journalists are not. “Any time you’re creating a computational world or self, you’re abstracting from the real world, taking some elements from the real world, leaving some out,” says MIT’s Harrell. “There are changes.” The imperative of remaining true to the reported facts is the same, regardless of whether the story is intended for the evening paper or the Oculus Rift.
