A few days ago, while having a meal with students and talking about papers, we got onto VR-related questions again. I suddenly remembered that essays in a similar “series” usually stop at part one or two, and I felt rather ashamed of myself, so here I am continuing the previous post.
I previously wrote Talking VR (II). After that there were actually three more pieces related to VR: On the Backboard — The Phenomenology of Beat Saber, AR Glasses and the “Brain in a Vat”, and Can VR Save Us from Our Smartphone Addiction? But the first two are mainly about philosophical questions, while the last one is a rewriting of Talking VR (I), so it is most appropriate to count this one as (III).
In (II) we had already discussed the sensory phenomena in VR experience, and also mentioned several games and their features. But at that time the analysis of specific games was not very deep, and moreover the games listed there are basically all games I no longer play now (except Beat Saber), so this time let’s talk a bit more about several other games.
1. Putting Knowledge into Practice
When talking with the student, I found that although on some viewpoints and positions he was in agreement with me, once we actually began to discuss things, he became stiff and mechanical, his grasp at the theoretical level was not sufficiently thorough, and he was not flexible enough in connecting theory to real cases.
What does it mean for one’s theoretical understanding to be thorough? In my view there is a simple criterion: whether you can be perfectly clear about every word you use—why you use this word, and why not that one. Whether academic terms or everyday vocabulary, every word should stand up to scrutiny, be replaceable at any time by other words, and also be able to resist substitution by certain words. That is what being thorough means. For example, the “thorough” and the “complete” I use here are roughly interchangeable, but they cannot be replaced by “comprehensive” or “accurate.” This is because “thorough” conveys an image of “bringing things into connection”; theoretical concepts and ordinary thinking are linked together, connected all the way through, and that is thoroughness. But “understanding comprehensively” has only the image of wholeness and none of connectedness, while “understanding accurately” implicitly presupposes something like a “standard answer,” which is even less apt.
We can subject every sentence we write to this kind of scrutiny, especially those key words: can some awkward terms be replaced by more everyday expressions? If so, then by all means use plain, simple words in place of obscure terminology. If not, then you must have complete confidence and be able to explain why they cannot be replaced. — So the obscurity of some philosophers is different from the self-conscious obscurity of others. In some cases, people’s thought really is subtle and nuanced to a very high degree, so they must express it through the flexible use of specialized vocabulary; but in other cases, people keep a great deal of obscure language simply because they are being deliberately mysterious or, quite simply, have not thought things through.
As for applying ideas flexibly to examples, that is in fact part of the same “connectedness.” Abstract concepts only realize their meaning when they ultimately land in intuitive, familiar life situations. For instance, if you have spent ages defining VR, but the things that are actually popular in the market under the name of VR do not fit your definition at all, then that requires explanation—the definition may not necessarily be wrong, but you must be very clear about where the problem lies, how this deviation came about, and why a definition with such a deviation can still hold. Indeed, it is precisely worth reflecting on ourselves: why is it that when definition and reality do not match, we nevertheless, at first glance, still find the definition reasonable and valid? What habitual patterns of thought does this neglect of deviation reveal? When I have recently discussed “technology,” I have often begun from this question.
2. The VR Mother-Base
To return to the main point, when we discuss VR, one core theoretical question is: what exactly is VR? Of course, our theoretical background is media philosophy, and of course we regard VR as a kind of “medium,” but what kind of medium is it? The view my student cited calls VR the “ultimate medium,” believing that VR technology will eventually extend all five human senses at once, enabling all human senses to transcend spatial barriers and become a perfect, ultimate medium. But I think VR is obviously not the “end point” of media, because human beings originally survived in the real world with all their senses, and all the media from antiquity to the present have developed out of “the whole sensorium.” Their tendency is not necessarily to allow all senses to obtain information more comprehensively and abundantly; on the contrary, many media are actually making reductions, such as the development of writing and symbols. In many situations we require the simplification and control of our senses in order to create a richer world of meaning.
Take chess, for instance: I actually do not want the “horse” in my hand to become furry and start kicking people, nor do I want to hear a deafening roar when I move the “cannon,” and still less do I want a pawn to start bleeding all over my hand when pieces are exchanged. I only want these pieces to be simple, plain chess pieces, appearing merely as easy-to-recognize abstract symbols, so that we can better immerse ourselves in the rules of the game. Even though many major 3D games nowadays make their scenes as realistic and rich as possible, at the level of operation itself we still, in many respects, demand simplification rather than richness.
The simplification, selection, and editing of sensory information by media occur simultaneously with enrichment, filling, and extension. McLuhan’s theory of “sense ratios” also points this out. He believed that human perceptual capacity is limited; to put it in plain language, our energy is only so much, and if there is too much information, we cannot take care of all of it. To receive information always requires selection and focus. No matter how VR develops, it can never be the end of media, because next it will always face the problem of how to find focus further within the virtual world.
So I would rather call VR some kind of “mother-base medium,” a starting platform still awaiting evolution and differentiation. It is like “paper,” which can become letters, books, newspapers, pictures, comics, punch paper tape, and so on; their characteristics cannot be lumped together in one stroke.
Of course, VR technology is still relatively early-stage at present, but even from its current situation, one can already see the existence of different developmental trends and different modes of operation. This requires analysis of concrete use cases.
3. Beat Saber
First, let me talk again about Beat Saber. Beat Saber does not provide a space that simulates the real world. What flies toward you are little blocks, one after another. These blocks have only two colors, plus an arrow, and the arrows generally point in only four directions, or eight directions. You need to use your left hand or right hand, corresponding to the color, and slash in the direction of the arrow.
The background of the space is also very concise. The later DLC added many different backgrounds, but on the whole it never becomes especially gaudy or complicated.
In games played on traditional computers or televisions, there is a saying about “music games” or “rhythm games”: the best rhythm game is one you don’t need your eyes for—you can play it well with your eyes closed, just listening to the rhythm and pressing the keys. But obviously, Beat Saber in VR is certainly a game that depends very heavily on vision; you can still play it with the sound turned off, but if you close your eyes, you probably won’t be able to play it at all.
But we can still see that it is not a vision-centered game. Although it requires vision, it does not, like book reading, require that vision be separated from the other senses. It organically ties vision together with hearing (musical rhythm), kinesthesis (bodily movement), and touch (vibration feedback) into a whole.
McLuhan often spoke of so-called “visual space” and “sound-touch space.” The latter is not the space of blind people; in essence, it is also a multisensory space in which “vision, hearing, and touch” coexist. But because vision occupies a dominant position among the human senses, when we intensify vision, we often crowd out the activity of the other senses. So if we want to promote the activity of hearing and touch, then visual information, conversely, must be somewhat suppressed.
The simplification of the visual aspect in Beat Saber is one example. In fact, rhythm games on traditional platforms are similar in this respect as well: visual images often occupy a subsidiary, supporting position. We might be able to replace the little blocks and arrows with vividly lifelike flying insects and chunks of flesh, but if what we want to play is a music rhythm game, then we do not need to do that. Simplification in the visual realm instead preserves the dominant position of hearing.
4. Richie’s Plank Experience
Literally translated, it is “Richie’s Plank Experience,” but we usually just call it the “jumping off a building simulator.” This very simple game became my first choice whenever I let students experience VR for the first time, and even Teacher Wu has come to try it.
It also has a somewhat more game-like mode of play, namely taking the role of a firefighter carrying a jetpack and flying around putting out fires. But its core experience is simply a very straightforward game of jumping off a building. It begins at the side of a road on the ground. You walk into an elevator, close the elevator door, and select the top floor. Then the elevator carries you upward, during which you can still see the rising scene through the crack in the elevator door. Finally you reach the top floor, and outside the elevator door is a plank not quite as long as it is said to be. Stepping beyond the plank is to jump off the building; in your field of view you accelerate toward the ground and finally enter a white light.
This plank can be simulated with a physical object. I once used a few pieces of cardboard to simulate it, and standing on it felt more real. Ordinary VR has a hand controller but no foot tracking; if you add an extra tracker and strap it to your shoes, you can also simulate the sight of your own feet stepping on the plank. But even if you add nothing at all, just a very plain and simple jumping-off-a-building experience is already enough to frighten many people.
The “jumping off a building simulator” prompts us to think about what “realism” means. Everyone knows that the VR scene is fake; when we put on the headset, we know this full well. Moreover, the whole scene has not actually been restored to a vividly lifelike degree either: the surrounding high-rises and the road under our feet obviously look fake, and at a glance one can tell that it is animation rather than a real scene. When standing on the plank and looking down, I also cannot see my own body, hands, or feet.
But even in such a “false” environment, the panic we feel is absolutely real. Several of the people who tried it did not complete the final jump; even when they only slowly stepped out beyond the plank, they retreated back to the “elevator” and took off the headset (many people first have the move of retreating to the elevator rather than taking off the headset directly). Even most of those who successfully “jumped off the building” admitted that the thrill and panic were real. In fact, once you actually step off the plank, it is not so frightening, because VR after all cannot simulate the weightlessness of free fall, but the feeling while on the plank is genuine.
In this experience, we find that “true” and “false” are no longer a binary judgment of “1 or 0,” but rather a kind of mixed, gradated experience. The sense of reality and the sense of unreality occur simultaneously, shining against each other.
5.Bowman
This is a very simple archery game. Actually, there’s an archery game in The Lab too, and it’s actually even a little more fun than this one. But it was while I happened to be playing Bowman that I suddenly realized the following little issue, so I’ve put it here.
The reasoning is very simple: when shooting arrows, I habitually close one eye in order to aim. It suddenly dawned on me that in a VR headset, closing one eye hardly feels different at all.
People can perceive depth through binocular parallax, which is a commonplace piece of knowledge and also one of the major physiological bases of VR technology. But we also know that one eye can perceive depth too; one-eyed perception of distance and depth is obviously not based on parallax, but on bodily experience and other bodily kinesthetic sensations.
I believe that someone who is congenitally one-eyed can also experience depth visually, but this sense of depth is not necessarily innate; rather, it forms gradually on the basis of bodily habit. This includes how we ignore binocular parallax and merge the visual fields of both eyes into a single field of vision, how we disregard the nose’s existence within that field, and so on. Achieving all this depends not only on the function of the eyes themselves, but also requires the participation of the body. And the VR experience bears this out: as far as depth perception is concerned, binocular parallax is important, but not necessary.
6.GORN
GORN is a truly excellent VR fighting game, with very strong immersion, impact, and violence. The friends in our reading group played it a few times too, but after one time when a classmate turned around and jabbed the controller into my face with his gun, safety considerations meant that this game was no longer open for public experience.
The game first and foremost wins with its “real” sense of impact: when you thrust with a sword or spear, blood sprays; when you swing a big hammer, it sends enemies flying. Although the tactile feedback is at most only vibration, the actions and visual feedback match extremely well, and basically every action takes effect in real time.
Paradoxically, this sense of realism depends on the artificiality of the enemy models. Visually, the enemies in this game all look too crude and cartoonish, but this surreal form instead makes the reactions they produce when struck feel more real. After all, if you were striking a real person or beast, you couldn’t have exaggerated effects like one hammer blow bending the whole body over. The simplicity and exaggeration of the visual image here happen to complement the simplification of the kinesthetic and tactile experience.
Another innovation of this game lies in its movement design. Unless you use a super-large empty room, or introduce a fully functional omnidirectional treadmill, moving around in VR has always been a difficult problem. Teleportation is unreal, while using the controller’s directional pad to move causes dizziness. GORN designs a “shuffling” scheme that is neither teleportation nor dizzying. The operation is similar to “rowing”: you move forward by holding down the controller and “pulling” backward. It’s like a person seated in a boat moving forward or backward by rowing. The movement speed is not fast, but it produces almost no dizziness.
I can’t help wondering: in the future development of VR, is an omnidirectional treadmill really necessary? Maybe not. Because acting in virtual space does not necessarily have to completely replicate the everyday movement patterns of human hands and feet. Human beings in the virtual world can already “soar to the sky and burrow into the earth,” unconstrained by the limits of hands and feet. The real problem to solve is not how to simulate leg movement, but how to deal with motion sickness. Motion sickness usually occurs when visual motion and bodily motion are mismatched: your eyes see yourself moving, but your body feels that you are not; or vice versa.
Even with an omnidirectional treadmill, the purpose is actually to create the illusion of bodily displacement while the body itself is not moving. GORN’s rowing mode proves that this illusion can also be created without moving the legs. Perhaps future home VR would be better off with a motion seat rather than a tiring omnidirectional treadmill, though of course the action design would need to be optimized in coordination with the seat.
7. Virtual Cinema
In discussions with classmates, one classmate once excluded VR cinemas from the definition of “VR.” Because this use is simply too un-“VR.” What I mean is not watching VR videos with VR goggles, but using VR goggles to watch traditional 3D films or even 2D films.
The iQIYI VR I bought is mainly about this function. It provides the experience of a virtual IMAX giant-screen cinema. It’s as if you were sitting in a movie theater, enjoying an IMAX film all by yourself. But what exactly is the difference between this experience and going to a movie theater to watch an IMAX film?
I only used the earlier iQIYI 4K headset to experience it, and it already felt capable of matching the actual moviegoing experience. Although the resolution is certainly not fine enough, the “giant screen” can be made larger than most real movie theaters.
Neck movement is still important. Precisely because the neck can move, the tangible sense of a vast cinema space becomes prominent. I believe that as the resolution of VR goggles continues to improve, the sense of presence in VR cinemas will continue to strengthen, and it may well become even more stunning than a real IMAX theater.
In addition to simulating the real giant-screen experience, VR cinemas can also do some further things. For example, friends can watch together. iQIYI long ago introduced this experimental feature, allowing friends to put on headsets and watch the same movie together, with avatar figures appearing as adjacent seats inside the theater, so that they can talk while watching. This is novel for the traditional moviegoing experience. In traditional shared viewing, one can only whisper in someone’s ear; in a VR cinema, one can speak aloud.
But this novelty is not necessarily a better experience. Traditionally, when couples like watching movies together, what they want is precisely this quiet atmosphere, where during the screening they can only communicate in a more private way. But when the space for communication becomes open, the crucial difference between film and television starts to fade.
8. Spice and Wolf VR
In addition to using VR to watch traditional films, of course we also use VR to watch specially made 180° or 360° videos. But the experience of this kind of VR video is not especially good at present, and the key issue is that the viewpoint cannot move freely; at most, one can enjoy the freedom of turning one’s neck. There are still very few VR videos that allow you to freely move while watching within a scene. The VR version of Spice and Wolf represents an attempt in this direction. The content of this interactive video is not much, and the scene is very limited too, amounting to no more than a bit of interactive dialogue in a small room, but it may foreshadow a new artistic form and narrative mode.
9.Half-Life: Alyx
Alyx is a hyper-modern VR game, feeling as though it revealed the shape of 3A games under VR ten years ahead of time. The innovative gravity-grab mode feels excellent, and both the scenes and the controls are superb.
But what shocked me most was playing the “Jeff” chapter; this level can be called a divine chapter. Jeff is an unkillable boss, and I had to solve mechanisms right under its nose, pass through the scene, and finally lure it under a trash compactor to crush it to death. Jeff has no vision; it acts by hearing, so I could remain still right under its nose and hope it would walk somewhere else, but once I made a noise with my footsteps (by moving), it would draw its attention. I could distract it by throwing beer bottles far away.
For a stretch, the entire scene is dark, and I only have a flashlight that can illuminate a small patch right in front of me. I need to find the power source and reconnect it. When fiddling with mechanisms, I’m usually facing a wall, so I can’t detect Jeff’s approach first by sight; thus Jeff’s footsteps also become the signal by which I judge when I need to respond. This is a battle centered on “audition.”
Here I experienced for the first time the sharp distinction between a VR 3A title and a traditional 3A title. In traditional games, it’s very hard for me to feel the oppression coming from “behind.” Jeff may have a frightening appearance, but what is most terrifying about it is precisely when it is “absent”; this sense of approach from outside the field of vision makes one tremble all over.
It should be noted that the crux of excellent VR games is not visual, but the linkage among audition, kinesthetics, and vision. Simply pursuing visual realism may not necessarily enhance bodily immersion.
Translated from the Chinese original with AI assistance. The original text is authoritative.
Leave a Reply