Document

Distant Beats

Chang, Issac, Jiyoon, Xander

Link to presentation: https://docs.google.com/presentation/d/1-2B7tiZN5a6YG-W14_FKoLXLWyJQNzQkp6jtrZVzBHM/edit?usp=sharing

Concept

Two people in different places view a scene together remotely, with its visuals affected by their presence and heartbeats, allowing them to share emotions through the visuals.

More specifically, 2 people’s inputs from a heartbeat and ultrasonic sensor are fed onto a Heroku server, which are then transmitted to a Unity scene both can view, subtly affecting its visuals and mood (with each player’s inputs controlling an aspect of the visuals--the clouds and grass). Inspired by the disconnectedness of life during this pandemic, we hoped to provide a passive, meditative, and peaceful means of connection. 

Collaboration

Collaboration and coordination was mostly done via text through a Discord server, with meetings about every week (more often towards the end). Isaac and Jiyoon also managed a github repo to share the code, and we used a Notion page to keep track of tasks. 

Roles

Chang: Built the Unity scene (grass, trees, UI, colors), C# (script for sensor data to interact with the shader, and camera), grass + tree shaders, and color concepts.

Issac: Networking (with node.js, a websocket server, and Heroku), and a Unity prototype for testing.

Jiyoon: Arduino setup, lighting, controller design and assembly, and installation setup.

Xander: Prototyped the concept for the scene. Modeled the buildings, and made the sky + building shaders.

Process

Installation & Controller:

From the outset, we decided we wanted a visual art piece rather than a game, and so an installation felt like a necessary thing to base this around. Jiyoon developed the initial idea, based around the user entering an immersive space, and in the final version this was realized with a roomscale projector and immersive LED lighting. 

Jiyoon also developed and created a large laser-cut physical receptacle to house the controller, with a space for the user to place their hand on the heartbeat sensor. Its design was inspired by traditional Korean patterns to complement the scene that was displayed on the projector.

Networking:

Networking was achieved by each active client sending values from the Arduino and sensors to a Node.js app, which was then transmitted via a string to a WebSocket server on Heroku. This was then fed into Unity, where the string was reconverted into individual numbers, which were then used to manipulate values on the scene. To develop this, Isaac did extensive prototyping using a Unity app, and then directed the team in Korea step by step how to duplicate his results.

Mechanics & Input:

+Pulse sensor: In this project, a total of 2 pulse sensors affect the virtual space. Each sensor will affect different elements; UserA will affect grasses, trees, and butterflies and UserB will affect sky, and clouds. 

If no one has access, the tree has no leaves and provides a dark sky.

If UserA accesses, it will make the tree grow.

If UserB accesses, the sky brightens and clouds appear.

If both user access, music will be turned on.

When the heart rate falls between 60-100, the color changes. To show a more dramatic scene, range is narrowed down.

UserA affects the grasses, trees, butterflies.

UserB affects the sky, clouds.

+Ultrasonic sensor: We made it possible to change the field of view value of the camera with an ultrasonic sensor. The closer the distance to the sensor, the smaller the value of the field of view(zoom in). The default state is the most zoomed out state.

+Keyboard: 

Space: Toggle heart UI

Esc: Manual operation function, Exit

Shaders:

We focused on creating a flat and stylized shader rather than creating realistic ones. We were inspired by the games Journey and Firewatch. 

Chang focused on the grass and trees, and was able to learn the optimization work to change the material of many grasses at once. This shader was a huge help in creating a lively scene efficiently. With very little resources, things like grass and trees, butterflies and clouds can move dynamically, creating natural feeling scenes. 

Xander focused on the clouds, created dynamically using Perlin noise, and with a volumetric effect achieved rendering on many layered planes. This allows the clouds to dynamically change size, which he attached to one of the input heartbeats to have the clouds pulse with the heartbeat, changing size depending on the intensity, and color depending on the scene’s general mood.

Color:

When choosing a color theme, we tried to visually show utopian and hopeful feelings. The color theme can be divided into two main types: when the two heart rates are highest and when both heart rates are low. At the highest values, red and blue, which are complementary colors, are used to give warm and intense feelings. When the value is low, I wanted to visually show that the values are low, but we didn’t want to give any negative or scary feelings. For this reason, Chang chose bright pink and light blue with a neon feel.

Models and scene concept:

The models for the scene were created using Blender, and, along with the shaders, aimed to achieve a flat and stylized look, hopefully providing both clarity in viewing and helping with optimization. 

We decided fairly early in the project to go for a traditional East Asian style scene, since for our demo we were connecting between the USA and Korea, and in the end settled on buildings and an environment in the style of a Korean traditional palace garden, since we felt it helped the peaceful mood we were going for. 

Insights & Findings

Organize objectives and focuses early in the project

While in the end we’re all happy with our results, a lack of planning early on caused us some grief; for scene conception especially, effort was devoted to features and ideas that we weren’t entirely sure how to integrate. Xander and Chang also both wrote code that had very similar functionality, since it wasn’t entirely clear who would be doing what. Better planning could have alleviated this a lot and perhaps resulted in an even better end product.

Don’t be afraid to change things

On the other hand, Isaac experimented a lot with different platforms to give us the best results with our networking, to great effect. Since we’re all still students and learning, sampling in many things can give us a very valuable broad perspective, and we also shouldn’t be afraid to change direction a bit if things go wrong. 

Different skill sets are helpful for collaboration

Since 2 people in our team are in the CT track, and 2 people are in the game design track, the two groups approached the project with very different ideas of the kinds of things they wanted to make, and also the means to achieve the goals, allowing a much wider range of possibilities than might have been possible individually. For instance, the CT students brought with them knowledge of a wide variety of different plugins and tools to allow the networking capabilities, and the game design students had knowledge of shaders and modeling in Unity that (subjectively) allowed for a relatively richly developed scene.

Improvements

Optimization

While we considered optimization throughout the creation of the Unity scene and its assets, the final result is still maybe not as well optimized as we would like it to be. With a participatory experience like this, designed for bringing people together, as low a barrier to entry as possible is always highly desirable, and not requiring an expensive computer to participate is definitely part of this.

Easier setup

Along the same lines, the setup relies on several connected components (like the Heroku server being active, and setting up the circuitry for the Arduino input) that the user will have to manage any time they want to participate. While it’s not necessarily realistic to massively streamline all of this (especially as students), alternative forms of input and more integrated modules might help accessibility.

More scenes

Finally, if we had more time, attaching more scenes and variation is another avenue for us to pursue. We’re happy with how the current scene looks, but something like changes in architecture to reflect the user’s origin, or more landscape variety, would be helpful if we were aiming for this to be something users participated in with regularity.