June Bee | NYU | Interactive Media Arts
Capstone 2024
This is June's blog page for Capstone 2024 project. I will be documenting my ongoing capstone process in a venue where their classmates and teacher can see them ( a public-facing blog). Regular posts will have my progress beyond those called for explicitly in assignments.
TABLE OF CONTENTS
01. Portfolio
Make a web-based portfolio of your work. Include at least three items in the portfolio. If you already have a web-based portfolio, make an experimental portfolio that takes a different approach from the portfolio you already have. Post a link to your portfolio on your blog.
02. Concept Abstract
Due at the beginning of week 3 (2024-02-06). Based on our in-class exercises, compose three to five speculative abstracts for Capstone project ideas. Write these abstracts as though you have already completed the project in question. Each abstract should be 100–200 words and accompanied by at least one reference to a project, paper, article, talk, or other piece of media that is adjacent to the project idea. For each abstract, identify audiences, venues, publications, etc. that might be a good “fit.” Make a post on your blog with these abstracts.
03. Sketches & Prototypes
Due at the beginning of week 5 (2024-02-20), but we’ll do check-ins in week 4. Using the techniques discussed in class, make lo-fi prototypes related to at least three concept abstracts from last week. How did you identify the underlying questions or problems that could be best addressed through prototyping a particular idea? What did it feel like to work on the prototype of one concept, versus another concept? What did you learn about the concept in the process of prototyping? Document your prototypes on your blog.
04. Introduction & Annotated Bibliography
Due before class on Week 6 (2024-02-27). Prepare an annotated bibliography with at least eight entries for your (hopefully now finalized?) project idea. The bibliography should include papers, articles, artworks or other media that are relevant to your project. Each item in the bibliography should have an annotation describing that item’s relevance to the project and how the reference has been or will be useful in executing the project. Additionally, rewrite the abstract for this project idea and compose a rough draft of your research paper’s “introduction” section based on the project idea. Bring copies of all of these to class.
05. Research Paper Rough Draft & Project Proposal
Due at the beginning of week 7 (2024-03-05). Have a completed rough draft of the “preliminary” research paper: Abstract, Introduction, Related Work, Methodology. The draft should be somewhere between 2000–3000 words (already the bulk of the paper!) and serves as your “project proposal”: what you intend to do, why you think it’s interesting, and how it will contribute to the field. Submit this to the homework form. Prepare a short presentation on your “project proposal” for class next week.
06.
Due week 8 (2024-03-12). Two deliverables this week:
Production plan: Develop a production timeline for your project. The production timeline should take the following into account: (1) work you’ve already done on the project; (2) material and logistics (e.g., estimated arrival times for parts and fabricated components; scheduling time in rehearsal facilities or studios, etc.); (3) your own availability (how much time can you reasonably spend working on your project?). We’ll discuss techniques and formats for developing this timeline in class. Post your timeline to your blog.
Peer review. You’ll be assigned a number of proposals to review from your fellow capstone students. Review the proposals and leave feedback in the categories supplied by the peer review form. (Link on class homepage.)
07. Production update
Due before week 9 (2024-03-26). You should be deep in production at this point. Post an update about your progress on project production to your blog.
08. Project Show and Tell
Due before class on session 1 of week 10 (2024-04-02). Prepare a “Show and Tell” presentation of your interactive project (6 to 12 mins), using the techniques and tools that we discussed in class. This should be a traditional slide presentation that shows your interactive project in its current state. This is a “rough draft” of your final presentation. Submit your slides through the homework form (optionally, share on your blog).
09.
Due in before class in week 11, session 1 (2024-04-09).
Research paper: At this point, you should have enough information to write rough drafts of a “Results” or “Evaluation” section of your research paper. Write this rough draft and bring it to class in week 11 session 1 (2024-04-09)
By using icons on the homepage and making them trigger the slideshow gallery with a hover, people can quickly get an idea of the projects you're into at just a glance.
On the "All Projects Page," you'll find tabs that show the themes projects are associated with. Users can click on these tabs to select multiple themes and explore projects related to various topics or mediums.
Given that my projects often aim to bridge different mediums and topics together, simply by glancing at the icons atop the featured image of a project, users can swiftly grasp the essence of what the project entails.
01. Portfolio
Understand the purpose, goals and methods of the capstone project
Create a portfolio of existing work
03. Sketches & Prototypes
-
Document your prototypes on your blog.
-
Continue work on concept prototypes
-
Learn techniques for literature reviews and annotated bibliographies
-
Understand research paper organization, format and requirements
XR City Navigation
This is an interactive web sketch prototype of envisioning the nvigation inside the city
A. Concept Re-Organize
As I dive deeper into the 4 potential topics of my capstone, I came to realize that these topics takes different forms acting as different components of a project.
-
Digitality of 3D with Market”: Practice
-
“Hyper-reality communication: Democratizing Internet”: Topic
-
"Smart Cities: Urbanism with Digital Twins": Medium
-
“3D Projection: Little Place”: Final Presentation
01 Digitality of 3D in Market
The first topic “Digitality of 3D with Market” is focusing value of digital 3D product in current market where XR services are blooming and slowly becoming the mainstream consumption of visual information and interactions. This notion is the idea of “digitality of 3d” is the act of practicing. Through the deep look and practicing of creating digital 3d product and critically analyzing it through the lens of business.
02 Hyper-reality Communication: Democratizing Internet
Secondly, the “Hyper-reality communication: Democratizing Internet” is the topic which the insights can only be gathered by observation from third person point of view or big picture analysis. In order to observe the power dynamic of hyper reality communication, building a simulation model of internet communication flow emerges as the optimal approach. Furthermore, this type social network is considered as a new form of urbanism where information and resources are gathered as small units in the internet. These units extend beyond the digital realm; indeed, owing to the nature of XR technology, the amalgamation of small units within the internet also relies on physical location. This delineates the social network as an urbanism where information and resources are geolocated alongside the internet.
03 Smart Cities: Urbanism with Digital Twins
Thirdly, "Smart Cities: Urbanism with Digital Twins" serves as the medium. Digital twin technology entails replicating and continually updating a clone of a physical space. It is typically used to study and manage the intricate relationships within a given space. If considering the market demand, usage, and consumption of XR as urban activities, mapping the city's relationships onto digital twins to host simulations will to be an ideal application.
04 3D Projection: Little Place
Eventually, “3D Projection: Little Place” is the final presentation. The Little Place device serves as a platform to exhibit the physical manifestation of the digital world. By performing 3D projection, digital content and live digitally not restrained by any physical limitations but in fact still exists in the physical world. As the project primarily focuses on simulation and observation, the Little Place resembles a small container, akin to a scientific vessel, under a controlled environment to allow the examination of specific components within a broader network.
Due at the beginning of week 5 (2024-02-20), but we’ll do check-ins in week 4. Using the techniques discussed in class, make lo-fi prototypes related to at least three concept abstracts from last week. How did you identify the underlying questions or problems that could be best addressed through prototyping a particular idea? What did it feel like to work on the prototype of one concept, versus another concept? What did you learn about the concept in the process of prototyping? Document your prototypes on your blog.
City: NYC Data
Using NYC REaltime Data to build the foundation of a city
-> fixed structure of a system
Urban Activities:
XR Food, Clothes, Living, and Commute Systems
Re-imagining the future of food, wearables, living, and transportation to design their XR interactions as the activities in an urban space.
Avatar: User Input Vehicle
Avatars created by user internet personality. It also functions as the vehicles of how the XR activities interact with each other being the POV of how the audiences to view
B. Project Overview
This project is storytelling base simulation and observation tool for XR business, envisioning the future of XR world. By imagining the future XR world, this project builds a simulation of the XR market in the next 50 years to explore the to inform decision-making for XR companies, test product viability, study human behavior toward XR adoption, and reimagine the XR landscape. Integrating reality technology, internet urbanism, business analysis, and studies of human-computer behavior, the project sheds light on the evolving implications of the XR ecosystem.
GOAL
-
for XR companies to decided what product to develop
-
for XR companies to test their developing product’s survival rate
-
studying human behavior towards the popularization of XR
-
re-imagining the XR world
TOPIC: Market Survival Rate of XR Business
-
Digital Twin City: Future NYC
-
XR Urban Activities: Food, Clothes, Living, and Transportation
-
Avatar User Input Vehicle
-
XR Device: 3D Little Place
City Simulation Overall Structure | Interactive Web Prototype: Hover over the building, colorful paths, and red people to see the insights and how they are related to each other
Hover & Drag
B1. Digital Twin City: Future NYC
//Images on the right is the digital twin city I built in Unreal Engine//
The city serves as the foundation of this project. My goal is to construct a digital twin city based on New York City data, replicating its foundational structures and while envision its future over the next 50 years in the context of XR technology. The approach acknowledges that XR is a almost parasite like services that is based on physical location (reality). Therefore, the mixed “X” in XR isn't meant to replace reality (R) but rather to enhance physical spaces. Departing from the one potential major event of the city is the usage of XR will cause land usage and makes people’s livelihood services grow. With this notion, the city’s function is redesigned based on the 1967 vision of the architect Moshe Sadie on the coalesce into small units of people living, providing a solution of affordable and efficient housing while maintaining a sense of community and green space within an urban environment. The city’s presentation is drawing inspiration from the director Naohisa Inoue’s style in Iblard Jikan, where the notion of "magic" is commonplace in a kingdom. XR technology symbolizes this magic, where people create everything with their imagination. In this enchanting world, floating islands and asteroids adorn the sky, while streets and houses are adorned with vibrant, nameless plants. Travel occurs via vehicles that effortlessly float in the air. Magic acting as XR with the hope ofcreating a dreamlike fantasy paradise where humans, animals, elves coexist harmoniously.
More Information:
Replicate the city base
-
Using NYC Data to build the foundation of a city -> fixed structure of a system
-
Habitat 67 by Safdiw Architects
-
X is not replacing R → X is parasit on R → X is enhancing R
-
the story of imposter and oil paint aesthetics
Base Events
-
land usage decay/shrink → people's livelihood usage grow
-
what is the office situation
-
-
emphasis on transportation
-
point to line to range used to be devloped like this TOD but now it might decay/shrinking to range → line → point (MRT station)
-
what is shopping 逛街 (before is shopping anf eatin now eating is more)
-
only thin you need to go out is birth and death → hospital
-
-
resturants are becoming popular
-
social and eating aspect
-
B2. Urban XR Activities & Business Study
The XR urban activities is the second component in this project. It serves the purpose of re-imagining the future of food, wearables, living, and transportation and designing their XR interactions as the activities in an urban space. Each XR services in the space will have their initial data of their XR attributes - level of comfort, visual, time saving, user growth, and emotional attachment. These attributes will influence whether if the avatars (people living in the city) decide to use the XR service or not. This allow the user (XR companies) to decided what kind of XR product is more popular in the market. There will be two kinds of XR activities: premade and input. The premade are the ones that is built in the city for demo reasons. The input XR services are the ones where the XR companies can upload their XR services to the system and see their product performance and survival rate in the market. The XR attributes are derived from the XR study I conducted last semester, which is presented in the two Google Slide embeds on this page. Based on the market study, I will also create a test for XR companies to upload their own XR services into my city and define their attributes.
B3. Avatar User Input
Market Consumption Vehicle
The avatars is the third component of this project. Avatars are the people who live in this virtual city. They function as the vehicles of how the XR usage is being consumed or not consumed in the market.
Each avatar is generated through an internet personality test, endowing them with customer personality attributes such as minimalist seekers, conservative homebodies, balanced optimists, cautious planners, impulsive spenders, empowered activists, undaunted strivers, and secure traditionalists.
By assessing the compatibility between their consumer type attributes and XR features, we can discern patterns in market behavior. comparing the compatibility of of their consumer type attributes and XR attributes, we can start to see the pattern of the market behaviors.
// on the left is the structure of the consumer personality test//
B4. Morphing XR Space: 3D Little Place
“3D Projection: Little Place” is fourth components of the project as well as the final presentation. The Little Place device serves as a platform to exhibit the physical manifestation of the digital world. The user can choose which avatar’s life they want to see. The little place device will start to project the camera image following that certain avatar onto the shape changing tube matrix on the 2 walls and floor. By performing 3D projection, digital content and live digitally not restrained by any physical limitations but in fact still exists in the physical world. As the project primarily focuses on simulation and observation, the Little Place resembles a small container, akin to a scientific vessel, under a controlled environment to allow the examination of specific components within a broader network.
POV
Avatars created by user internet personality. It also functions as the vehicles of how the XR activities interact with each other being the POV of how the audiences to view
C. Paper & Plastic Prototypes
I made two paper prototypes to test different key ideas. The first prototype is the white isometric cube. It demonstrates the overview of how the set up of the little place would be. The cylinder like tubes attached onto the 2 walls and floor will be mechanically move outward and inward to give an impression of morphing. According to what is projected onto this tube matrix, the tubes are aligned with the object depth of the projected image.
04. Annotated Bibliographies
-
Prepare an annotated bibliography with at least eight entries for your (hopefully now finalized?) project idea.
-
The bibliography should include papers, articles, artworks or other media that are relevant to your project.
-
Each item in the bibliography should have an annotation describing that item’s relevance to the project and how the reference has been or will be useful in executing the project.
-
Additionally, rewrite the abstract for this project idea and compose a rough draft of your research paper’s “introduction” section based on the project idea. Bring copies of all of these to class.
06. Production Plan
Develop a production timeline for your project. The production timeline should take the following into account:
(1) work you’ve already done on the project;
(2) material and logistics (e.g., estimated arrival times for parts and fabricated components; scheduling time in rehearsal facilities or studios, etc.);
(3) your own availability (how much time can you reasonably spend working on your project?). We’ll discuss techniques and formats for developing this timeline in class. Post your timeline to your blog.
Digitality of 3D (Materiality)
In recent years, there has been a significant shift in media, transitioning from traditional 2D formats to increasingly immersive and dynamic 3D experiences. However, this transformation reveals a scenario where technology is outpacing market readiness. The implementation of 3D hasn't always been successful in the market, resulting in losses for many companies investing in VR and AR products. This discrepancy raises questions about the ideal content for the 3D format and its effectiveness compared to 2D in conveying information efficiently.
Why 3D? Is it true that 3D offers a more intuitive mode of communication, considering our existence in a three-dimensional material world? To explore these questions, we can turn to pioneers like Pixar Animation Studios, where the development of 3D graphics technology revolutionized the landscape of CG films.
This project delves into the materiality of 3D and compares its performance with the digital world we currently are in. This is a study bridging computer graphics knowledge and market analysis for business orientated researchers on Extended Reality (XR) topics.
Hyper-Reality Communication:
Democratizing Internet
In 1981, Baudrillard introduced the theory of hyperreality, defining it as "the generation by models of a real without origin." Initially controversial, this concept has since become a pervasive aspect of contemporary life. The internet has significantly expanded people's interactions and exposure to information, particularly in real-time. Telecommunication has progressed to the point where activities can occur simultaneously in different locations via the cloud.
While such communication appears to distribute power, it ultimately contributes to solidifying decentralized networks, as proposed by Tung-Hui Hu in his book "The Shape of Network" (2015). This project delves into the key factors necessary for developing communication networks in a future where decentralization replaces reliance on monopolistic big tech companies. The utilization of AR applications can serve as a significant technological vehicle for synchronizing activities and geospatial data, and potentially enabling more intricate tasks such as trading.
3D Projections in Real Space
Influenced by decades of science fiction movies, most people’s imagination of the future is holographic devices all over urban spaces. The central idea behind this vision is the notion of bringing digital content into the real world. Information can live digitally not following the material world restrictions but exists and interactable in the physical spaces. In the past 5 years, many products has launched aim to replicate similar user experiences such as 3D phones/tablets, XR glasses, 3D glasses free billboards, and VR devices. However, not all have been successful, often lacking in accessibility, comfort, or content creation capabilities. This project attempts to look into the key components to trick human eyes into seeing three dimension visual content and develop projection systems that enables users to transform a corner of their physical space into a visually distinct environment using lasers and digital content. The minimum viable product is a hologram device called “Little Place” that transforms user’s table into a virtual friend's room. The LLM driven avatar engages in conversations, prioritizing caring validation for those who work from home and might feel unappreciated and overlooked at their jobs.
02. Concept Abstract
Based on our in-class exercises, compose three to five speculative abstracts for Capstone project ideas. Write these abstracts as though you have already completed the project in question. Each abstract should be 100–200 words and accompanied by at least one reference to a project, paper, article, talk, or other piece of media that is adjacent to the project idea. For each abstract, identify audiences, venues, publications, etc. that might be a good “fit.” Make a post on your blog with these abstracts.
Reference:
-
The Pixar Story (Documentary about their process of creating 3D graphics engine)
-
What is New Media (Manovich, 2001)
Experiment:
Video documentation narrating the relations between 3D graphics development and marketing research (target audience interviews)
Reference:
-
The Shape of Network (Hu, 2015)
-
On Distributed Communications Networks (Baran, 1962)
-
A Material History of Bits (2011)
Experiment:
-
Geospatial AR app for data trading
Reference:
-
On Software or the Persistence of Visual Knowledge (Chun 2005)
Experiments:
-
Little Place: A 3D holographic device transforms your table into a virtual friend's room. The LLM driven avatar engages in conversations, prioritizing caring validation for those who work from home and might feel unappreciated and overlooked at their jobs.
-
Real Space: A projection system that enables users to transform a corner of their physical space into a visually distinct environment using lasers and digital content.
Reference:
-
Toyota Woven City Case Study
-
How Can Autonomous and Connected Vehicles, Electromobility, BRT, Hyperloop, Shared Use Mobility and Mobility-As-A-Service Shape Transport Futures for the Context of Smart Cities? (Nikitas, 2017)
-
The Line (Saudi Arabia)
Experiments:
Smart Cities: Urbanism with Digital Twins
In our current world, at the intersection of sustainable transportation, urbanization, and climate change, we are confronted with a pressing question: how can we more efficiently distribute resources and reduce waste through data sharing to address issues such as an aging population, gentrification, and urban density? Through the examination of initiatives like the world’s first smart city - Toyota Woven City - and the upcoming self-sustained city complex in Saudi Arabia known as the Line, this project seeks to explore potential solutions.
Central to this exploration is the investigation of the coexistence of micro-mobility and public transport systems. This entails examining mobility flow, economic and social structures, and redefining suburban landscapes. The analysis also encompasses the potential of shared autonomous mobility on demand (AMoD) and electric scooters as viable alternatives to private car ownership.
MIT research indicates that AMoD could improve efficiency when competing with public transit, albeit with varied impacts on passengers. Additionally, electric scooters emerge as cost-effective options, capitalizing on advancements in autonomous technology. Through this multifaceted examination, we aim to shed light on innovative approaches to transportation and urban development that can address the complex challenges of our modern world.
Trailer
Explains why do "If XR"
Overview of the final project
Coming Soon
07. Production Update
I devoted my whole Spring Break to Capstone because of the scale of the project. I have lots of progress but also didn't expect so much workload. Every tiny decisions in the world takes up a long time. Most projects out there for future city envisioning are usually on the street and only seeing the outside of the buildings and not emphasizing on the activities. My project is more about the XR activities and its system in the world, so I have a lot more work to do and things become very detailed. I've been trying different ways to figure out how to maintain both.
What I did:
🔵 World Building
- static (road and building systems): this look me a long time to structure since transportation system becomes complicated when there are many different volumes and function in the city
- procedural (programming spaces that follows from my XR city design principles): I wrote a procedural code to program houses and pathways inside a building.
🔵 Activity Programming
designing XR activities in each place (ex. things happening in a super market from selecting an object to paying for it)
🔵 NPC Management
It took me a long time to figure out how to manage such a large amount of smart AI people walking in the city and having their own desires and deciding where they go etc. I finally figured out and have all of them walk on street instead of grass field or get stuck somewhere
🔵 Holograms Made
Hologram materials are hard to be visible since it is hard to radiate light under sunlight in day time. I figured out a wya to program hologram shader to let it be visible and has the digital look. I made two methods: niagara ad video planes
🔵 Paint Shader
I wanted to make the look of the city humane. Therefore I wanted to have the city look like being painted. I wrote a atmosphere shader that allows every pixel to render as paint brush texture running in runtime.
