Splatchemy!: Transforming the real world with magic and creativity

Developer Jessica Sheng challenges players to combine 3D objects and transform a simple park gazebo into a magical, plant-filled sanctuary in her latest creation, Splatchemy! Using a Gaussian splat captured with Niantic Scaniverse and imported into Niantic Studio, Jessica crafted a WebXR experience that blends innovation with play. In Splatchemy!, players combine objects to unlock new creations, transforming a plain park picnic area into a magical, plant-filled environment. Jessica’s project highlights the versatility of Niantic Studio, showcasing how its features can be used to create captivating and interactive experiences grounded in the real world.

We had the chance to dive into Jessica's development process and learn more about her inspiration, challenges, and the steps she took to bring Splatchemy! to life.

What inspired you to create Splatchemy!?
I was inspired by Little Alchemy, a game that encourages experimentation and creativity without set objectives. I wanted to capture that same sense of curiosity while showcasing the power of Gaussian splats. Turning a plain park picnic table into a magical, plant-filled space felt like the perfect way to combine creativity with interactivity.

How did Niantic Studio help bring your vision for this project to life? Were there specific tools or features that were essential?
The core features that made Splatchemy! possible include:

• Gaussian splats: These were essential for defining interaction areas and creating an engaging, real-world environment. Using Scaniverse to export splat files directly into Niantic Studio made the process seamless.
• Entity-Component System (ECS): Niantic Studio’s modular framework allowed me to define reusable components for merging objects, handling interactions, and spawning entities.
• Web-based development: The browser-based approach of Niantic Studio accelerated development and eliminated platform-specific constraints.
• Three.js integration: Compatibility with Three.js enabled a smooth integration of 3D assets and raycasting for object interactions.

Were there any challenges along the way? If so, how did you overcome them?

As Niantic Studio is still in beta, some features are still being developed, which presents unique opportunities to find creative workarounds.

Prefab System: Currently, Niantic Studio doesn’t include a built-in prefab system, which means I couldn’t easily reuse preconfigured objects. To spawn a predefined object, I developed a workaround: iterating through all components of an entity and applying them to a new entity, effectively cloning the object’s components and properties. This approach allowed me to dynamically spawn objects with consistent configurations, and I’m excited to see how Niantic Studio evolves in this area.

Built-In Tagging System: While Niantic Studio doesn’t yet have a built-in way to tag or categorize entities for interaction purposes, I implemented a custom tagging system using the MergeableComponent schema (level) to track object states and determine valid combinations during merging. This worked well for my project and provided flexibility in defining my own interaction logic.

UI System Complexity: The current UI system offers robust capabilities but requires more manual effort for dynamic elements like pop-up messages or buttons. For example, the inspector properties closely resemble raw HTML/CSS code, which meant I had to build and iterate to preview layouts. To streamline the process, I injected HTML and CSS directly using JavaScript, giving me more control over styling and functionality while integrating seamlessly with the game logic.

These challenges allowed me to get creative with solutions while showcasing the flexibility and potential of Niantic Studio, and I’m excited to see how the platform continues to grow and support developers’ needs.

What advice would you give other developers interested in building real-world games or WebAR experiences with Niantic Studio?

• Start small: Focus on implementing one core mechanic before expanding your project.
• Leverage ECS: Use the Entity-Component System to modularize your project for easier debugging and reusability.
• Custom solutions: Be prepared to build workarounds for missing features like prefabs or tagging systems.
• Iterate often: Regularly test your project on different devices to ensure consistent performance.

Where do you see the future of AR going, and how does Niantic Studio fit into that vision?
AR has immense potential as the most accessible form of XR technology for everyday users. I believe technology is best used to enhance our lives and foster human-to-human connection, and AR is the most likely to achieve that within the realm of XR. Unlike VR, which isolates users from their surroundings, AR enhances reality by overlaying digital content onto real-world environments—making it inherently social and immersive.

Niantic Studio is well-positioned to drive this vision forward because it is entirely web-based. I believe one of the biggest barriers in XR today is accessibility—many tools require specialized hardware or software expertise—but Niantic Studio lowers that barrier by enabling developers to create experiences directly in the browser. This makes it easier for creators from diverse backgrounds to experiment with AR/3D projects and bring their ideas to life. Additionally, almost everyone has access to the web. This will make AR/3D projects more accessible to the public as well, removing the need for a dedicated headset for people just to experience XR applications. Niantic Studio can play a pivotal role in integrating XR technologies into our daily lives while connecting people through shared experiences.

Top Features in Splatchemy!:

• Dynamic object merging: Combine everyday objects like water, cups, and soil to create potted plants, fairy lights, and more.
• Immersive Gaussian splat environment: Transform a park gazebo into a magical space by interacting with splats.
• Interactive UI: Players are guided through the experience with intuitive messages and score tracking, enhancing engagement.
• Custom collision logic: Implemented to handle object interactions and define valid combinations for merging.
  
  
  

Pro Tips for Aspiring Developers: 

• Define Core Mechanics and POCs:
  Before starting development, I outlined the core mechanics of merging objects and transforming the environment. This helped me focus on building reusable components (MergeableComponent, MergeHandler) that could handle these interactions efficiently. I started off with some proof of concepts, making sure that some features were simply viable and that this game could be made. Initially, my idea was to create elements and let the users apply them to affect the material of the splat. However, the way splats are rendered make that impossible, so I pivoted to adding various 3D objects to the scene to enhance the splat.
• Modular Component Design:
  Using Niantic Studio’s ECS framework, I created modular components for merging logic, UI interactions, and object tagging. This made it easier to iterate on specific features without affecting unrelated parts of the project.
• Custom cloning system:
  Since there is not yet a prefab system, I implemented a component cloning function that dynamically copies components from one entity to another. This allowed me to spawn new objects based on valid combinations.
• Raycasting for interactions:
  To detect clicks or taps on objects, I used Three.js raycasting combined with Niantic Studio’s event listeners. This enabled precise interaction detection for merging mechanics.
• Iterative testing & debugging:
  Throughout development, each feature was tested ASAP to make sure it didn't break anything. Debugging tools like console logs helped identify issues with component schemas or event handling early on.
  

Ready to explore the magic of Splatchemy!?
Dive into Jessica’s development process with her step-by-step tutorial and learn how you can create your own interactive Web3D game.

Looking to create your own project? Clone Jessica's sample projects to get started:

• Merge Objects Sample Project: Learn how to combine objects and trigger new interactions.
• Detect Object Type On Tap Sample Project: Understand how to use Gaussian splats to define unique interaction areas in your scene.

Jessica’s Splatchemy! is a testament to what’s possible with Niantic Studio. Let her journey inspire your next project—happy building!