Spatial awareness is at the core of Augmented Reality

Spatial awareness

Spatial awareness is the capability of the AR device to understand and interpret the user's physical environment in real time. The key aspects of it include mapping the environment around the user, tracking the user's position, and controlling interactions of the virtual and physical objects in the scene.

The following is true for Microsoft Hololens 2 however other advanced headsets and smartphones work similarly.

For mapping, Hololens builds mesh representing the geometry of the environment. It identifies walls, floors, other surfaces, and physical objects using its depth sensing (ToF) camera. The size of the polygons varies from half an inch to several inches depending on the surface features, lighting, and mood of the pixels on that day. This is to say that this mesh is not surgically precise but sufficient to wrap surroundings with 2-3 inch accuracy.

The user's position inside the mesh is tracked with the in-built IMU (accelerometer + gyroscope + magnetometer). This informs the system where he or she is within the environment.

Finally, a holographic image displayed by the headset for the user is configured using the knowledge of mapping and tracking. A notable example is the occlusion or the ability to overlay holographic content on real-world objects accurately.

Refer to MRTK documentation or watch the video from Microsoft to learn what is under the hood of spatial awareness in detail.

Capabilities of the Apple Vision Pro headset that entered the market in 2024 are still to be discovered.

Impact on handling information

Availability of the spatially aware devices on the shop floor can profoundly change the way we create, label, and access technical information. Think about the data in your ERP, PLM, or knowledge databases for a moment, what's in there? There are mostly tables with process data, documents with work instructions, and engineering drawings.

Inventory, products, equipment as well as job cards, work orders, and production stage reports are indexed by serial numbers. The same applies to the Standard Operating Procedures (SOPs) such as maintenance manuals and repair instructions — technicians find them by the S/N or name of the process, machine, or batch.

The attribute that is downplayed is location although every item in the factory has one. Of course, there are some proxies for location and you know that "compressor X is in hangar Y" or "palette A is in warehouse B". However spatial computing can bring the localization of information to a whole new level. You could leverage [x, y, z] coordinates of every point of space in your factory, warehouse, or workshop to a) simplify search of already existing spatial data and b) bring more data into spatial context.

Benefits for the factory workers

First of all, you should make sure you are familiar with the generic use cases of industrial AR. Next, explore how technicians use and create information within the spatial context.

Use information

Imagine the convenience of accessing information at the point of use without the need to search for it! Imagine:

• receiving the aircraft for C-check and having historical information on every component and subsystem attached to them;
• going to repair the CNC machine and finding the maintenance manual already waiting for you next to it;
• inspecting painted surfaces for cracks and observing reference pictures and tolerance limits specific to the component in focus.

Contextual work instructions is the right term to use when referring to this capability.

Create information

You can "keep" data at the point where you "found" it. Think about performing an inspection which is part of pretty much every industrial process. Say you are looking for corrosion, leakage, misplaced material, or out-of-range sensor reading. Once found, you can take a picture, describe the problem, and pin it to the location — all through an AR headset.

Using the power of [x, y, z] coordinates you could reference all findings of your inspections to the very place where they happened. Someone will need to come back and fix the issues after all. Why not simplify the work for them and not create a spatial quality record that keeps evidence of the initial check alongside the timestamp, inspector name, and hyperlinks to the servicing documentation?

Going one step further, you could engage Artificial Intelligence to help you perform visual inspection faster and more accurately.

Conclusion

Location is one of the most intuitive and obvious parameters of physical objects. Spatial awareness of the AR devices allows engaging it much more actively for saving time spent on creating, handling, and using technical information. This increases the labor productivity of the technical personnel, improves the utilization rates of equipment, and increases factory throughput as a result.

Please note that we wrote this article without saying digital twin even a single time.