> For the complete documentation index, see [llms.txt](https://stage-precision.gitbook.io/grid/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://stage-precision.gitbook.io/grid/spatial-interactive-experiences/applications/depth-images-for-external-processing.md). # Depth Images for External Processing Not every spatial application is best represented by clusters, IDs, and 3D positions. Sometimes the receiving system needs a visual representation of the spatial environment instead. Grid Studio can render the processed point cloud into a real-time image and expose the result as an **Image Provider**. This makes it possible to use the same spatial sensor environment for external applications such as: * Real-time visual effects * Interactive content * Silhouette and mask generation * Blob detection * Custom touch detection * Image-based interaction * Additional computer vision processing The resulting image can then be sent to external systems through media outputs such as **NDI** or **SDI**. This page focuses on designing image-based workflows. For the complete step-by-step configuration, see **Generate Depth Images** in the Project Guide. *** ### Choose Between Tracking Data and Image Data The first decision is whether the receiving system actually needs structured tracked objects. Cluster output provides information such as: * Position * Scale * ID * Individual tracked objects An image provides a completely different representation. Instead of asking: **Where is person 17?** the receiving application receives a spatial image and can decide for itself what that image means. This is useful when the external system wants to perform its own: * Segmentation * Blob analysis * Thresholding * Mask generation * Visual effects * Custom interaction logic A useful distinction is: **Cluster Output** → Grid Studio performs the object analysis. **Image Output** → Grid Studio provides the spatial representation and the receiving system performs additional analysis. *** ### Use Both When Appropriate Image output and cluster output do not need to be mutually exclusive. The same project can use tracked clusters for one purpose while simultaneously generating an image for another. For example: **Point Cloud Volume** → Final People Clusters\ → OSC Position Output while the same spatial environment also provides: **Point Cloud Volume** → Image Rendering\ → Depth Image\ → NDI / SDI\ → Real-Time Visual System This allows one sensor environment to serve both structured tracking and image-based applications. *** ### The Image Is a Projection of the Point Cloud The generated image should not be treated as if it were a conventional RGB camera image. It is created by projecting spatial point data into an image. Its appearance therefore depends on factors such as: * Sensor point density * Sensor viewing angles * Occlusion * Distance * Input filtering * Projection Camera * Point Size * Erode / Dilate * Smoothing Sparse spatial measurements may therefore produce gaps in the image. Areas with high point density may appear much more complete. This is expected behavior for a point-cloud-based image. {% hint style="info" %} The goal is not necessarily to create a visually perfect image. The goal is to create the representation that provides the most useful information for the receiving system. {% endhint %} *** ### Decide Where Image Rendering Happens in the Pipeline **Image Rendering** is part of the Point Cloud Volume's **Input Filter** chain. Its position in that chain matters. The Image Rendering filter receives the point cloud as it exists at that stage of processing. For example: **Tracking Area** → Floor Removal\ → Noise Filtering\ → Image Rendering produces an image after those cleanup operations have already been applied. If additional Input Filters are placed after Image Rendering, those later filters do not contribute to that rendered representation. This makes the position of Image Rendering an application-design decision rather than simply a visual setting. In most cases, placing it near the end of the Input Filter chain provides a useful cleaned representation. *** ### Do Not Filter More Than the Image Requires An external image-processing system may benefit from different point-cloud characteristics than the clustering pipeline. For example, a filter that improves people clustering may remove small or sparse measurements that would still be useful for an external image effect. Evaluate the image itself when deciding which processing should happen before Image Rendering. {% hint style="warning" %} Point-cloud filters can be computationally expensive. Do not add additional filters only to make the image appear cleaner unless they produce a meaningful improvement for the final application. Millions of spatial points may need to be evaluated every frame, so unnecessary processing can reduce real-time performance. {% endhint %} *** ### Design the Projection Around the Application The projection determines how the 3D spatial environment becomes a 2D image. The Project Guide uses **Camera Projection** because it provides an intuitive and flexible way to define that view. The Camera determines: * Position * Rotation * Perspective or Orthographic projection * Resolution * Near Clipping * Far Clipping Think of the Camera as the coordinate system of the generated image. Its placement should therefore be based on how the receiving application wants to interpret the physical environment. *** ### Use Orthographic Projection for Surface-Like Areas An Orthographic Camera can be useful when perspective distortion is undesirable. Typical examples include: * Interactive floors * Stage areas * Overhead occupancy images * Rectangular interaction regions * Image analysis that expects consistent spatial scale A top-down orthographic view can provide a simple spatial representation where objects maintain a more consistent image scale across the area. For example: **Tracking Area** → Top-Down Orthographic Camera\ → Depth Image\ → External Blob Analysis This can make image-based processing easier because the representation corresponds closely to the physical floor plane. *** ### Use Perspective Projection When Viewpoint Matters Perspective projection can be useful when the generated image should represent the environment from a specific viewpoint. Examples include: * Front-facing interaction * Projection-aligned effects * Camera-like depth representations * Visual compositing workflows Objects further from the projection Camera naturally occupy a different image area than objects close to it. Whether this is desirable depends on how the receiving application uses the image. *** ### Use the Camera View While Designing The Image Rendering Camera is not only a technical output device. It is also a useful way to design the spatial region seen by the external application. Switching the Viewport into the Camera view makes it possible to navigate the scene while directly adjusting the Camera framing. Keeping both: * The 3D Viewport * The Image Rendering Viewer visible at the same time makes it much easier to understand the relationship between physical point data and the generated image. *** ### Define a Useful Depth Range The Camera's **Near Clipping** and **Far Clipping** define the useful depth range of the projection. Do not automatically include the entire possible environment. For example, if the application only cares about people within a limited interaction depth, a much smaller range may provide a more useful result. A carefully chosen range helps: * Remove irrelevant foreground or background information * Focus the image on the interaction * Make depth differences more useful to the receiving system Start by defining the physical range required by the application before adjusting the visual appearance of the individual points. *** ### Tune Point Coverage for the Receiving System Point-cloud measurements do not naturally create a completely filled image. The Image Rendering settings can be used to adjust how those points are represented. #### Point Size **Point Size** controls how much image area each rendered point covers. Larger values can help close gaps between sparse measurements. However, very large values can also merge nearby structures and reduce spatial detail. #### Erode / Dilate **Erode / Dilate** can shrink or expand the rendered regions. This can be useful for: * Closing small gaps * Expanding silhouettes * Removing small edges * Adjusting masks for downstream processing #### Smooth **Smooth** can soften the resulting representation. This may help an external visual system, but excessive smoothing can also remove useful detail. {% hint style="success" %} Tune the depth range first. Then adjust **Point Size**, followed by **Erode / Dilate**, and finally **Smooth** where required. {% endhint %} These controls modify the image representation. They do not create additional physical sensor measurements. *** ### Design for the Processing That Comes Next The best image settings depend heavily on what happens after the image leaves Grid Studio. #### Visual Effects For reactive visual content, a slightly smoother and more continuous silhouette may be preferable to preserving every individual point. #### Blob Detection Blob analysis may benefit from larger Point Size or dilation so that a person forms one connected region. #### Touch Analysis A touch-processing system may instead need small spatial details and therefore require less smoothing or dilation. #### Masks When the image is used as a mask, closing small holes may be more important than maintaining exact point boundaries. The correct image is the one that makes the next processing stage reliable. *** ### Use Different Images for Different Applications A Point Cloud Volume can contain multiple **Image Rendering** filters. Each one can use a different projection Camera and produce its own Image Provider. This makes it possible to create several spatial views from the same environment. For example: **Global Point Cloud Volume** → Top Camera\ → Orthographic Depth Image\ → Floor Interaction System → Front Camera\ → Perspective Depth Image\ → Projection System → Side Camera\ → Depth Image\ → External Analysis The sensor and tracking environment remain shared while each external application receives the projection that best suits its purpose. *** ### Consider the Performance Cost Image generation adds another real-time processing stage to the spatial pipeline. The total cost depends on the complete project, including: * Number of sensor points * Input Filters * Number of Image Rendering filters * Output resolution * Number of media outputs * Other tracking and clustering processing Avoid generating additional views simply because they are available. Each Image Rendering output should serve a clear application requirement. If several external systems can use the same image representation, sharing that output may be preferable to rendering several equivalent images. *** ### Separate Image Generation from Media Transport There are two distinct parts of the workflow: **Image Rendering** creates the image inside Grid Studio. **Media Input / Sender** transports that Image Provider to another system. Conceptually: **Point Cloud Volume** → Image Rendering\ → Image Provider\ → Media Input\ → Sender\ → NDI / SDI\ → External System This separation is useful when troubleshooting. If the image already looks incorrect in the Grid Studio Viewer, investigate the spatial data, projection, or Image Rendering settings. If the image looks correct in the Viewer but not in the receiving system, investigate the media output path instead. *** ### Verify the Final Image in the Receiving System Always evaluate the output where it is actually used. Keep the Grid Studio Viewer and the external application visible simultaneously where possible. Move through the complete interaction area and compare: **Physical movement** → **Grid Studio point cloud** → **Rendered image** → **Received image** → **Final external processing** This makes it much easier to identify where a problem begins. *** ### Typical Architecture: Reactive Floor A large tracking environment can generate a dedicated overhead image for an interactive floor. **Spatial Sensors** → Point Cloud Volume\ → Processed Point Cloud\ → Orthographic Image Rendering\ → Image Provider\ → NDI / SDI\ → Real-Time Content System The external application can then use the image for: * Blob detection * Masks * Particle interaction * Generative content No individual cluster positions are required if the visual system only needs the spatial image. *** ### Typical Architecture: Projection Interaction A projection-oriented system may use a Camera positioned to match the required content view. **Processed Point Cloud** → Projection Camera\ → Depth Image\ → Media Output\ → Projection Content System The receiving system can then use depth or silhouettes to modify visual content relative to people or objects in the environment. *** ### Typical Architecture: Parallel Tracking and Image Processing Some installations benefit from both approaches. **Point Cloud Volume** → Clustering\ → Tracked People\ → OSC and simultaneously: → Image Rendering\ → Depth Image\ → NDI\ → TouchDesigner / Notch The structured tracking data can control application logic while the image provides richer information for visual processing. *** ### Common Failure Patterns #### The Image Contains Large Gaps Check: * Sensor point density * Camera distance and projection * Point Size * Sensor coverage * Whether useful points were removed by Input Filters Do not automatically increase Point Size until the physical sensor coverage has been evaluated. *** #### People Disappear at Certain Distances Check: * Camera Near Clipping * Camera Far Clipping * Sensor coverage * Point density at that distance * Input Filters Compare the Image Rendering Viewer with the 3D point cloud to determine whether the measurements exist before rendering. *** #### The Image Looks Good but External Detection Is Poor A visually pleasing image is not necessarily ideal for machine or interaction processing. Evaluate what the receiving algorithm actually needs. Try adjusting: * Point Size * Erode / Dilate * Smooth * Depth range * Projection based on the final detection result rather than appearance alone. *** #### The Image Is Correct in Grid but Incorrect Externally If the Image Rendering Viewer is correct, investigate the output stage. Check the Media Input, selected Image Provider, Sender configuration, device, and receiving system. Do not change the point-cloud processing to solve a transport problem. *** #### Rendering Another Camera Causes Performance Problems Every additional image can add processing and output cost. Verify that the additional projection is actually required. Also inspect the complete Point Cloud Volume filter chain and remove processing that does not meaningfully improve either clustering or the generated image. *** ### Use the Representation That Best Fits the Application Cluster data and depth images solve different problems. Use clusters when the application needs structured objects: **Object → ID → Position → Scale** Use an image when the application needs a spatial representation: **Environment → Projection → Pixels → External Processing** And use both when the installation benefits from both representations. The goal is not to force every spatial interaction into the same processing model. Grid Studio can provide the structured tracking result where appropriate while also making the underlying spatial environment available to external real-time systems. For the complete setup process, see **Generate Depth Images** in the Project Guide. For detailed information about Image Rendering projection modes, Point Color modes, rendering parameters, and Image Providers, see the corresponding **Image Output Reference**.