Creating 3D models of reality is about much more than just shapes and colors. In October 2022, we had a chance to talk to Dr. Ludwig Friedman of BMW Group about his initiative “OpenMATERIAL”.
The key question driving Ludwig is to define the right means to develop, test and validate automated driving functions. As we are all aware, real-world testing alone will not do the trick, and quite some testing has to be done in simulated environments.
This simulation can be done for perception, planning, decision, and control – just to name the main stages of an automated driving stack. And it is in the perception stage where accurate 3d environments will be needed most.
But how accurate is accurate enough and who is going to acquire all the relevant data and create sufficiently attributed 3d models? And what attributes are needed for what kind of simulation? Can this effort be handled by any single company in a proprietary way, or would it be better to share the effort in a community?
These are the questions that puzzled Ludwig and led to his answer: the OpenMATERIAL initiative. It is a lively interest group of OEMs, sensor developers and universities.
The general idea behind this open-source project is that relevant 3d data representations of reality can only be created at a large-enough scale if all players in the market have the means and tools to co-operate. Modelers, land surveying offices and surveying companies all need to find a way to describe 3d models in a level of detail sufficient for their own use case but still in other use cases, such as real-time simulation.
So far, 3d environments for simulating automated driving are mostly optimized to “look nice” to the human eye using shaders as visual effects during the image generation. Whether they are useful for sensor simulation across various wavelengths is a different question. But it is the question Ludwig is trying to answer.
He proposes two things: First, that models be structured in a way that their semantics are reflected by the node structure, and second, that material attribution be performed in identical ways, irrespective of the granularity. Materials themselves are to be described in a unified way with unified attributes.
For the model structure, his initiative currently proposes a specification, consisting of node hierarchy, node labeling, and coordinate systems to specify the semantic structure and potential means of animation for 3d models. As a starting point, he provides a specification for two-axle passenger vehicles. Although this just covers a very small fraction of the entire variety of 3d models in a virtual environment, it is considered to be a beginning like a blueprint and to initiate the discussion about further standardization.
Similarly, the project proposes describing materials in all relevant properties for various sensor modalities, corresponding wavelengths and wave-matter interaction, and may act as a template for future standardization. But is it clear that parameters describing the wave-matter interaction (i.e. reflection of light rays or electromagnetic waves at corresponding obstacles) are the only needed information? It depends on the specific modeling level whether a sensor model generates objects lists or returns raw data. If the latter is the case, the sensor model may need information about atmospheric conditions as well: information such as density, humidity, temperature among others. In order to create a common understanding and a consistent set of input parameters for all relevant sensor modalities, sensor developers should provide feedback on their specific requirements here.
Even if all this information was available still the problem of providing it in a consistent way has to be considered: A standardized technical format to build on is very helpful in this case. OpenMATERIAL leverages this advantage by building on Khronos glTF, a standardized file format for 3d models in real-time use-cases. OpenMATERIAL uses the extension concept of glTF to enrich 3d models by sensor-specific material definitions. JSON-based material definitions originating from the OpenMATERIAL project are used to add relevant physical parameters as look-up tables. The OpenMATERIAL initiative provides a first initial parameter set with one generic sensor model for each sensor modality to be used as single point of truth. Additionally, there are reference materials defined based on validated measurements.
So, why all this effort?
The clear goal is to enable consistent use of 3d models and materials in simulation for automated driving and to enable the build-up of large collections of digital environment representations at varying granularity but with unified semantics and material attribution. With this, integrated but highly parallelized simulation setups using frameworks incorporating environment simulation and heterogenous sensor simulation applications are becoming an option.
These setups would allow, for example, to run a basic traffic simulation or driving simulation for the general environment, publish data about static and dynamic objects (e.g. in ASAM OSI format) and subscribe to these data in a series of specialized sensor simulation models. Those models may use either just the object data (such as bounding boxes) or link 3d models and annotated material definitions in order to execute physical computations of varying fidelity on them (e.g. using ray-tracing methods).
But what is more, different parties could even feed their proprietary sensor simulation code with identical 3d models and provide the results of their detections etc. back to the basic simulation code and, further downstream, to the planning and acting stages of the autonomous stack.
A similar setup has recently been shown within the framework of the SET Level project where a distributed simulation framework was set up using heterogeneous simulation components. In this proof of concept, the unified definitions of materials and 3d structure specified by OpenMATERIAL enabled integration of different environment simulations as well as reflection-based sensor models parallel to object-based sensor models, confirming the modularity concept behind OpenMATERIAL.
By separating the simulation of environment and sensors, also responsibilities and liabilities, can be clearly separated. Ludwig pointed in our talk to the three parties OEMs (responsible for the driving function), tool providers (responsible for environment and scenario simulation) and sensor suppliers (responsible for specific sensor models). Each can specialize on their strength but, at the same time, can ensure validity of their share within the validation tool chain.
OpenMATERIAL sees itself in an exploratory stage where its definition and direction are to be decided by applying its principles in various research projects. The above-mentioned SET Level project is one example and expectations are that there will be a few more over the next two years. After this, it is assumed to be clear enough what the requirements of a corresponding standardization project that is aiming at unifying the description of 3d environments and their semantic and material attribution will be.
Several of the next steps in this endeavor are:
Identify the interested parties (e.g. include public authorities and surveying companies)
Conduct further proof of concept studies
Identify the required fidelity and granularity of all kinds of environment models along the different use cases
Create initial sets of representative 3d models and materials that can be used across tools
Provide rules for modeling and attribution for various use cases
Which standardization organization will be the right one to take OpenMATERIAL into its portfolio is still to be decided. Promising candidates are the KHRONOS Group, owner of glTF, and ASAM, owner of OpenDRIVE, OpenCRG and OpenSCENARIO. It would also be helpful to find powerful stakeholders outside the driving simulation group to establish such kind of format generally in the computer graphics domain or sensor domain and not only focusing on the automotive domain.
More and more stakeholders think about formats they could use to store their measurements; they also have to define how detailed the formats – and the measurements – have to be to be suitable as input for simulation-based development and testing. OpenMATERIAL could be one answer.
Overall, we got the impression that OpenMATERIAL currently is an excellent collection of ideas and proposals. It shows a large variety of solutions but still lacks some depth and generalization. Therefore, it will be more than interesting to see what the upcoming projects make from it.
Thanks again to Ludwig for taking his time and talking to us.