On the stage of modern technology, "visual perception technology" is playing an extremely important role. From self-driving cars to dexterous robots, to the ubiquitous intelligent surveillance systems, the performance of image sensors directly determines the success or failure of these technologies.
However, when faced with dynamic, variable, and unpredictable environments, traditional image sensors often fall short and face many challenges. These challenges mainly include limited dynamic range, data redundancy, and perceptual latency.
Dynamic range refers to the range of all pixels that can be captured from the darkest to the brightest visible range in an image or video. The larger the dynamic range, the more pixel changes the device can capture, and details from deep black to bright can be more clearly displayed. However, the dynamic range of traditional sensors is very limited, making it difficult to capture clear images in both strong and weak light environments at the same time.
Data redundancy refers to the large amount of data generated by high-resolution and high-speed sensors, which increases the burden of processing and transmission. Perceptual latency refers to the delay in perception that sensors may experience in rapidly changing environments due to the limitations of processing speed, affecting the timeliness of decision-making.
Advertisement
These issues are particularly evident in fields such as autonomous driving, robotics, and artificial intelligence. For example, in autonomous driving, sensors must be able to quickly and accurately identify road conditions and potential dangers, but traditional sensors often perform poorly when dealing with complex scenarios (such as suddenly appearing pedestrians or vehicles).These technical barriers limit the application of image sensors in complex environments and have given rise to an urgent demand for more advanced visual perception technology. Scientists are constantly researching the excellent human visual system in an attempt to find solutions.
Inspiration from the Human Visual System
The human visual system (HVS) excels in processing complex visual information. That is, the primary visual cortex, as the initial processing area for visual information, breaks down the visual information into its basic components, such as color, direction, and motion, and passes this information to the dorsal and ventral streams, which are then processed through two main pathways:
1. Cognition Pathway
The ventral stream connects to the temporal lobe and is mainly responsible for high-precision cognition and detail recognition, such as color and shape. This pathway allows us to clearly see the details and colors of objects and to accurately perceive the environment.2. Path of Action
The dorsal stream connects to the parietal lobe and is primarily responsible for rapid response and motion detection, such as direction and speed. Through this pathway, we can quickly identify moving objects and make corresponding reactions, such as avoiding obstacles or chasing targets.
Dorsal stream, ventral stream, and primary visual cortex (Image source: VISUAL SYSTEM: CENTRAL PROCESSING)
This dual-path processing method enables humans to perceive and react efficiently and accurately in various complex environments. Based on the imitation of the human visual system, a research team from Tsinghua University has developed the world's first brain-like complementary vision chip - the Tianmu chip, which overcomes the shortcomings of traditional visual perception chips and provides an unprecedented efficient and accurate visual perception solution.
The Birth of Tianmu ChipThe design concept of the Tianmu chip is based on an in-depth study of the human visual system, incorporating a hybrid pixel array and a parallel heterogeneous readout architecture.
The hybrid pixel array mimics the cone and rod cells in the human visual system, which are responsible for color and motion detection, respectively. The epithelial cells support and protect the layer of photoreceptor cells. Cone cells are primarily responsible for capturing color information, allowing us to see rich color details under bright light. Rod cells are extremely sensitive to changes in light intensity and are particularly suitable for low-light environments, helping us to see the contours and movement of objects in dim conditions.
![](/images/f4f6dce2f3a0f9dada0c2b5b66452017.png)
The parallel heterogeneous readout architecture is the core part of the Tianmu chip. Its function is to convert electrical signals from different pixels (such as cone and rod pixels) into digital data at high speed and with high precision. The advantage of this architecture is its ability to handle both high dynamic range and high-speed perception requirements simultaneously, effectively reducing data redundancy and maintaining high performance under complex lighting conditions.
By applying these new technologies, the Tianmu chip possesses three characteristics: high-speed perception ability, wide dynamic range, and bandwidth optimization, addressing the shortcomings of traditional sensors.
1. High-speed perception abilityThe Tianmou chip can achieve a speed of up to 10,000 frames per second, ensuring that clear images can still be captured in rapidly changing environments. This high frame rate perception capability is crucial for real-time perception and response applications such as autonomous driving and robotics.
2. Wide dynamic range
The unit of dynamic range calculation is dB (decibels), and the dynamic range of traditional sensors is usually between 60 and 80dB, while the dynamic range of the human eye is about 120dB. The Tianmou chip has a dynamic range of up to 130dB, providing clear images in both strong and weak light environments. This means that even in complex lighting environments where sunlight and shadows coexist, the Tianmou chip can provide fine picture details.
3. Bandwidth optimization
Through adaptive technology, the Tianmou chip can reduce bandwidth requirements by 90%, effectively reducing the burden of data transmission and processing. This bandwidth optimization technology not only improves data transmission efficiency but also reduces power consumption, making the Tianmou chip more suitable for mobile devices and Internet of Things applications.Application Cases of Tianmu Chips
The application of Tianmu chips in autonomous driving systems is a significant demonstration of their powerful performance. They can provide accurate, fast, and robust perception in complex road environments, even reacting swiftly in corner cases. This is of great importance for enhancing the safety and reliability of autonomous driving systems.
For instance, during autonomous driving tests, Tianmu chips have shown their superior performance in dealing with suddenly appearing pedestrians and vehicles, significantly reducing the probability of accidents.
In addition to autonomous driving, Tianmu chips can also be widely applied in fields such as drones and security surveillance. For example, in security surveillance, Tianmu chips can provide high-quality video images in environments with drastic changes in lighting, which helps to detect potential security threats in a timely manner. In drone applications, the high dynamic range and high-speed perception capabilities of Tianmu chips enable drones to navigate and monitor efficiently under complex terrain and lighting conditions.
The results of various experiments indicate that Tianmu chips not only have a high dynamic range and high resolution but also maintain excellent perceptibility under high-speed motion and extreme lighting conditions, performing far better than traditional sensors in extreme environments.The Tianmu chip holds endless possibilities for the future development of technology. As technology continues to advance, it will play an indispensable role in more fields. Just imagine how the ultra-high-quality visual experience brought by the Tianmu chip in augmented reality (AR) and virtual reality (VR) will completely change our perception and interaction methods? This is just the beginning.
In the future, when the Tianmu chip is deeply integrated with artificial intelligence technology, what disruptive changes will it bring to the fields of smart city construction, medical image analysis, industrial automation, etc.? How will it lead us into a more intelligent and interconnected world? The suspense remains, and we look forward to it with anticipation.