Partner: Innoviz
When the advert for the Xbox Kinect was developed, do you think they considered how it would inspire technology for autonomous driving?
The advert showed the console utilising a range of motion sensors, designed for gamers to play using their bodies rather than a handheld controller. It uses infrared light to detect range and movement and then feed the information back into whichever game is being played.
Omer Keilaf, a manufacturer of high-performance Lidar sensors and perception software for driverless cars and drones, CEO of Innoviz said, “That was a spark moment, for me, when I realised that 3D is something that is going to be super interesting to take further. And I was trying to figure out how I might take that technology from something that works only in the living room … and make that available everywhere.”
But, 2012 was the world of mobile phones, the iPhone 5, the boom of Instagram, “that set a very high bar on the size of the product and the price of the product.” Keilaf’s describes his problem then, being that his development was going to be much larger than a phone. Shelving the idea for four years allowed time for a new obsession in the tech world, driverless cars.
Driverless cars have been promised for years, but a number of technological and regulatory hurdles have prevented their entrance into mainstream use. To truly understand this field, it is necessary to have a knowledge of the basics.
There are five ‘levels’ of autonomy for driverless cars, as set out by the US-based Society of Automotive Engineers. A good way to think about these levels is:
Level 1 – no automation whatsoever.
Level 2 – ‘Hands-on’ shared control
Level 3 – ‘Eyes off’
And Level 4 is known as ‘Mind off’. The only limitations are determined by the supplier. Which could mean closed environments, weather limitations, certain types of routes.
Level 5 is true ‘steering wheel optional’ in any situation. No limitations whatsoever.
As of early 2022, Level 3 cars are as good as it gets. That’s the level that allows a driver to take their hands off the wheel and take their ‘eyes off’ the road in certain situations. Moving further up the levels requires critical safety considerations. Cars need to be able to assess and react to their surroundings with absolute certainty before regulators will allow these systems more freedom on the road.
An elite team
It has taken an elite team of engineers, who first met in a technology division of the Israel Defence Forces, years of careful work, and more than a few grey hairs, to create on of the best range finding technologies available on the market today, this, is Lidar.
According to Keilaf, “A Lidar is like a light radar, meaning that you have a laser scanner that scans the scene rapidly, and provides you in fine details, a 3D picture or video, you would say, because it’s done in a certain framerate of everything that is around you.”
A device targets objects around it with a laser and measures how long it takes for the reflected light to return, giving an incredibly accurate picture of the physical world. Having a three-dimensional understanding of the scene around the car that updates many times per second provides the decision-making software with the best information possible and allows it to make good decisions.
With the basic concept understood, Keilaf’s team had to consider industry practicalities, in which you cannot have a single point of failure. So, a Lidar system is used to back up cameras, providing “that perfect vision in situations where the camera fails. The problem with, you know, Lidars for many years was that they were very, very big and very expensive to get to very high resolution. And you need to use in traditional classical lidars would use, hundreds of lasers, maybe 10s of hundreds of lasers. And that creates a very expensive device very big and unreliable.”
Here lay Keilaf’s task, developing a Lidar sensor that did not require many components, but still reliably achieved the high-quality resolution.
Keilaf’s team for this mission consisted of “the best engineers in the different disciplines. Optics, mechanics, signal processing, computer vision, hardware, etc. Like we brought like the A team. That I mean, we were about 18 people after one month. And through the work of this team, we did those iterations of design changes. Today, we are 420 people, a 30% of the team is from my unit.” His old unit from his military days, a group from which he keeps recruiting, so secret that until recently, he could not name, a spin-out from the larger intelligence unit, 8-200.
“It’s unit 81’ Keilaf says, ‘like the technology unit, much better, but also smaller. And this unit, they take only the best engineers out of school, like from, like, after you finish your first degree in the university. And then after seven years there where you go through the best, you know, school of product engineering, we pick the best. So we very much enjoy that relationship.”
To approach the Lidar problem, they began by examining the traditional equipment setup. Keilaf describes that “the traditional lighters would use tens of lasers (or hundreds) and basically mount them one under the other and spin them in a 360 manner. And every laser will determine a certain line in which you will get when the laser would scan only in that line and, and the resolution, the vertical resolution is very linearly connected to the number of components.” This is where the expense of the Lidar comes in, in a similar fashion to the resolution rating of a television, the more lines, the higher the resolution.
Innoviz decided to use a different method, a micro eletromechanical systems chip. This is a small silicon chip coated with silver or gold (depending on the wavelength of the laser in use). Keilaf explains that the redirect the light, saving the need to move the laser emitter itself, “and that silicon can move very fast in a 2D manner, or a two-axis manner … That obviously, is needs to carry much less weight. Right. And, and if you, you do that fast enough, and if the mirror can move in a 2D access in a linear manner, I mean, it’s not without steps. Theoretically, you can actually meet any resolution you want, because you can bend the light in any direction. And you don’t need multiples of lasers to reach to any resolution like light, right? I mean, imagine a scanner that moves in a in a raster manner, like the old television sets. And the tighter you can get those scanlines in and faster scan lines, you can get to high resolution.”
Due to the really small size of this mirror, a very fast scan is possible, and Innoviz had their breakthrough.
The improvement
In the first generation of their product, they managed 256 lines, compared to the alternative best (five years ago) of just 64 lines in a frame. “Our new Lidar reaches 800. So this is the second generation now that we are introducing. So that one will have already 800 and we are also working on a new version that will have above 1,000. So if you think about comparing that with HD cameras, which you know, HD is like 720. It’s higher than an HD camera, but in a cost that is dramatically lower than those, you know, big lidars.”
To understand the real-world relevance of the resolution numbers, it is best to look at the requirements of the autonomous systems. A safe driverless car is one that can react in sufficient time to stop the car safely. If the car drives at 80 miles per hour, the minimum braking distance is about 120m. That is without any thinking time on human timescales (easy enough for a car), and in reality you would also build in a buffer so that the car is not pushing the limits of its brakes.
Another aspect for Keilaf’s team to navigate is that “anything that is taller than about 10 centimetre, or 14 centimetres is actually a damaging object to the car, because the suspension length of the of the wheel has a certain you know, it’s actually limited, if you try to drive over a step, it will cause damage to the car and could actually hurt the person inside. So if you take that height, and you want to be able to detect that at a distance of 120 or 150 metres.”
From considering those geometrical calculations, they realised “that the optical resolution, it’s uh, it’s kind of like 0.05, or even lower, or higher resolution now.
Keilaf explains, “Now, try to imagine that you need also to see yourself and field of view, vertical field of view in front of you, that needs to see the ground and needs to see, like under drivable, like, you know, things that stand out from trucks that you don’t want to be too close to it, you get to a certain field of view, vertically off about, let’s say 30 degree or 40 degrees. Now, if you take that resolution, which I mentioned earlier, you realise that the number of lines that you need to have in the field of view is quite high, right. So, you don’t need really that resolution across the entire vertical field of view, but at least 10 degrees, where, you know, for very long range, obviously, the view is more limited. Only by that you get to around 200 lines. And, and that, you know, without the rest of the field of view. So, there are many trade-offs.”
Fortunately, Keilaf has written a white paper for anyone interested in a deep dive on how Lidar can meet autonomous level 3 requirements, it’s available on the Innoviz website. He ensured that this includes “details on you know, why you need a certain field of view, why you need a certain resolution, frame rate, horizontal field of view range, eventually, those are, you know, translated from real life use cases that safety engineers taking into account when driving a car. And it’s quite easy and straightforward to translate them back to kind of resolution range frame rate, you know, frame rate is reaction time, right. Resolution is the smaller object and an ability to do classification, vertical field of view, is to be able to do road recognition, under drivable etc. Horizontally field of view is related to cutting scenarios.”
So we see a bundle of use-cases that all drivers experience on a day-to-day basis, sticking within a margin of safety to make a decision based on the mechanical requirements of the vehicle and what people would deem acceptable.
Importantly, he explains that there are “other sensors that are taking into account cleaning system requirements, industrial design of the car, is one of the things that has a huge impact on the design of the sensor. Because you know, it needs to stand in a certain height, location, size aspect ratio of You can’t imagine how much of the design is related to asks that come from the design of the car. The visual window of the of the of the sensor, the angle of the window, the cleaning system. Those add a lot of I would say the constraints and requirements of the sensor So, and it comes together with, you know, all of the other requirements of, you know, visual capabilities, cost, reliability, you know, temperatures, etc.”
Innoviz has two serious production deals, the first being contracts with BMW working on both their first and second generation. The second working with an as of yet undisclosed German company on a shuttle bus that will be Level 4 classified.
Keilaf’s excitement extends into this public transport use case. ‘Because I think it’s much more simple than you know, going on the highway you need, you can drive quite slowly, it gives a lot of value, even if it drives slow.’
There have been challenges throughout this project, but one in particular that Keilaf did not consider at the beginning of his work on Lidar technology.
“When I started in 2016, my understanding of the problem was based on what I was able to see in the market at that time, which was a sensor with 64 lines, 80 metres range and but cost like $50,000 and very, very big and unreliable.”
All focus was on this prohibitive cost and size, and so Keilaf thought his challenge would be to find a way to make the same technology cheaper and smaller.
Luckily, he “had access to talk with real customers in automotive, I learned that the actual gap is much bigger so it’s like 80 metres not enough they want 200 metres or 250 metres … Half a degree resolution is not enough. They want like 10 times bigger than that. better than that. And, and reflectivity at 200 metres not 80%. It’s like 10%.”
In fact, he learned that the gap between capability and requirement was a factor of about 1,000.
When describing this discovery, Keilaf says it “was a very unpleasant surprise, when I started in a thought that my, the technical gap is somewhere simple. And, you know, that’s where like an entrepreneur is actually measured that you’re now facing a problem that is 1000 times bigger. And you can make a decision, like, Okay, I just started and maybe it’s okay, that, you know, maybe it’s impossible, and I got it wrong. And, and I, you know, I could decide to, you know, stop, like, like, 1000 times gap is huge.”
He says that the key to overcoming a gulf in capability, if it can be done at all, is to examine your assumptions. Do not try to fix the problem with existing components or thinking. Instead, start by thinking what the optimum would look like, what would the best components look like. Then let the imagination run wild.
Motivated to succeed
The motivation that drove Keilaf to work to improve the automotive sector. “But with all seriousness, I was very committed to make this happen, I was, I’m coming from a family where my older sister was in a very serious accident when I was younger. And I completely familiar with the hurdles of car accidents, and I was not even personally involved in the car accident, but my life completely changed only through the fact that my older sister was in an accident.”
His confidence to succeed, that was from his network of engineers. “It was very clear to me that coming from my unit, and, and, you know, the level of engineers that I was able to bring into my team, I was pretty sure that if my team eventually would not be able to solve it, then no one will, really, like I was I took the like, decided to take the role of, if we decide it’s impossible, then it’s impossible. And that would be very bad for humanity.”
