Document Details

Extracted Text:

PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA To: Granite Vehicle Ventures From: Alden Harris, HPC LLP Date: March 4, 2024 Re: Proposed Patent Infringement Lawsuit v. Tesla — U.S. Pat. Nos. 11,597,402 and 11,738,765 This memo summarizes a proposed patent infringement lawsuit against Tesla in the Marshall division of the Eastern District of Texas asserting infringement of US Pat. Nos. 11,597,402 and 11,738,765 (the “Asserted Patents”). Please contact me if you have any questions: aharris@hpcllp.com; 713-221-2011. I. Patented Subject Matter and Infringement Allegations The feature Tesla calls “Full Self Driving” (“FSD”) infringes the Asserted Patents. Detailed infringement allegations are found in the claim charts attached as Exhibits A & B. We have investigated infringement thoroughly and believe the claims read directly on Tesla’s FSD implementation, as shown by the evidence and analysis in the charts. The Asserted Patents are generally directed to safety features in self-driving vehicles (“SDVs”). The specification describes, in part, using sensors to determine information about the driver and information about the SDV itself and using this information to determine who should control the SDV — the driver or the computer. This determination can be based on the competence level of the driver, the competence level of the SDV, and whether a “fault” or “operational anomaly” has occurred. These states can be determined based on “active learning data” (e.g., past data about other SDVs and their drivers) using weighted voting (such as a feed-forward neural network). A mapping between faults and corrective actions can be encoded in a “fault remediation table.” Tesla FSD functionality monitors driver attentiveness and will alert the driver and transfer control from FSD to the driver if it detects signs of inattentiveness (such as eyes deviating from the road or hands not touching the steering wheel). This situation is one scenario in which infringement occurs. A December 2023 NHTSA recall required Tesla to add these types of FSD safety features to its “Basic Autopilot” package, which shows that these safety features are critical for Tesla to meet regulatory compliance obligations. ! Tesla FSD functionality will also alert the driver and transfer control from the FSD to the driver if it is unable to safely drive autonomously. Conditions such as poor visibility, bad weather, emergency vehicles with active sirens, absence of lane markings, and similar situations can trigger ' https://static.nhtsa.gov/odi/rel/2023/RCLRPT-23V838-8276.PDE (“The remedy will incorporate additional controls and alerts to those already existing on affected vehicles to further encourage the driver to adhere to their continuous driving responsibility whenever Autosteer is engaged, which includes keeping their hands on the steering wheel and paying attention to the roadway.”).  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA this shift from FSD to driver control. This type of situation is another scenario in which infringement occurs. More details about both of these scenarios are in the attached claim charts. IL. Prosecution and Validity Since 2020, prosecution activities have resulted in the issuance of patent claims that are litigation worthy, including identifying the closest prior art, disclosing it to the patent office, and drafting claims that are robust against validity challenges. The claims of the ’402 and ’765 Patents represent the results of these efforts. A continuation of the family remains open (App. No. 18/222,774) and new claims will be added to that pending application. In light of the open continuation, the client retains the ability to prosecute new claims in the future based on the patent family’s shared specification. Potential prior art disclosed to (and considered by) the patent office includes Pat. No. 9,342,074 (a Google Waymo patent) and Tesla’s own Model S Manual from 2015. This art appears on the faces of the Asserted Patents, meaning the claims were examined by the PTO and allowed over this prior art. A complete analysis of all potential prior art is beyond the scope of this memo. However, should you have any specific questions about specific art, please let me know and I’ll be happy to address them. As is typical in patent litigation, we expect Tesla to mount validity challenges and possibly file IPRs. We are optimistic about our ability to meet these challenges, in part, because the PTO has considered the closest known prior art and allowed the claims over that art. Til. Damages This discussion assumes no pre-suit damages, although such damages may be available. Tesla sells the FSD software package for $12,000.* The FSD software package appears to be the smallest salable unit that embodies the accused features, making it a good candidate for a damages base calculation. See, e.g., Finjan, Inc. v. Blue Coat Sys., 879 F.3d 1299, 1310 (Fed. Cir. 2018) (“The smallest salable unit principle directs that ‘in any case involving multi-component products, patentees may not calculate damages based on sales of the entire product, as opposed to the smallest salable patent-practicing unit’...”). About 19% of U.S. Tesla customers opt for FSD.? Tesla sold 490,000 vehicles in the US in 2022. This means about 93,100 FSD packages were sold in the US in 2022. (Note that Tesla vehicles “made” in the US but sold outside of the US will also be subject to the damages model.) The Asserted Patents expire September 25, 2035 (11 years, 7 months from now). If we assume a flat rate of sales, a total of 1,078,098 FSD packages will be sold in the US between now and the ? https://insideevs.com/news/684708/tesla-fsd-price-cut-september-2023/ (the price was previously $15,000). 3 https://insideevs.com/news/629094/tesla-how-many-buy-fsd/ (“roughly 19% of customers opted for FSD”).  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA expiration date of the Asserted Patents. This represents a revenue base of $12,000 x 1,078,098 = $12,937,176,000 attributable to the smallest salable unit during the damages period. To this royalty base, a royalty rate must be applied. A damages expert will derive the appropriate royalty rate by looking at evidence such as comparable license rates. Let us assume a very conservative royalty rate of 1% (which in reality can be much higher in litigation). Multiplying this 1% rate by the base yields a lump sum damages figure of about $129 million. An actual lump sum damages model at trial would need to be reduced somewhat to account for net present value. Tesla’s worldwide sales increased by a factor of 1.4 in 2023 compared to 2022.* Public data suggests US sales increased by about the same amount. Thus, if 2023 sales numbers are used instead of 2022 numbers, the estimated lump sum damages figure would be about $181 million. Tesla has seen significant sales growth for the past several years, so these flat sales estimates are likely conservative. It is also likely that the rate of FSD adoption will increase as Tesla continues to upgrade the technology. We believe any upgraded version of FSD remains likely to infringe. IV. Venue We propose to file suit against Tesla in the Marshall division of the Eastern District of Texas. Venue is proper in EDTX owing to the presence of at least three Tesla facilities in Plano and Tyler. The judges in Marshall are experienced with patent litigation and HPC has significant experience litigating in this venue. Tesla may move to transfer venue under 28 U.S.C. § 1404. If this happens, the court would consider the factors outlined by In re Volkswagen of Am., Inc., 545 F.3d 304 (5th Cir. 2008). The inquiry generally focuses, in part, on the locations of relevant witnesses and documents. If Tesla moves to transfer to the Northern District of California, we believe we will have a strong opposition to that motion. Favorable facts ascertainable from public information include: e Tesla is headquartered in Austin and has a large ““Gigafactory” near Austin. e We have identified at least 32 Tesla employees in Texas with materially relevant job descriptions. Venue discovery will likely uncover more. e Samsung’s Austin Semiconductor fab has produced Tesla’s FSD chips for several years® and it appears that this relationship has continued for the latest generation of FSD boards.’ * https://www.reuters.com/markets/us/wall-st-looks-set-subdued-start-2024-apple-dips-2024-01- 02/ (1.8 million Tesla units sold worldwide in 2023, up from 1.27 million in 2022, a 1.4 increase). > https://tridenstechnology.com/tesla-sales-statistics/ ° https://en.wikichip.org/wiki/tesla_(car_company)/fsd_chip 7 https://insideevs.com/news/535446/tesla-fsd-chip-produced-samsumg/  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA If Tesla moves to transfer to the Western District of Texas instead of the Northern District of California, we would likely consent to transfer owing to Tesla’s large presence in Austin. By filing in the Eastern District of Texas, we will force Tesla to choose between seeking transfer to the Western District of Texas only, Northern District of California only, or Northern District of California primarily with Western District of Texas in the alternative. Tesla does not always move to transfer. In the cases where it has done so successfully, it appears that either the subject matter of the lawsuit was different (e.g., Unicorn Energy GmbH v. Tesla, Inc., EDTX No. 2:20-cv-00338, a case that involved battery charging technology) or the plaintiff did not mount a serious opposition to the transfer motion (e.g., Arsus, LLC v. Tesla, Inc., WOTX 6:22-cv-00276).  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Exhibit A — U.S. Pat. No. 11,597,402 v. Tesla Current Tesla Models (including Models S, 3, X, and Y) include hardware capable of executing downloadable software that infringes USS. Pat. No 11,597,402. Specifically, two levels of software packages include infringing functionality: (1) Enhanced Autopilot, which features the infringing functionality known as “Navigate on Autopilot”; and (2) Full Self Driving Capability, which includes the infringing functionality known as “Autosteer on City Streets” (which is currently being tested through the public “Full Self Driving Beta Test,” otherwise known as “FSD Beta”). The infringing functionality is referred to collectively in this chart as “Full Self Driving Functions.” Claim 4 4. [PRE] A self- This preamble is likely non-limiting. To the extent the preamble is limiting, Tesla vehicles including Models driving vehicle S, 3, X, and Y (collectively, “Teslas”) are examples of self-driving vehicles. (SDV) comprising: As either an option at purchase or as a monthly subscription, a Tesla owner can “unlock” the software that allows the performance of “Full Self Driving Functions.” For $6,000 at purchase, Tesla owners can unlock “Enhanced Autopilot”, which includes a Full Self Driving Function called “Navigate on Autopilot” that autonomously navigates the vehicle from the on ramp of a freeway to a desired exit, where the driver takes back over to complete the rest of the trip. Enhanced Autopilot $6,000 . e Auto Lane Change e Autopark « Summon e Smart Summon  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA https://www.tesla.com/models/design#overview® When using Autosteer on a controlled-access highway (a main highway on which road users enter and exit using on-ramps and off-ramps). Navigate on Autopilot guides Model S to off-ramps and interchanges based on your navigation route. Along the highway portion of a navigation route, Navigate on Autopilot also changes lanes to prepare for exits (route-based lane changes) and to minimize the driving time to your destination (speed-based lane changes). https://www.tesla.com/ownersmanual/2012_2020_models/en_us/GUID-0535381F-643F-4C60-85AB- 1783E723B9B6.html For $15,000 at purchase or $200 a month, a Tesla owner can unlock “Full Self-Driving Capability”, which promises to extend the Full Self Driving Function Navigate on Autopilot beyond the highway and onto city streets. Full Self-Driving Capability $15,000 « All functionality of Basic Autopilot and Enhanced Autopilot e Traffic Light and Stop Sign Control Coming Soon « Autosteer on city streets 8 All references to one Tes la Model apply equally to all other Tesla models.  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA https://www.tesla.com/models/design#overview Subscription Pricing Your vehicle’s current Autopilot package of Basic Autopilot or Enhanced Autopilot will determine the FSD capability subscription price. Basic Autopilot to FSD capability $199.00 per month Enhanced Autopilot to FSD capability $99.00 per month https://www.tesla.com/support/full-self-driving-subscriptions Owners who either purchase or subscribe to FSD Capability are also eligible to apply for access to the “Full Self-Driving Beta” program, which is currently testing and collecting data on FSD capability on city streets. v Can! request Full Self-Driving Beta if | am subscribed to Tesla Full Self-Driving capabilities? Yes. As long as you have the option to request Full Self-Driving Beta from your vehicle’s touchscreen, you are eligible to enroll regardless of whether you have purchased Tesla Full-Self Driving capabilities with a one-time payment or subscription. To view if you have access to Full Self-Driving Beta, select ‘Controls’ > ‘Autopilot’ > ‘Request Full Self-Driving Beta.’ https://www.tesla.com/support/full-self-driving-subscriptions  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Tesla released the beta testing version of its new Full Self-Driving suite last week, and it has already started pulling a lot of data from the vehicles using the feature. The company mentioned that the data used from this beta testing will be used for improving the neural networks for the self-driving features. The amount of data they got is so high, that they are already planning an update to the test version. https://www.vehiclesuggest.com/tesla-collecting-huge-amount-of-data-through-fsd-testing/ (dated October 26, 2020) The equipment on Teslas include a variety of sensors (e.g., cameras, radar, and sonar) and a computing platform specially designed to perform the computations required to perform the Full Self Driving Functions. The most recent version of this hardware—which is required for FSD Capability—is called “Hardware 3” (alternatively known as “HW3”) and includes the following equipment: Cameras: Eight cameras covering all angles. Sensors: Continental Radar with 558 ft range & 12 Sonar Sensors with 26 ft range. Computers: Two bespoke Tesla-designed units. https://www.currentautomotive.com/the-ultimate-guide-to-tesla-autopilot/ Note: Depending on the specific model, some cars may lack radar or both radar and sonar, depending on their date of manufacture. All cars with HW3 come with cameras and the Tesla designed FSD computer.  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Safety is at the core of our design and engineering decisions. In 2021, we began our transition to Tesla Vision by removing radar from Model 3 and Model Y, followed by Model S and Model X in 2022. Today, in most regions around the globe, these vehicles now rely on Tesla Vision, our camera-based Autopilot system. https://www.tesla.com/en_eu/support/transitioning-tesla- vision#:~:text=In%202021%2C%20we%20began%20our,our%20camera%2Dbased%20Autopilot%20system. Hardware 4 (or “HW4’’) is expected to include similar equipment, with reintegration of radar (standard, as opposed to varying from car to car), and a more powerful computing platform. https://electrek.co/2023/02/15/tesla-self-driving-hw4-computer-leaks-teardown/ [A] a sensor Tesla cars include a sensor system comprising a plurality of sensors. See Claim 4[PRE], supra. system comprising a plurality of All Tesla vehicles have a set of sensors, comprising at least several cameras. The current hardware package sensors; (including all sensors) is Hardware 3. The next generation of hardware will be known as Hardware 4. Each of these hardware packages necessarily includes sensor systems that collect and relay information to the FSD computer.  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA How It Works Your Model 3 includes the following components that actively monitor the surrounding area: 1. Acamera is mounted above the rear license plate. 2. Ultrasonic sensors (if equipped) are located in the front and rear bumpers. 3. A camera is mounted in each door pillar. 4. Three cameras are mounted to the windshield above the rear view mirror. 5. A camera is mounted to each front fender. 6. Radar (if equipped) is mounted behind the front bumper. Mode! 3 is also equipped with high precision electronically-assisted braking and steering systems. NOTE: Ensure all cameras and sensors (if equipped) are clean before each drive. See Cleaning Cameras and Sensors on page 79 for more information. Dirty cameras and sensors, as well as environmental conditions such as rain and faded lane markings, can affect Autopilot performance. Tesla 3 Owner’s Manual at 77. In addition to these sensors and more typical vehicle sensors (e.g., speedometer, GPS, etc.), Tesla cars also have camera sensors and steering wheel sensors that monitor the status of the driver: 10  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA The cabin camera can determine driver inattentiveness and provide you with audible alerts, to remind you to keep your eyes on the road when Autopilot is engaged. Tesla 3 Owner’s Manual at 86, 115. Hold Steering Wheel Autosteer determines how best to steer Model 3. When active, Autosteer requires you to hold the steering wheel. If it does not detect your hands on the steering wheel for a period of time, a flashing blue light appears at the top of the car status section of the touchscreen and the following message displays: Apply slight turning force to steering wheel Autosteer detects your hands by recognizing slight resistance as the steering wheel turns, or from you manually turning the steering wheel very lightly (without enough force to take over steering). Autosteer also qualifies your hands as being detected if you engage a turn signal or use a button or scroll wheel on the steering wheel. NOTE: When your hands are detected, the message disappears and Autosteer resumes normal operation. Vehicle testing confirmed that this limitation is present in the accused models. [B] vehicle controls comprising: engine throttle, steering mechanism, and braking system; Tesla cars include a steering wheel, an engine throttle, and a braking system. These vehicle controls can be operated either by the driver or by the vehicle, depending on the mode of operation. 11  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA il Interior Overview N BRR SS wpenawaw b. V7. 18. Door open button (Opening Doors from the interior on page 14) Tum signal stalk (High Beam Headtights on page $4), Turn Signals on page 54, and Windshield Washers on page 59) . Horn (Horn on page 48) . Drive stalk (How to Shift on page 51, Traffic-Aware Crulse Controt on page 80, Autosteer on page 85) Touchscreen (Touchscreen Overview on page 5) Driver dome light (Lights on page 53) Cabin camera (Cabin Camera on page 115) Hazard warning flashers (Hazard Warning Flashers on page 55) Passenger dome light (Lights on page 53) Climate control vent (see Climate Controls on page 116) Power window switches (Windows on page 16) Manual door release (Opening Doors from the Interior on page 14) Left scroll button (Scroll Buttons on page 47) Brake pedal (Braking and Stopping on page 60) Accelerator pedal (Regenerative Braking on page 61) Right scroll button (Scroll Buttons on page 47) Center consote (interior Storage and Electronics on page 22) Glovebox (Glovebox on page 22) 12  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Tesla 3 Owner’s Manual at 4. Vehicle testing confirmed that this limitation is present in the accused models. [C] a computer system comprising a processor coupled to a non- transitory computer readable storage medium containing program code, the program code readable and executable by a processor; Tesla cars include a computer system comprising a processor coupled to a non-transitory computer readable storage medium containing program code, the program code readable and executable by a processor. See Claim 4[PRE], supra. Tesla cars equipped with HW3 and HW4 each have a computing platform that combines a processor coupled to a non-transitory computer readable storage medium containing the program code relating to the Full Self Driving Functions’ software, which is readable and executable by the processor of the computing platform. HW3 utilizes a specialized computer system designed in-house at Tesla. The silicon dies for processors for this computer system are manufactured by Samsung Austin Semiconductor. HW3 comprises at least two processors, highlighted here: 13  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA HW3 also contains one or more non-transitory computer readable storage media coupled to these processors. These computer readable media store the program code for the FSD and Navigate on Autopilot features, and this program code is readable and executable by the processor(s). https://www.autopilotreview.com/tesla-custom-ai-chips-hardware-3/ The so-called “FSD Computer” (part of HW3) was lauded by Elon Musk himself as a major advancement over the previously utilized Nvidia chips. https://www.youtube.com/watch?v=NJ VcsvQ30AQ Tesla’s Full Self Driving Compatible software requires the FSD Computer. 14  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Hardware upgrades to the Full Self-Driving computer are not included with Full Self- Driving capability subscriptions. To be eligible for FSD capability subscriptions, the FSD computer must be installed in your vehicle. To install the FSD computer, schedule an installation appointment from the Tesla app. https://www.tesla.com/support/full-self-driving-subscriptions HW4 is expected to include a more powerful computing platform. https://electrek.co/2023/02/15/tesla-self-driving-hw4-computer-leaks-teardown/ [D] the computer system is capable of receiving a sensor reading from the system of sensors; A Tesla’s computing platform is capable of receiving a sensor reading from the system of sensors. The computing platforms included in HW3 and HW4 are each capable of, and reliant on, receiving sensor readings from the system of sensors. 15  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Rearward Looking Side Cameras Wide Forward Camera Main Forward Camera Narrow Forward Camera Max distance 100m Max distance 60m Max distance 150m Max distance 2501 - Rear View Camera Forward Looking Side Cameras Max distance 50m Max distance 80m Tesla Vision To make use of a camera suite this powerful, each Tesla car has a powerful set of vision processing tools developed by Tesla. Built on a deep neural network, Tesla Vision deconstructs the car's environment at greater levels of reliability than those achievable with classical vision processing techniques. https://www.tesla.com/autopilot 16  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Neural Networks Apply cutting-edge research to train deep neural networks on problems ranging from perception to control. Our per-camera networks analyze raw images to perform semantic segmentation, object detection and monocular depth estimation. Our birds-eye-view networks take video from all cameras to output the road layout, static infrastructure and 3D objects directly in the top-down view. Our networks learn from the most complicated and diverse scenarios in the world, iteratively sourced from our fleet of millions of vehicles in real time. A full build of Autopilot neural networks involves 48 networks that take 70,000 GPU hours to train @. Together, they output 1,000 distinct tensors (predictions) at each timestep. https://www.tesla.com/en_eu/AI The computing platforms take the information collected from the sensors and use it to recreate the world around the car. https://www.pemag.com/news/tesla-is-developing-a-self-driving-system-that-only-uses-cameras See also https://youtu.be/eOL_rCK59Z1?t=28831 Further evidence that the Full Self Driving Functions are reliant on receiving a sensor reading from the system of sensors is that the functions become unavailable if the sensors are malfunctioning, obstructed, or damaged: 17  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Limitations Many factors can impact the performance of Autopilot components, causing them to be unable to function as intended. These include (but are not limited to): * Poor visibility (due to heavy rain, snow, fog, etc.). * Bright light (due to oncoming headlights, direct sunlight, etc.). + Damage or obstructions caused by mud, ice, snow, etc. * Interference or obstruction by object(s) mounted onto the vehicle (such as a bike rack). * Obstruction caused by applying excessive paint or adhesive products (such as wraps, stickers, rubber coating, etc.) onto the vehicie. Traffic signs that do not conform to standard recognizable formats, such as digital or temporary speed signs. * Narrow or winding roads. + A damaged or misaligned body panel. Use of gray or aftermarket glass. Interference from other equipment that generates ultrasonic waves. Extremely hot or cold temperatures. Tesla Model 3 Owner’s Manual at 78-79. In situations where Autosteer is temporarily unavailable, the Autosteer icon disappears. For example, your driving speed is not within the speed required for Autosteer to operate. Autosteer may also be unavailable if it is not receiving adequate data from the camera(s). Tesla Model 3 Owner’s Manual at 85. 18  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Autosteer and its associated functions are particularly unlikely to operate as intended when: « A camera(s) or sensor(s) is obstructed, covered, or damaged. ¢ Bright light (such as direct sunlight) is interfering with the view of the camera(s). * Model 3 is being driven very close to a vehicle in front of it, which is blocking the view of the camera(s). Tesla Model 3 Owner’s Manual at 88. WARNING: Autosteer is a hands-on feature. You must keep your hands on the steering wheel at all times. WARNING: Autosteer is intended for use on controlled-access highways with a fully attentive driver. When using Autosteer, hold the steering wheel and be mindful of road conditions and surrounding traffic. Do not use Autosteer in construction zones, or in areas where bicyclists or pedestrians may be present. Never depend on Autosteer to determine an appropriate driving path. Always be prepared to take immediate action. Failure to follow these instructions could cause damage, serious injury or death. Tesla Model 3 Owner’s Manual at 85-86. Autosteer determines how best to steer Model 3. When active, Autosteer requires you to hold the steering wheel. If it does not detect your hands on the steering wheel for a period of time, a flashing blue light appears at the top of the car status section of the touchscreen and the following message displays: Apply slight turning force to steering wheel Autosteer detects your hands by recognizing slight resistance as the steering wheel turns, or from you manually turning the steering wheel very lightly (without enough force to take over steering). Autosteer also qualifies your hands as being detected if you engage a turn signal or use a button or scroll wheel on the steering wheel. 19  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Tesla Model 3 Owner’s Manual at 115. Vehicle testing suggests this limitation is met: the Full Self Driving Functions utilize sensor readings collected from at least the cameras and the steering wheel sensors. 20  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA [E] the computer system is capable of operating the vehicle controls; A Tesla’s computing platform is capable of operating the vehicle controls. The computing platforms included in HW3 and HW4 are each capable of operating the vehicle controls, including accelerating, braking, and steering. For example, “[the Full Self Driving Function] Navigate on Autopilot [operates the vehicle controls to] guide [the] Model 3 to off-ramps and interchanges based on your navigation route. Along the highway portion of a navigation route, Navigate on Autopilot also [operates the vehicle controls to] change[] lanes to prepare for exits (route-based lane changes) and to minimize the driving time to your destination (speed-based lane changes).” Tesla Model 3 Owner’s Manual at 90. Furthermore, Navigate on Autopilot operates the vehicle controls to do “Speed Based Lane Changes”: * Speed Based Lane Changes: Navigate on Autopilot is designed to perform both route-based and speed- based lane changes. Route-based lane changes are designed to keep you on your navigation route (for example, moving you into an adjacent lane to prepare for an upcoming off-ramp) whereas speed-based lane changes are designed to maintain a driving speed (not to exceed your cruising speed) that allows you to minimize the time it takes to reach your destination (for example, moving into an adjacent lane to pass a vehicle in front of you). Speed-based Tesla Model 3 Owner’s Manual at 90; see also Tesla Model 3 Owner’s Manual at 91 (“If Require Lane Change Confirmation is turned off, Navigate on Autopilot engages the appropriate turn signal, checks for vehicles and objects, and when appropriate, maneuvers Model 3 into the adjacent lane.”’). Navigate on Autopilot also has a setting, which if enabled, will operate the vehicle controls to utilize HOV lanes: 21  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA NOTE: When determining navigation routes, and maneuvers at interchanges, Navigate on Autopilot considers whether or not you want to use High Occupancy Vehicle (HOV) lanes. Therefore, ensure the Use HOV Lanes setting is appropriate for your circumstances (see Maps and Navigation on page 142). If the setting is off, Navigate on Autopilot never uses a HOV lane, regardless of time of day. If the setting is on, Navigate on Autopilot uses HOV lanes, whenever applicable. Tesla Model 3 Owner’s Manual at 91. Similarly, HW3 and HW4 performing the Full Self Driving Function “Autosteer on City Streets” (as part of the Full Self Driving Capability package) is capable of operating the vehicle controls, including accelerating, braking, and steering. This video (showing FSD beta 11.3.6) shows how Autosteer on City Streets accelerates, brakes, and steers the vehicle: https://www.youtube.com/watch?v=bH9fD5tB33s Vehicle testing confirmed that this limitation is present in the accused models. [F] the computer system is capable of determining the operational state of the self-driving vehicle (SDV); A Tesla’s computing platform is capable of determining the operational state of the self-driving vehicle. The computing platform is able to determine a variety of details about the operational state of the vehicle, such as: (1) its location and direction (via GPS and cameras); (2) its current speed (via speedometer and cameras); (3) the state of the driver (through the cabin camera and the steering wheel sensors); (4) the state of the road the vehicle is traveling on (through cameras); and (5) the location of things around it (through cameras and, on some models, radar and sonar). See Claim 4[A], [D], supra. The display below shows the current speed on the top left, the state of the road and the orientation of the car on it on the left, and the location of the vehicle via GPS on the right. 22  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA 4 ee e #0 00: seer j Sybawson Oo FE Goreme ce oe Be nesice eG 2 Roy: zi Stonewall Jackson 1oe's Stegghote @ f = Loe a S SSE 40% Ri pea) (TTT https://teslamotorsclub.com/tme/threads/why-didnt-tesla-put-the-map-on-the-left-side-of-the-ui.257818/ Tesla displays various types of objects around the car as detected by the sensors. https://www.notateslaapp.com/tesla-reference/636/all-tesla-fsd-visualizations-and-what-they-mean The Full Self Driving Functions may only be available on certain roads: 23  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA When using Autosteer on a controlled-access highway (a main highway on which road users enter and exit using on-ramps and off-ramps). Navigate on Autopilot guides Mode! 3 to off-ramps and interchanges based on your navigation route. Along the highway portion of a navigation route, Navigate on Autopilot also changes lanes to prepare for exits (route-based lane changes) and to minimize the driving time to your destination (speed-based lane changes). Tesla Model 3 Owner’s Manual at 90. More generally, the Functions are only available under certain conditions: The speed at which you can initiate Autosteer can vary depending on various conditions and whether or not a vehicle is detected ahead of you. When no vehicle is detected ahead of you, you must be driving at least 18 mph (30 km/h), unless certain vehicle and environmental conditions are met, in which case, you may be able to initiate it at lower speeds. When a vehicle is detected ahead of you, you can initiate Autosteer at any speed, even when stationary, provided Model 3 is at least 5S feet (150 cm) behind the detected vehicle. Tesla Model 3 Owner’s Manual at 85. In other conditions, the Functions may not be available at all: In situations where Autosteer is temporarily unavailable, the Autosteer icon disappears. For example, your driving speed is not within the speed required for Autosteer to operate. Autosteer may also be unavailable if it is not receiving adequate data from the camera(s). Tesla Model 3 Owner’s Manual at 85. Of course, Tesla requires the driver’s hands to be on the wheel at all times: 24  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Autosteer requires that you pay attention to your surroundings and remain prepared to take control at any time. If Autosteer still does not detect your hands on the steering wheel, the flashing light on the car status section of the touchscreen increases in frequency and a chime sounds. If you repeatedly ignore Autosteer’s prompts to apply slight force to the steering wheel, Autosteer disables for the rest of the drive and displays the following message requesting you to drive manually. If you don't resume manual steering, Autosteer sounds a continuous chime, turns on the warning flashers, and slows the vehicle to a complete stop. Autosteer unavailable for the rest of this drive, Hold steering wheel to drive manually. For the rest of the drive, you must steer manually. Autosteer is available again on your next drive (after you stop and shift Model 3 into Park). Tesla Model 3 Owner’s Manual at 86. [G] the computer system is capable of determining a vehicle fault; A Tesla’s computing platform is capable of determining a vehicle fault. During the operation of the vehicle, one or more faults may arise that limit the functionality of the Full Self Driving Functions. A non-exhaustive list of faults is set forth below: 25  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Limitations Many factors can impact the performance of Autopilot components, causing them to be unable to function as intended. These include (but are not limited to): * Poor visibility (due to heavy rain, snow, fog, etc.). * Bright light (due to oncoming headlights, direct sunlight, etc.. + Damage or obstructions caused by mud, ice, snow, etc. + Interference or obstruction by object(s) mounted onto the vehicle (such as a bike rack). Obstruction caused by applying excessive paint or adhesive products (such as wraps, stickers, rubber coating, etc.) onto the vehicle. * Traffic signs that do not conform to standard recognizable formats, such as digital or temporary speed signs. * Narrow or winding roads. + A damaged or misaligned body panel. + Use of gray or aftermarket glass. * Interference from other equipment that generates ultrasonic waves. + Extremely hot or cold temperatures. Tesla Model 3 Owner’s Manual at 78-79. Most of these limitations are based on obstruction or damage to the sensors. Other faults may occur. A WARNING: The list above does not represent an exhaustive list of situations that may interfere with proper operation of Autopilot components. Never depend on these components to keep you safe. It is the driver's responsibility to stay alert, drive safely, and be in control of the vehicle at all times. 26  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Tesla Model 3 Owner’s Manual at 79. For example, faults may be based on the behavior of the driver. The Full Self Driving Functions are not fully autonomous, but instead specify that the driver must remain alert and be prepared to take over at any time. Thus, Tesla cars are designed so that the drivers keep their hands on the wheel: WARNING: Autosteer is a hands-on feature. You must keep your hands on the steering wheel at all times. WARNING: Autosteer is intended for use on controlled-access highways with a fully attentive driver. When using Autosteer, hold the steering wheel and be mindful of road conditions and surrounding traffic. Do not use Autosteer in construction zones, or in areas where bicyclists or pedestrians may be present. Never depend on Autosteer to determine an appropriate driving path. Always be prepared to take immediate action. Failure to follow these instructions could cause damage, serious injury or death. Autosteer determines how best to steer Model 3. When active, Autosteer requires you to hold the steering wheel. If it does not detect your hands on the steering wheel for a period of time, a flashing blue light appears at the top of the car status section of the touchscreen and the following message displays: Apply slight turning force to steering wheel Autosteer detects your hands by recognizing slight resistance as the steering wheel turns, or from you manually turning the steering wheel very lightly (without enough force to take over steering). Autosteer also qualifies your hands as being detected if you engage a turn signal or use a button or scroll wheel on the steering wheel. Tesla Model 3 Owner’s Manual at 85-86. Similarly, the car is capable of determining a number of other contextual faults: 2]  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA A A CAUTION: If available in your market region, Model 3 detects lights from an emergency vehicle when using Autosteer at night on a high speed road, the driving speed is automatically reduced and the touchscreen displays a message informing you of the slowdown. You will also hear a chime and see a reminder to keep your hands on the steering wheel. When the light detections pass by or cease to appear, Autopilot resumes your cruising speed. Alternatively, you may tap the accelerator to resume your cruising speed. WARNING: Never assume that your ability to see a traffic light, stop sign, or road marking (especially at a complex intersection, or an intersection in which a traffic light or sign is partially obstructed, etc.) means that Model 3 can also see it and respond appropriately. WARNING: Even the most recent map data does not include all traffic lights and stop signs. Therefore, Traffic Light and Stop Sign Control relies heavily on the ability of the cameras to detect traffic lights, stop signs, road markings, etc. As a result, Model 3 may ignore an intersection that is blocked from the camera's view (for example, obstructed by a tree or a large vehicle or object, or located near a steep hill or sharp curve). WARNING: Traffic Light and Stop Sign Control is not a substitute for attentive driving and sound judgment. Tesla Model 3 Owner’s Manual at 86, 94. Canceling Autosteer Autosteer cancels when: * You press the brake pedal. * You start steering manually. + You exceed the maximum speed at which Autosteer operates - 90 mph (150 km/h). + You move the drive stalk upwards. + A door is opened. + An Automatic Emergency Braking event occurs (see Collision Avoidance Assist on page 111). WARNING: In some situations, Traffic Light and Stop Sign Control may inaccurately detect a traffic light or stop sign, causing Model 3 to slow down unexpectedly. Be prepared to take immediate action at all times. 28  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Tesla Model 3 Owner’s Manual at 115. Vehicle testing confirmed that this limitation is present in the accused models. [H] the computer _| A Tesla’s computing platform is capable of determining a competence level of the processor. system is capable of determining a 29  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA competence level of the processor; The computing platform of the Tesla weighs the collected sensor data and determines the level of competence the processor has in the current situation. For example, if the degree of difficulty is low, sufficient amounts of data can be collected, and the processor is well-trained, the computer system will determine the processor has a high competence level. If, on the other hand, the situation is more complex and complicated, the data collected is missing or unintelligible, and/or the processor is poorly trained, the computer system will determine the processor has a low competence level. When competence levels are high, the vehicle is able to conduct the Full Self Driving Functions without issue. For example, consider the following video which documents a fairly smooth drive when conditions are ideal: https://www.youtube.com/watch?v=gNRIf-UXunU When processor competence levels are low, the self-driving functions may become unavailable. See Claim 4[G], supra. [1] the computer system is capable of determining competence level of a human driver; A Tesla’s computing platform is capable of determining competence level of a human driver. For example, the Tesla’s computing platform can use sensor data collected from the steering wheel and the cabin camera (discussed with regard to limitation 4[A], supra) to determine the competence level of the human driver—whether the driver is alert and with their hands on the wheel or distracted and ill-prepared to take control of the vehicle. See Claim 4[G], supra. Further evidence of this was demonstrated through testing the vehicle itself. Additionally, the vehicle collects information on the habits of the human driver and calculates a “safety score” which allows for comparisons between two Tesla drivers’ relative degree of safe driving. 30  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Safety Score 90 The Safety Score ®¢t? is an assessment of your driving behavior based on several metrics called Safety Factors. These are combined to estimate the likelihood that your driving could result in a future collision. We combine your daily Safety Scores (up to 30 days) to calculate the aggregated Safety Score, displayed on the main ‘Safety Score’ screen of the Tesla app. You can find details around your daily Safety Score by selecting ‘Daily Details’ at the bottom of the screen. https://www.tesla.com/support/safety-score#version-2.0 Tesla originally used this score to determine the order of drivers who received access to FSD beta. https://www.notateslaapp.com/news/993/tesla-is-pushing-fsd-beta-automatically-to-owners-who-qualify Additionally, Tesla drivers risk losing access to FSD beta for unsafe driving. https://www.teslarati.com/tesla- resets-fsd-beta-strikes-forced-disengagement/; see also https://www.youtube.com/watch?v=adBzEjL_cO0 (driver loses access to FSD beta for repeated safety violations) [J] the computer system is capable of determining a corrective action using the A Tesla’s computing platform is capable of determining a corrective action using the competence level of the processor and the competence level of the human driver. Using both the competence level of the processor and the competence level of the driver, the computing platform of the Tesla can determine, for example, if the proper corrective action is to hand control of the 31  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA competence level of the processor and the competence level of the human driver; vehicle back to the driver. Specifically, when the competence of the processor is low, e.g., due to inadequate sensor readings, and the competence of the human driver is high, e.g., attentive and prepared to take control, the computing platform may determine the proper corrective action is to hand control back to the human driver. See Claim 4[G], supra. Alternatively, the computing platform of the Tesla can determine, when the competence level of the processor is high but the competence level of the driver is low, to take the corrective action of alerting the driver to pay attention. See Claim 4[G], supra. This is useful because, even though the competence level of the processor is currently high, that may change, and the driver must be prepared to take over immediately. When the competence level of both the processor and the human driver are low, the computing platform may determine the proper corrective action is to bring the vehicle to a stop. See Claim 4[G], supra. [K] the computer system is capable of implementing the corrective action; and A Tesla’s computing platform is capable of implementing the corrective action. See Claim 4[J], supra (discussing various forms of corrective actions). The computing platform of the Tesla is capable of implementing the corrective action from Claim 4[J], supra, including handing control back to the driver. Take Over Immediately In situations where Autosteer is unable to steer Model 3, Autosteer sounds a warning chime and displays the following message on the touchscreen: Take over immediately When you see this message, TAKE OVER STEERING IMMEDIATELY. Tesla Model 3 Owner’s Manual at 86. 32  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA As stated earlier, when the computing platform of the Tesla determines the proper corrective action is to hand control of the vehicle back to the driver, it can do so. When the computing platform of the Tesla determines the proper corrective action is to alert the driver to pay attention, it can do so. And when the computing platform of the Tesla determines the proper corrective action is to decelerate the vehicle and pull it over, it can do so. Additionally, Full Self Driving Functions implement many corrective actions while operating in the respective self-driving modes. See, e.g., https://www.youtube.com/watch?v=rwP W2z6gcDM (testing FSD Beta in an obstacle course). [L] the computer system is capable of issuing an alert indicating the corrective action. A Tesla’s computing platform is capable of issuing an alert indicating the corrective action. The computing platform of the Tesla is capable of issuing an alert indicating the corrective action, including making an audible chime and flashing a notification on the media unit to the driver. See Claim 4[K], supra. Vehicle testing confirmed that this limitation is present in the accused models. Claim 6 6. [PRE] The SDV of claim 4, further comprising: See Claim 4[PRE]-[L], supra. [A] the sensor system comprises sensors that detect a physical state of the human driver; The Tesla includes a sensor system which itself includes sensors that detect a physical state of the human driver, such as sensors on the steering wheel to sense the driver’s hands and a cabin camera to monitor driver behavior. See Claim 4[A], supra. Vehicle testing confirmed that this limitation is present in the accused models. [B] the sensor readings comprise an input from a steering mechanism sensor; The Tesla includes a sensor system which collect sensor readings including input from a steering mechanism sensor—the steering wheel. See Claim 4[A], supra. 33  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Autosteer determines how best to steer Model 3. When active, Autosteer requires you to hold the steering wheel. If it does not detect your hands on the steering wheel for a period of time, a flashing blue light appears at the top of the car status section of the touchscreen and the following message displays: Apply slight turning force to steering wheel Autosteer detects your hands by recognizing slight resistance as the steering wheel turns, or from you manually turning the steering wheel very lightly (without enough force to take over steering). Autosteer also qualifies your hands as being detected if you engage a turn signal or use a button or scroll wheel on the steering wheel. Tesla Model 3 Owner’s Manual at 86. See also https://www.youtube.com/watch?v=adBzEjL_cO0 at 2:00 (Tesla FSD alerts the driver when there is no input from the steering wheel). Vehicle testing confirmed that this limitation is present in the accused models. [C] the computer system determines the competence level of the processor using active learning data, said active learning data including The computing platform of the Tesla determines the competence level of the processor using active learning data, said active learning data including information from other SDVs. Tesla’s neural networks are trained using active learning data collected from other SDVs on the road. “Our networks learn from the most complicated and diverse scenarios in the world, iteratively sourced from our fleet of millions of vehicles in real time. A full build of Autopilot neural networks involves 48 networks that take 70,000 GPU hours to train O. Together, they output 1,000 distinct tensors (predictions) at each timestep.” 34  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA information from other SDVs; https://www.tesla.com/en_eu/AI “Andrej Karpathy explains the active learning procedure at Tesla, which they call the Data Engine. For example, in an object detection task and for a bike attached to the back of a car, the neural network should detect just one object (car) for downstream tasks such as decision-making and planning. ... They find a few images that show this pattern and use a machine learning mechanism to search for similar examples in their fleet to fix this problem. ... Then human annotators will annotate these examples as single cars, and the neural network will be trained on these new examples. So, in the future, the object detector will understand that it is just an attached bike to a car and consider that as just a single car. They do this all the time for all the rare cases. So their model will become more and more accurate over time. After collecting some initial data, the models are trained. Then, wherever the model is uncertain, or there is human intervention or disagreement between the human behavior and the model output, which is running in shadow mode, the data will be selected to be annotated by humans, and the model will be trained on that data.” https://medium.com/aiguys/active-learning-and-data-auto-labeling-in-autonomous-driving- 5d6bec956a38#b36a The neural networks in the Tesla SDV use their training, which comprises active learning data from other SDVs, to determine the competence level of the processor. For competence level of a processor, see Claim 4[H], supra. [D] the computer system determines the competence level of the human driver using the sensor readings; See Claim 4[I], supra. [E] the corrective action is to transfer vehicle controls to the human driver; See Claim 4[K], supra. 35  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA [F] the alert indicates take over immediately. See Claim 4[L], supra, where the alert indicates take over immediately. Claim 8 8. The SDV of claim 4, further comprising: See Claim 4[PRE]-[L], supra. [A] the sensor system comprises sensors that detect a physical state of the human driver; See Claim 6[A]. [B] the sensor readings comprise an input from a steering mechanism sensor; See Claim 6[B]. [C] the computer system determines the corrective action using weighted voting, wherein a weighted voting parameter is determined based on active learning data, said active learning data including information from other SDVs; The computing platform of the Tesla determines the corrective action using weighted voting, wherein a weighted voting parameter is determined based on active learning data, said active learning data including information from other SDVs. Tesla’s neural networks are trained using active learning data collected from other SDVs. The Tesla neural network uses weighted voting to determine the corrective action. The weights applied to the input by the neural network are determined, at least in part, based on the training using active learning data from other SDVs. “Each cycle, 256 bytes of activation data and an additional 128 bytes of weight data is read from the SRAM into the MACs array where they are combined. Each NPU has a 96x96 multiply-accumulate array for a total of 9,216 MACs and 18,432 operations. ... Under normal operation, the neural network program is loaded at the start and is kept in memory for the entire duration in which the chip is powered. Running is done by setting the input buffers address (e.g., newly taken image sensor photo), setting the output buffer address, and 36  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA weight buffer address (e.g., network weights), set the program address, and run. The NPU will asynchronously run on its own the entire neural network model until reaching a stop instruction which triggers an interrupt, letting the CPU post-process the results.” https://en.wikichip.org/wiki/tesla_(car_company)/fsd_chip (emphasis added) “The behavior of a NN is not programmed. Just like a biological NN, it is trained by experience. A NN program without the training is good for nothing. It extracts the characteristics of “right” and “wrong” examples from the thousands or millions of samples it is fed during training. All those characteristics are assigned a weight for their importance. When a trained NN is fed a new event, it breaks it down into recognizable characteristics, and based on the weights of those characteristics, it decides how to react to the event.” https://cleantechnica.com/2020/1 1/2 1/tesla-dojo-supercomputer-explained-how-to-make-full-self-driving-ai/ (emphasis added) “Tesla extended the reduced precision support further, and introduced the Configurable Float8 (CFloat8), an 8-bit floating point format, to further reduce the enormous pressure on memory storage and bandwidth in storing the weights, activations, and gradient values necessary for training the increasingly larger networks.” https://tesla-cdn.thron.com/static/MXMU3S_tesla-dojo-technology_1WDVZN.pdf?xseo=&response-content- disposition=inline%3B filename%3D%22tesla-dojo-technology.pdf%22 (emphasis added) “This V9 network is a monster, and that’s not the half of it. When you increase the number of parameters (weights) in an NN by a factor of 5 you don’t just get 5 times the capacity and need 5 times as much training data. In terms of expressive capacity increase it’s more akin to a number with 5 times as many digits. So if V8’s expressive capacity was 10, V9’s capacity is more like 100,000. It’s a mind boggling expansion of raw capacity. And likewise the amount of training data doesn’t go up by a mere 5x. It probably takes at least thousands and perhaps millions of times more data to fully utilize a network that has 5x as many parameters.” https://electrek.co/2018/10/15/tesla-new-autopilot-neural-net-v9/ “An artificial neuron receives signals then processes them and can signal neurons connected to it. The ‘signal’ at a connection is a real number, and the output of each neuron is computed by some non-linear function of the sum of its inputs. The connections are called edges. Neurons and edges typically have a weight that 37  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA adjusts as learning proceeds. The weight increases or decreases the strength of the signal at a connection. Neurons may have a threshold such that a signal is sent only if the aggregate signal crosses that threshold.” https://en.wikipedia.org/wiki/Artificial_neural_network (emphasis added) [D] the computer system determines the competence level of the human driver using the sensor readings; See Claim 6[D], supra. [E] the corrective action comprises the computer system using the engine throttle. The computing platform of the Tesla may determine the corrective action comprises using the engine throttle. For example, the computer system may determine by weighted voting to implement the corrective action of shutting off the engine throttle, thus decelerating the vehicle. This may occur, for example, if the driver has repeatedly ignored notifications to the driver to take over immediately. If you repeatedly ignore Autosteer's prompts to apply slight force to the steering wheel, Autosteer disables for the rest of the drive and displays the following message requesting you to drive manually. If you don't resume manual steering, Autosteer sounds a continuous chime, turns on the warning flashers, and slows the vehicle to a complete stop. Autosteer unavailable for the rest of this drive. Hold steering wheel to drive manually. For the rest of the drive, you must steer manually. Autosteer is available again on your next drive (after you stop and shift Model 3 into Park). Tesla Model 3 Owner’s Manual at 86. 38  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA The Full Self Driving Functions often either engage or disengage the throttle to resolve any number of situations that may arise while operating the SDV. See, e.g., https://www.youtube.com/watch?v=rwP W2z6gcDM (testing FSD Beta in an obstacle course). Further evidence of this was demonstrated through testing the vehicle itself. Claim 9 9. A computer program product for controlling a driving mode of a self- driving vehicle (SDV), the computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith the program code readable and executable by a processor to perform a method comprising: See Claim 4[C], supra. [A] determining a competence level of a human driver wherein the See Claim 4[I], supra. 39  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA competence level describes a competence level of the human driver in controlling the SDV; [B] receiving sensor readings from a system of sensors about the competence level of the human driver, wherein the SDV is operable to provide autonomous control of driver controls comprising: engine throttle, steering mechanism, braking system, and navigation; See Claim 4[PRE], [B], [I], supra. [C] determining a competence level of a processor; See Claim 4[H], supra. [D] determining a corrective action; See Claim 4[J], supra. [E] the SDV implementing the corrective action wherein: See Claim 4[K], supra. 40  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA [F] the sensor readings comprise information about a current weather condition of a roadway on which the SDV is currently traveling; Tesla cars include a suite of sensors that collect sensor readings. See Claim 4[A], supra. One specific type of sensor readings these sensors collect is about the weather conditions occurring on the roadway on which the Tesla is currently traveling. For example, this article details “what Tesla Autopilot can see in a rainstorm.” https://electrek.co/2019/05/07/tesla-autopilot-see-rain-storm/ The article explains “[o]ne of the main concerns with self-driving vehicles is how they will react to different climates and weather conditions.” See also https://www.youtube.com/watch?v=sRCxSagclxM at 2:00 (Tesla disengages FSD and alerts the human driver to take over immediately in snowy conditions). [G] determining a corrective action comprises determining whether a fault has occurred and whether the fault exceeds a threshold for danger; A Tesla’s Full Self Driving Functions are reliant on sensor readings. See Claim 4[D], supra. In situations where the weather conditions are adverse, a sensor may become obstructed, misfunction, become damaged, or fail to collect sufficiently reliable inputs. In this situation, the computing platform must determine: (1) whether one of these situations has occurred (i.e., a fault has occurred, see Claim 4[G], supra); and (2) whether the fault exceeds a threshold for danger. In this circumstance, the threshold for danger includes whether the sensors can collect enough data to continue to operate one of the Full Self Driving Functions despite the fault. Evidence of this is based on, in circumstances where the fault of obstructed, misfunctioning, damaged, or otherwise failed sensors create a situation where sensor data is insufficient to continue operation, the Full Self Driving Functions are no longer available, and control is transferred back to the driver. See Claim 4[G], supra. Further evidence of this can be demonstrated through testing the vehicle itself. [H] the sensor readings comprise a reading from a GPS sensor; See Claim 4[D], supra. [I] determining whether the fault exceeds the threshold for danger comprises weighted voting; See Claims 8[C] (weighted voting), 9[G] (threshold for danger), supra. 41  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA [J] a weighted voting parameter comprises active learning data, said active learning data including information from other SDVs. See Claims 6[C] (active learning), 8[C] (weighted voting), supra. 42  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Exhibit B — U.S. Pat. No. 11,738,765 v. Tesla Current Tesla Models (including Models S, 3, X, and Y) include hardware capable of executing downloadable software that infringes the claims of U.S. Pat. No. 11,738,765 (“the ’765 Patent’’). Specifically, two levels of software packages include infringing functionality: (1) Enhanced Autopilot, which features the infringing functionality known as “Navigate on Autopilot”; and (2) Full Self Driving Capability, which includes the infringing functionality known as “Autosteer on City Streets” (which is currently being tested through the public “Full Self Driving Beta Test,” otherwise known as “FSD Beta”). The infringing functionality is referred to collectively in this chart as “Full Self Driving Functions.” Claim 1 1. A computer program product for controlling a driving mode of a self-driving vehicle (SDV), the computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, the program code readable and executable by a processor to perform a method comprising: This preamble is likely non-limiting. To the extent the preamble is limiting, Tesla vehicles including Models S, 3, X, and Y (collectively, “Teslas”) are examples of self-driving vehicles that include a computer program product for controlling a driving mode of a self-driving vehicle (SDV), the computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, the program code readable and executable by a processor to perform a method. Specifically accused herein are Teslas which have access to the software packages known as “Enhanced Autopilot” and “Full Self Driving Capability.” For $6,000 at purchase, Tesla owners can unlock “Enhanced Autopilot”, which includes a Full Self Driving Function called “Navigate on Autopilot” that autonomously navigates the vehicle from the on ramp of a freeway to a desired exit, where the driver takes back over to complete the rest of the trip. 43  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Enhanced Autopilot $6,000 . e Auto Lane Change e Autopark « Summon e Smart Summon https://www.tesla.com/models/design#overview” When using Autosteer on a controlled-access highway (a main highway on which road users enter and exit using on-ramps and off-ramps). Navigate on Autopilot guides Model S to off-ramps and interchanges based on your navigation route. Along the highway portion of a navigation route, Navigate on Autopilot also changes lanes to prepare for exits (route-based lane changes) and to minimize the driving time to your destination (speed-based lane changes). https://www.tesla.com/ownersmanual/2012_ 2020 models/en_us/GUID-0535381F-643F-4C60-85AB- 1783E723B9B6.html For $15,000 at purchase or $200 a month, a Tesla owner can unlock “Full Self-Driving Capability”, which promises to extend the Full Self Driving Function Navigate on Autopilot beyond the highway and onto city streets. 9 All references to one Tesla Model apply equally to all other Tesla models. 44  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Full Self-Driving Capability $15,000 e All functionality of Basic Autopilot and Enhanced Autopilot e Traffic Light and Stop Sign Control Coming Soon « Autosteer on city streets https://www.tesla.com/models/design#overview Subscription Pricing Your vehicle’s current Autopilot package of Basic Autopilot or Enhanced Autopilot will determine the FSD capability subscription price. Basic Autopilot to FSD capability $199.00 per month Enhanced Autopilot to FSD capability $99.00 per month https://www.tesla.com/support/full-self-driving-subscriptions Owners who either purchase or subscribe to FSD Capability are also eligible to apply for access to the “Full Self-Driving Beta” program, which is currently testing and collecting data on FSD capability on city streets. 45  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA v Can! request Full Self-Driving Beta if | am subscribed to Tesla Full Self-Driving capabilities? Yes. As long as you have the option to request Full Self-Driving Beta from your vehicle’s touchscreen, you are eligible to enroll regardless of whether you have purchased Tesla Full-Self Driving capabilities with a one-time payment or subscription. To view if you have access to Full Self-Driving Beta, select ‘Controls’ > ‘Autopilot’ > ‘Request Full Self-Driving Beta.’ https://www.tesla.com/support/full-self-driving-subscriptions Tesla released the beta testing version of its new Full Self-Driving suite last week, and it has already started pulling a lot of data from the vehicles using the feature. The company mentioned that the data used from this beta testing will be used for improving the neural networks for the self-driving features. The amount of data they got is so high, that they are already planning an update to the test version. https://www.vehiclesuggest.com/tesla-collecting-huge-amount-of-data-through-fsd-testing/ (dated October 26, 2020) The equipment on Teslas includes a variety of sensors (e.g., cameras, radar, and sonar) and a computing platform specially designed to perform the computations required to perform the Full Self Driving Functions. The most recent version of this hardware—which is required for FSD Capability—is called “Hardware 3” (alternatively known as “HW37”) and includes the following equipment: 46  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Cameras: Eight cameras covering all angles. Sensors: Continental Radar with 558 ft range & 12 Sonar Sensors with 26 ft range. Computers: Two bespoke Tesla-designed units. https://www.currentautomotive.com/the-ultimate-guide-to-tesla-autopilot/ Note: Depending on the specific model, some cars may lack radar or both radar and sonar, depending on their date of manufacture. All cars with HW3 come with cameras and the Tesla designed FSD computer. Safety is at the core of our design and engineering decisions. In 2021, we began our transition to Tesla Vision by removing radar from Model 3 and Model Y, followed by Model S and Model X in 2022. Today, in most regions around the globe, these vehicles now rely on Tesla Vision, our camera-based Autopilot system. https://www.tesla.com/en_eu/support/transitioning-tesla- vision#:~:text=In%202021%2C%20we%20began%20our,our%20camera%2Dbased%20Autopilot%20system. Hardware 4 (or “HW4”’) is expected to include similar equipment, with reintegration of radar (standard, as opposed to varying from car to car), and a more powerful computing platform. https://electrek.co/2023/02/15/tesla-self-driving-hw4-computer-leaks-teardown/ Tesla cars equipped with HW3 and HW4 each have a computing platform that combines a processor coupled to a non-transitory computer readable storage medium containing the program code relating to the Full Self Driving Functions’ software, which is readable and executable by the processor of the computing platform. 47  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA HW3 utilizes a specialized computer system designed in-house at Tesla. The silicon dies for processors for this computer system are manufactured by Samsung Austin Semiconductor. HW3 comprises at least two processors, highlighted here: HW3 also contains one or more non-transitory computer readable storage media coupled to these processors. These computer readable media store the program code for the FSD and Navigate on Autopilot features, and this program code is readable and executable by the processor(s). https://www.autopilotreview.com/tesla-custom-ai-chips-hardware-3/ The so-called “FSD Computer” (part of HW3) was lauded by Elon Musk himself as a major advancement over the previously utilized Nvidia chips. https://www.youtube.com/watch?v=NJVcsvQ30AQ Tesla’s Full Self Driving Compatible software requires the FSD Computer. 48  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Hardware upgrades to the Full Self-Driving computer are not included with Full Self- Driving capability subscriptions. To be eligible for FSD capability subscriptions, the FSD computer must be installed in your vehicle. To install the FSD computer, schedule an installation appointment from the Tesla app. https://www.tesla.com/support/full-self-driving-subscriptions [A] receiving sensor readings from a system of sensors, wherein the sensor readings describe a current operational state of a SDV; All Tesla vehicles have a set of sensors, comprising at least several cameras. See Claim 1[PRE] (discussing HW3 and HW4). Each of these hardware packages necessarily includes sensor systems that collect and relay information to the FSD computer. 49  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA How It Works Your Model 3 includes the following components that actively monitor the surrounding area: 1. Acamera is mounted above the rear license plate. 2. Ultrasonic sensors (if equipped) are located in the front and rear bumpers. 3. A camera is mounted in each door pillar. 4. Three cameras are mounted to the windshield above the rear view mirror. 5. A camera is mounted to each front fender. 6. Radar (if equipped) is mounted behind the front bumper. Mode! 3 is also equipped with high precision electronically-assisted braking and steering systems. NOTE: Ensure all cameras and sensors (if equipped) are clean before each drive. See Cleaning Cameras and Sensors on page 79 for more information. Dirty cameras and sensors, as well as environmental conditions such as rain and faded lane markings, can affect Autopilot performance. Tesla 3 Owner’s Manual at 77. In addition to these sensors and more typical vehicle sensors (e.g., speedometer, GPS, etc.), Tesla cars also have camera sensors and steering wheel sensors that monitor the status of the driver: 50  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA The cabin camera can determine driver inattentiveness and provide you with audible alerts, to remind you to keep your eyes on the road when Autopilot is engaged. Tesla 3 Owner’s Manual at 86, 115. Hold Steering Wheel Autosteer determines how best to steer Model 3. When active, Autosteer requires you to hold the steering wheel. If it does not detect your hands on the steering wheel for a period of time, a flashing blue light appears at the top of the car status section of the touchscreen and the following message displays: Apply slight turning force to steering wheel Autosteer detects your hands by recognizing slight resistance as the steering wheel turns, or from you manually turning the steering wheel very lightly (without enough force to take over steering). Autosteer also qualifies your hands as being detected if you engage a turn signal or use a button or scroll wheel on the steering wheel. NOTE: When your hands are detected, the message disappears and Autosteer resumes normal operation. The computing platforms included in HW3 and HW4 are each capable of, and reliant on, receiving sensor readings from the system of sensors. 51  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Rearward Looking Side Cameras Wide Forward Camera Main Forward Camera Narrow Forward Camera Max distance 100m Max distance 60m Max distance 150m Max distance 2501 - Rear View Camera Forward Looking Side Cameras Max distance 50m Max distance 80m Tesla Vision To make use of a camera suite this powerful, each Tesla car has a powerful set of vision processing tools developed by Tesla. Built on a deep neural network, Tesla Vision deconstructs the car's environment at greater levels of reliability than those achievable with classical vision processing techniques. https://www.tesla.com/autopilot 52  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Neural Networks Apply cutting-edge research to train deep neural networks on problems ranging from perception to control. Our per-camera networks analyze raw images to perform semantic segmentation, object detection and monocular depth estimation. Our birds-eye-view networks take video from all cameras to output the road layout, static infrastructure and 3D objects directly in the top-down view. Our networks learn from the most complicated and diverse scenarios in the world, iteratively sourced from our fleet of millions of vehicles in real time. A full build of Autopilot neural networks involves 48 networks that take 70,000 GPU hours to train @. Together, they output 1,000 distinct tensors (predictions) at each timestep. https://www.tesla.com/en_eu/AI The computing platforms take the information collected from the sensors and use it to recreate the world around the car. https://www.pemag.com/news/tesla-is-developing-a-self-driving-system-that-only-uses-cameras See also https://youtu.be/eOL_rCK59Z1?t=28831 Further evidence that the Full Self Driving Functions are reliant on receiving a sensor reading from the system of sensors is that the functions become unavailable if the sensors are malfunctioning, obstructed, or damaged: 33  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Limitations Many factors can impact the performance of Autopilot components, causing them to be unable to function as intended. These include (but are not limited to): * Poor visibility (due to heavy rain, snow, fog, etc.). * Bright light (due to oncoming headlights, direct sunlight, etc.). + Damage or obstructions caused by mud, ice, snow, etc. * Interference or obstruction by object(s) mounted onto the vehicle (such as a bike rack). * Obstruction caused by applying excessive paint or adhesive products (such as wraps, stickers, rubber coating, etc.) onto the vehicie. Traffic signs that do not conform to standard recognizable formats, such as digital or temporary speed signs. * Narrow or winding roads. + A damaged or misaligned body panel. Use of gray or aftermarket glass. Interference from other equipment that generates ultrasonic waves. Extremely hot or cold temperatures. Tesla Model 3 Owner’s Manual at 78-79. In situations where Autosteer is temporarily unavailable, the Autosteer icon disappears. For example, your driving speed is not within the speed required for Autosteer to operate. Autosteer may also be unavailable if it is not receiving adequate data from the camera(s). Tesla Model 3 Owner’s Manual at 85. 54  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA * Bright light (such as direct sunlight) is interfering with the view of the camera(s). * Model 3 is being driven very close to a vehicle in front of it, which is blocking the view of the camera(s). Tesla Model 3 Owner’s Manual at 88. The computing platform is able to use data from said system of sensors to determine a variety of details about the operational state of the vehicle, such as: (1) its location and direction (via GPS and cameras); (2) its current speed (via speedometer and cameras); (3) the state of the driver (through the cabin camera and the steering wheel sensors); (4) the state of the road the vehicle is traveling on (through cameras); and (5) the location of things around it (through cameras and, on some models, radar and sonar). The display below shows the current speed on the top left, the state of the road and the orientation of the car on it on the left, and the location of the vehicle via GPS on the right. 55  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA 4 ee e #0 00: seer j Sybawson Oo FE Goreme ce oe Be nesice eG 2 Roy: zi Stonewall Jackson 1oe's Stegghote @ f = Loe a S SSE 40% Ri pea) (TTT https://teslamotorsclub.com/tme/threads/why-didnt-tesla-put-the-map-on-the-left-side-of-the-ui.257818/ Tesla displays various types of objects around the car as detected by the sensors. https://www.notateslaapp.com/tesla-reference/636/all-tesla-fsd-visualizations-and-what-they-mean The Full Self Driving Functions may only be available on certain roads: 56  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA When using Autosteer on a controlled-access highway (a main highway on which road users enter and exit using on-ramps and off-ramps). Navigate on Autopilot guides Mode! 3 to off-ramps and interchanges based on your navigation route. Along the highway portion of a navigation route, Navigate on Autopilot also changes lanes to prepare for exits (route-based lane changes) and to minimize the driving time to your destination (speed-based lane changes). Tesla Model 3 Owner’s Manual at 90. More generally, the Functions are only available under certain conditions: The speed at which you can initiate Autosteer can vary depending on various conditions and whether or not a vehicle is detected ahead of you. When no vehicle is detected ahead of you, you must be driving at least 18 mph (30 km/h), unless certain vehicle and environmental conditions are met, in which case, you may be able to initiate it at lower speeds. When a vehicle is detected ahead of you, you can initiate Autosteer at any speed, even when stationary, provided Model 3 is at least 5S feet (150 cm) behind the detected vehicle. Tesla Model 3 Owner’s Manual at 85. In other conditions, the Functions may not be available at all: In situations where Autosteer is temporarily unavailable, the Autosteer icon disappears. For example, your driving speed is not within the speed required for Autosteer to operate. Autosteer may also be unavailable if it is not receiving adequate data from the camera(s). Tesla Model 3 Owner’s Manual at 85. Of course, Tesla requires the driver’s hands to be on the wheel at all times: 57  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Autosteer requires that you pay attention to your surroundings and remain prepared to take control at any time. If Autosteer still does not detect your hands on the steering wheel, the flashing light on the car status section of the touchscreen increases in frequency and a chime sounds. If you repeatedly ignore Autosteer’s prompts to apply slight force to the steering wheel, Autosteer disables for the rest of the drive and displays the following message requesting you to drive manually. If you don't resume manual steering, Autosteer sounds a continuous chime, turns on the warning flashers, and slows the vehicle to a complete stop. Autosteer unavailable for the rest of this drive, Hold steering wheel to drive manually. For the rest of the drive, you must steer manually. Autosteer is available again on your next drive (after you stop and shift Model 3 into Park). Tesla Model 3 Owner’s Manual at 86. Vehicle testing suggests this limitation is met: the Full Self Driving Functions are able to tell if a sensor is not transmitting data or if the data is insufficient or incomplete. For example, when cameras were covered (and thus incapable of collecting sensor data), the Full Self Driving Functions were not accessible. This indicates that the Full Self Driving Functions were determining the operational state of the vehicle from the (lack of) sensor data. [B] determining based on the sensor readings, by one or more processors, During the operation of the vehicle, one or more faults may arise that limit the functionality of the Full Self Driving Functions. A non-exhaustive list of faults is set forth below: 58  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA whether a fault has occurred; Limitations Many factors can impact the performance of Autopilot components, causing them to be unable to function as intended. These include (but are not limited to): * Poor visibility (due to heavy rain, snow, fog, etc.). * Bright light (due to oncoming headlights, direct sunlight, etc.. + Damage or obstructions caused by mud, ice, snow, etc. + Interference or obstruction by object(s) mounted onto the vehicle (such as a bike rack). Obstruction caused by applying excessive paint or adhesive products (such as wraps, stickers, rubber coating, etc.) onto the vehicle. * Traffic signs that do not conform to standard recognizable formats, such as digital or temporary speed signs. * Narrow or winding roads. + A damaged or misaligned body panel. + Use of gray or aftermarket glass. * Interference from other equipment that generates ultrasonic waves. + Extremely hot or cold temperatures. Tesla Model 3 Owner’s Manual at 78-79. Most of these limitations are based on obstruction or damage to the sensors. Other faults may occur. A WARNING: The list above does not represent an exhaustive list of situations that may interfere with proper operation of Autopilot components. Never depend on these components to keep you safe. It is the driver's responsibility to stay alert, drive safely, and be in control of the vehicle at all times. 59  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Tesla Model 3 Owner’s Manual at 79. For example, faults may be based on the behavior of the driver. The Full Self Driving Functions are not fully autonomous, but instead specify that the driver must remain alert and be prepared to take over at any time. Thus, Tesla cars are designed so that the drivers keep their hands on the wheel: WARNING: Autosteer is a hands-on feature. You must keep your hands on the steering wheel at all times. WARNING: Autosteer is intended for use on controlled-access highways with a fully attentive driver. When using Autosteer, hold the steering wheel and be mindful of road conditions and surrounding traffic. Do not use Autosteer in construction zones, or in areas where bicyclists or pedestrians may be present. Never depend on Autosteer to determine an appropriate driving path. Always be prepared to take immediate action. Failure to follow these instructions could cause damage, serious injury or death. Autosteer determines how best to steer Model 3. When active, Autosteer requires you to hold the steering wheel. If it does not detect your hands on the steering wheel for a period of time, a flashing blue light appears at the top of the car status section of the touchscreen and the following message displays: Apply slight turning force to steering wheel Autosteer detects your hands by recognizing slight resistance as the steering wheel turns, or from you manually turning the steering wheel very lightly (without enough force to take over steering). Autosteer also qualifies your hands as being detected if you engage a turn signal or use a button or scroll wheel on the steering wheel. Tesla Model 3 Owner’s Manual at 85-86. Similarly, the car is capable of determining a number of other contextual faults: 60  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA A A CAUTION: If available in your market region, Model 3 detects lights from an emergency vehicle when using Autosteer at night on a high speed road, the driving speed is automatically reduced and the touchscreen displays a message informing you of the slowdown. You will also hear a chime and see a reminder to keep your hands on the steering wheel. When the light detections pass by or cease to appear, Autopilot resumes your cruising speed. Alternatively, you may tap the accelerator to resume your cruising speed. WARNING: Never assume that your ability to see a traffic light, stop sign, or road marking (especially at a complex intersection, or an intersection in which a traffic light or sign is partially obstructed, etc.) means that Model 3 can also see it and respond appropriately. WARNING: Even the most recent map data does not include all traffic lights and stop signs. Therefore, Traffic Light and Stop Sign Control relies heavily on the ability of the cameras to detect traffic lights, stop signs, road markings, etc. As a result, Model 3 may ignore an intersection that is blocked from the camera's view (for example, obstructed by a tree or a large vehicle or object, or located near a steep hill or sharp curve). WARNING: Traffic Light and Stop Sign Control is not a substitute for attentive driving and sound judgment. Tesla Model 3 Owner’s Manual at 86, 94. Canceling Autosteer Autosteer cancels when: * You press the brake pedal. * You start steering manually. + You exceed the maximum speed at which Autosteer operates - 90 mph (150 km/h). + You move the drive stalk upwards. + A door is opened. + An Automatic Emergency Braking event occurs (see Collision Avoidance Assist on page 111). WARNING: In some situations, Traffic Light and Stop Sign Control may inaccurately detect a traffic light or stop sign, causing Model 3 to slow down unexpectedly. Be prepared to take immediate action at all times. 61  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Tesla Model 3 Owner’s Manual at 115. Vehicle testing confirmed that this limitation is present in the accused models. Vehicle testing suggests this limitation is met: the Full Self Driving Functions are able to tell if a sensor is not transmitting data or if the data is insufficient or incomplete. For example, when cameras were covered (and 62  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA thus incapable of collecting sensor data), the Full Self Driving Functions were not accessible. This indicates the Full Self Driving Functions were able to determine a fault had occurred. [C] determining, by the one or more processors, whether the fault exceeds a threshold for danger; When the Full Self Driving Functions determine a fault exists (see Claim 1[B]), they also determine whether the whether the fault exceeds the threshold for danger. For example, the threshold for danger may include whether the sensors (see Claim 1[A]) can collect enough data to continue to operate one of the Full Self Driving Functions despite the fault. Evidence of this is based on, in circumstances where the fault of obstructed, misfunctioning, damaged, or otherwise failed sensors create a situation where sensor data is insufficient to continue operation, the Full Self Driving Functions are no longer available, and control is transferred back to the driver. See Claim 1[B], supra. [D] determining a corrective action associated with the fault using a fault- remediation table; The Full Self Driving Functions determine a corrective action associated with the fault using a fault remediation table. The Full Self Driving Functions utilize a neural network to perform the complicated decision-making processes that occur during driving a vehicle. “Our networks learn from the most complicated and diverse scenarios in the world, iteratively sourced from our fleet of millions of vehicles in real time. A full build of Autopilot neural networks involves 48 networks that take 70,000 GPU hours to train O. Together, they output 1,000 distinct tensors (predictions) at each timestep.” https://www.tesla.com/en_eu/AI “Andrej Karpathy explains the active learning procedure at Tesla, which they call the Data Engine. For example, in an object detection task and for a bike attached to the back of a car, the neural network should detect just one object (car) for downstream tasks such as decision-making and planning. ... They find a few images that show this pattern and use a machine learning mechanism to search for similar examples in their fleet to fix this problem. ... Then human annotators will annotate these examples as single cars, and the neural network will be trained on these new examples. So, in the future, the object detector will understand that it is just an attached bike to a car and consider that as just a single car. They do this all the time for all the rare cases. So their model will become more and more accurate over time. After collecting some initial data, the models are trained. Then, wherever the model is uncertain, or there is human intervention or disagreement between the human behavior and the model output, which is running in shadow mode, the data will be selected to be annotated by humans, and the model will be trained on that data.” 63  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA https://medium.com/aiguys/active-learning-and-data-auto-labeling-in-autonomous-driving- 5d6bec956a38#b36a An example of these neural networks at work is the occurrence of a fault. See Claim 1[B]. When a fault occurs, the Full Self Driving Programs are trained to implement a specific corrective action which was instilled during the training phase. This is akin to cross-referencing a table in a database, where a specific fault can be cross-referenced with specific corrective action. Specific corrective actions correspond to specific faults. For example, with reference to the faults of Claim 1[B], the corrective action may be to issue a warning to the driver that Autopilot is degraded, to pay attention or hold the wheel, or to take over immediately. The corrective action may also be to decelerate the vehicle and pull over to the side of the road. Vehicle testing suggested this limitation was met: upon occurrence of a fault, the Full Self Driving Functions would determine the above-discussed corrective actions. [E] the SDV implementing the corrective action; Once the proper corrective action is determined, the Tesla implements the corrective action. See Claim 1[D]. Take Over Immediately In situations where Autosteer is unable to steer Model 3, Autosteer sounds a warning chime and displays the following message on the touchscreen: Take over immediately When you see this message, TAKE OVER STEERING IMMEDIATELY. Tesla Model 3 Owner’s Manual at 86. 64  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA As stated earlier, when the computing platform of the Tesla determines the proper corrective action is to hand control of the vehicle back to the driver, it can do so. When the computing platform of the Tesla determines the proper corrective action is to alert the driver to pay attention, it can do so. And when the computing platform of the Tesla determines the proper corrective action is to decelerate the vehicle and pull it over, it can do so. Additionally, Full Self Driving Functions implement many corrective actions while operating in the respective self-driving modes. See, e.g., https://www.youtube.com/watch?v=rwP W2z6gcDM (testing FSD Beta in an obstacle course). Vehicle testing suggested this limitation was met: upon occurrence of a fault, the Full Self Driving Functions would implement the above-discussed corrective actions. [F] wherein, the sensor readings comprise a reading from a GPS sensor; See Claim 1[A] (discussing the GPS sensor and its associated readings). [G] the fault comprises a current weather condition of the roadway on which the SDV is currently traveling; Tesla cars include a suite of sensors that collect sensor readings. See Claim 1[A], supra. One specific type of sensor readings these sensors collect is about the weather conditions occurring on the roadway on which the Tesla is currently traveling. For example, this article details “what Tesla Autopilot can see in a rainstorm.” https://electrek.co/2019/05/07/tesla-autopilot-see-rain-storm/ The article explains “[o]ne of the main concerns with self-driving vehicles is how they will react to different climates and weather conditions.” Upon collecting this data, the Full Self Driving Programs may determine that the weather condition of the roadway on which the SDV is currently traveling comprises a vehicle fault: this is because the weather conditions on the road may impair one or more sensors or create a situation where it is hard for the computer to control the vehicle. See Claim 1[A]. 65  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Limitations Many factors can impact the performance of Autopilot components, causing them to be unable to function as intended. These include (but are not limited to): * Poor visibility (due to heavy rain, snow, fog, etc.). * Bright light (due to oncoming headlights, direct sunlight, etc.). + Damage or obstructions caused by mud, ice, snow, etc. * Interference or obstruction by object(s) mounted onto the vehicle (such as a bike rack). * Obstruction caused by applying excessive paint or adhesive products (such as wraps, stickers, rubber coating, etc.) onto the vehicie. Traffic signs that do not conform to standard recognizable formats, such as digital or temporary speed signs. * Narrow or winding roads. + A damaged or misaligned body panel. Use of gray or aftermarket glass. Interference from other equipment that generates ultrasonic waves. Extremely hot or cold temperatures. Tesla Model 3 Owner’s Manual at 78-79. In situations where Autosteer is temporarily unavailable, the Autosteer icon disappears. For example, your driving speed is not within the speed required for Autosteer to operate. Autosteer may also be unavailable if it is not receiving adequate data from the camera(s). Tesla Model 3 Owner’s Manual at 85. 66  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA * Bright light (such as direct sunlight) is interfering with the view of the camera(s). * Model 3 is being driven very close to a vehicle in front of it, which is blocking the view of the camera(s). Tesla Model 3 Owner’s Manual at 88. [H] determining whether the fault exceeds a threshold for danger comprises determining a control processor competence level; As mentioned previously, the Full Self Driving Programs determine whether the fault exceeds a threshold for danger. See Claim 1[C]. One input to that determination is determining a control processor competence level. The computing platform of the Tesla weighs the collected sensor data and determines the level of competence the processor has in the current situation. For example, if the degree of difficulty is low, sufficient amounts of data can be collected, and the processor is well-trained, the computer system will determine the processor has a high competence level. If, on the other hand, the situation is more complex and complicated, the data collected is missing or unintelligible, and/or the processor is poorly trained, the computer system will determine the processor has a low competence level. When competence levels are high, the vehicle is able to conduct the Full Self Driving Functions without issue. For example, consider the following video which documents a fairly smooth drive when conditions are ideal: https://www.youtube.com/watch?v=gNRIf-UXunU When processor competence levels are low, the vehicle is likely to determine the fault exceeds a threshold for danger, and the self-driving functions may become unavailable. See Claim 1[C]-[E], supra. [I] the corrective action comprises transferring driver controls to manual control and alerting a human driver to The computing platform of the Tesla is capable of implementing the corrective action from Claim 1[D]-[E], supra, including handing control back to the driver. 67  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA take over immediately. Take Over Immediately In situations where Autosteer is unable to steer Model 3, Autosteer sounds a warning chime and displays the following message on the touchscreen: Take over immediately When you see this message, TAKE OVER STEERING IMMEDIATELY. Tesla Model 3 Owner’s Manual at 86. See also https://www.youtube.com/watch?v=sRCxSagclxM at 2:00 (Tesla disengages FSD and alerts the human driver to take over immediately in snowy conditions). Claim 3 3. [PRE] The computer program product of claim 1, further comprising: See Claim |[PRE]-[]], supra. [A] determining the control processor competence level comprises using a weighted voting system, the weighted voting system comprising: The Full Self Driving Programs determine the control processor competence level. See Claim 1[H], supra. As stated previously, the Full Self Driving Functions utilize a neural network to perform the complicated decision-making processes that occur during driving a vehicle. See Claim 1[D], supra. The Tesla neural network uses weighted voting to determine the control processor competence level. “Each cycle, 256 bytes of activation data and an additional 128 bytes of weight data is read from the SRAM into the MACs array where they are combined. Each NPU has a 96x96 multiply-accumulate array for a total of 9,216 MACs and 18,432 operations. ... Under normal operation, the neural network program is loaded at 68  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA the start and is kept in memory for the entire duration in which the chip is powered. Running is done by setting the input buffers address (e.g., newly taken image sensor photo), setting the output buffer address, and weight buffer address (e.g., network weights), set the program address, and run. The NPU will asynchronously run on its own the entire neural network model until reaching a stop instruction which triggers an interrupt, letting the CPU post-process the results.” https://en.wikichip.org/wiki/tesla_(car_company)/fsd_chip (emphasis added) “The behavior of a NN is not programmed. Just like a biological NN, it is trained by experience. A NN program without the training is good for nothing. It extracts the characteristics of “right” and “wrong” examples from the thousands or millions of samples it is fed during training. All those characteristics are assigned a weight for their importance. When a trained NN is fed a new event, it breaks it down into recognizable characteristics, and based on the weights of those characteristics, it decides how to react to the event.” https://cleantechnica.com/2020/1 1/2 1/tesla-dojo-supercomputer-explained-how-to-make-full-self-driving-ai/ (emphasis added) “Tesla extended the reduced precision support further, and introduced the Configurable Float8 (CFloat8), an 8-bit floating point format, to further reduce the enormous pressure on memory storage and bandwidth in storing the weights, activations, and gradient values necessary for training the increasingly larger networks.” https://tesla-cdn.thron.com/static/MXMU3S_tesla-dojo-technology_1WDVZN.pdf?xseo=&response-content- disposition=inline%3Bfilename%3D%22tesla-dojo-technology.pdf%22 (emphasis added) “This V9 network is a monster, and that’s not the half of it. When you increase the number of parameters (weights) in an NN by a factor of 5 you don’t just get 5 times the capacity and need 5 times as much training data. In terms of expressive capacity increase it’s more akin to a number with 5 times as many digits. So if V8’s expressive capacity was 10, V9’s capacity is more like 100,000. It’s a mind boggling expansion of raw capacity. And likewise the amount of training data doesn’t go up by a mere 5x. It probably takes at least thousands and perhaps millions of times more data to fully utilize a network that has 5x as many parameters.” https://electrek.co/2018/10/15/tesla-new-autopilot-neural-net-v9/ “An artificial neuron receives signals then processes them and can signal neurons connected to it. The ‘signal’ at a connection is a real number, and the output of each neuron is computed by some non-linear function of 69  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA the sum of its inputs. The connections are called edges. Neurons and edges typically have a weight that adjusts as learning proceeds. The weight increases or decreases the strength of the signal at a connection. Neurons may have a threshold such that a signal is sent only if the aggregate signal crosses that threshold.” https://en.wikipedia.org/wiki/Artificial neural_network (emphasis added) [1] a first plurality of inputs and a first plurality of weights, wherein the first plurality of inputs comprise sensor readings from a first camera sensor; and As stated previously, the Full Self Driving Programs implement sensor readings from the Tesla’s suite of sensors to inform operation of the vehicle, including the sensor readings from the set of cameras. See Claim 1[A], supra. These sensor readings constitute the a first plurality of inputs to the neural network. The neural network then applies a first plurality of weights to said inputs. See Claim 3[A], supra. [B] multiplying at least one of the first plurality of inputs by a weight from among the first See Claim 3[A][1], supra (“The neural network then applies a first plurality of weights to said inputs.” (citing Claim 3[A], supra)). weights are based on active learning data; and | plurality of weights; [1] wherein the | As stated previously, the Tesla neural network uses weighted voting to determine the control processor first plurality of competence level. The weights applied to the input by the neural network are determined, at least in part, based on the training using active learning data from other SDVs. See Claim 3[A], supra. [2] wherein the active learning data includes weather condition data from a cohort of other SDVs, wherein the weather condition data shares one or As stated previously, the Tesla weights applied to the input by the Tesla neural network are determined, at least in part, based on the training using active learning data from other Teslas. See Claim 3[A], supra. The other Teslas from which the active learning data is collected constitute a cohort of other SDVs. The data collected from these other Teslas includes weather condition data. 70  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA more traits with the current weather condition of the roadway on which the SDV is currently traveling. As previously discussed generally, see Claim 1[D], supra, the Tesla neural network has been previously trained on this active learning data so that it may be able to determine what to do when it encounters a current condition with one or more shared traits. This behavior is supported by the situations discussed with reference to Claim 1[B], supra. Several weather related faults were discussed, such as “heavy rain, snow, fog . . . mud, ice, snow . . . extremely hot or cold temperatures”: Limitations Many factors can impact the performance of Autopilot components, causing them to be unable to function as intended. These include (but are not limited to): * Poor visibility (due to heavy rain, snow, fog, etc.). * Bright light (due to oncoming headlights, direct sunlight, etc.). + Damage or obstructions caused by mud, ice, snow, etc. * Interference or obstruction by object(s) mounted onto the vehicle (such as a bike rack). + Obstruction caused by applying excessive paint or adhesive products (such as wraps, stickers, rubber coating, etc.) onto the vehicle. * Traffic signs that do not conform to standard recognizable formats, such as digital or temporary speed signs. * Narrow or winding roads. + A damaged or misaligned body panel. + Use of gray or aftermarket glass. Interference from other equipment that generates ultrasonic waves. Extremely hot or cold temperatures. 71  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Tesla Model 3 Owner’s Manual at 78-79. Each of these situations may constitute a current weather condition with one or more trait in common with the weather condition data collected by other Teslas and present in the active learning data provided as training data to the Tesla neural network. Claim 4 4. [PRE] The computer program product of claim 3, further comprising: See Claim 3[PRE]-[B][2], supra. [B] the SDV autonomously maintains a buffer of space from other vehicles around the SDV and the SDV autonomously controls the steering of the SDV while autonomously controlling the driver controls, without requiring the human driver to operate the driver controls; and When engaged, the Full Self Driving Programs maintains a buffer of space from other vehicles and obstacles around it. 72  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA https://insideevs.com/news/659435/watch-tesla-fsd-beta-v 1 1-obstacle-avoidance-testing/ (showing the Tesla navigating around an “obstacle” (a box thrown into the middle of the road) and creating a buffer area around it and the car); see also https://www.youtube.com/watch?v=rwPW2z6gcDM. [C] when the fault exceeds the threshold for danger, the SDV takes the corrective action. As previously stated, the Tesla determines whether a fault exceeds a threshold for danger. See Claim 1[C], supra. When the fault exceeds the threshold for danger, the Tesla implements the corrective action. See Claim 1[E], supra. 73  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Claim 5 5. [PRE] The computer program product of claim 4, further comprising: See Claim 4[PRE]-[C], supra. [A] determining a competence level of a human driver, comprising using the weighted voting system, the weighted voting system further comprising: As stated previously, the Tesla neural network uses weighted voting to determine the control processor competence level. See Claim 3[A], supra. By the same process, the Tesla neural network uses weighted voting to determine the competence level of the human driver. For example, the Tesla’s computing platform can use sensor data collected from the steering wheel and the cabin camera (discussed with regard to limitation 2[A], supra) to determine the competence level of the human driver—whether the driver is alert and with their hands on the wheel or distracted and ill- prepared to take control of the vehicle. See Claim 1[B], supra. Further evidence of this was demonstrated through testing the vehicle itself. Additionally, the vehicle collects information on the habits of the human driver and calculates a “safety score” which allows for comparisons between two Tesla drivers’ relative degree of safe driving. 74  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Safety Score 90 The Safety Score ®¢t? is an assessment of your driving behavior based on several metrics called Safety Factors. These are combined to estimate the likelihood that your driving could result in a future collision. We combine your daily Safety Scores (up to 30 days) to calculate the aggregated Safety Score, displayed on the main ‘Safety Score’ screen of the Tesla app. You can find details around your daily Safety Score by selecting ‘Daily Details’ at the bottom of the screen. https://www.tesla.com/support/safety-score#version-2.0 Tesla originally used this score to determine the order of drivers who received access to FSD beta. https://www.notateslaapp.com/news/993/tesla-is-pushing-fsd-beta-automatically-to-owners-who-qualify Additionally, Tesla drivers risk losing access to FSD beta for unsafe driving. https://www.teslarati.com/tesla- resets-fsd-beta-strikes-forced-disengagement/ [1] a second plurality of inputs and a second plurality of weights, wherein the second plurality of inputs comprise sensor See Claim 1[A] (disclosing Teslas have eight external cameras and one internal camera), claim 3[A][1], supra. 75  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA readings from a second camera sensor; and [2] multiplying at least one of the second plurality of inputs by a weight from among the second plurality of weights; See Claim 3[A][1], supra (“The neural network then applies a first plurality of weights to said inputs.” (citing Claim 3[A], supra)). [3] wherein the second plurality of weights are based on active learning data that includes information about a cohort of human drivers of other SDVs. As stated previously, the weights applied to the input by the neural network are determined, at least in part, based on the training using active learning data from other SDVs. See Claim 3[B][1], supra. Claim 7 7. [PRE] A computer program product for controlling a driving mode of a self-driving vehicle (SDV), the computer program product comprising a non-transitory computer readable storage medium having program code embodied See Claim 1[PRE], supra. 76  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA therewith, the program code readable and executable by a processor to perform a method comprising: [A] determining a competence level of a human driver, wherein the competence level describes a competence level of the human driver in controlling the SDV; A Tesla’s computing platform is capable of determining competence level of a human driver. See Claim 5[A], supra (discussing how the Full Self Driving Programs determine the human driver competence level). [B] receiving sensor readings from a system of sensors about the competence level of the human driver, wherein the SDV is operable to provide autonomous control of driver controls comprising: engine throttle, steering mechanism, braking system, and navigation; The Full Self Driving Programs receive sensor readings from a system of sensors about the competence level of the human driver, including at least sensor readings from the steering wheel sensors and the cabin camera. See Claim 1[A], supra. Tesla cars include a steering wheel, an engine throttle, and a braking system. These vehicle controls can be operated either by the driver or by the vehicle, depending on the mode of operation. 77  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA il Interior Overview N BRR SS wpenawaw b. V7. 18. Door open button (Opening Doors from the interior on page 14) Tum signal stalk (High Beam Headtights on page $4), Turn Signals on page 54, and Windshield Washers on page 59) . Horn (Horn on page 48) . Drive stalk (How to Shift on page 51, Traffic-Aware Crulse Controt on page 80, Autosteer on page 85) Touchscreen (Touchscreen Overview on page 5) Driver dome light (Lights on page 53) Cabin camera (Cabin Camera on page 115) Hazard warning flashers (Hazard Warning Flashers on page 55) Passenger dome light (Lights on page 53) Climate control vent (see Climate Controls on page 116) Power window switches (Windows on page 16) Manual door release (Opening Doors from the Interior on page 14) Left scroll button (Scroll Buttons on page 47) Brake pedal (Braking and Stopping on page 60) Accelerator pedal (Regenerative Braking on page 61) Right scroll button (Scroll Buttons on page 47) Center consote (interior Storage and Electronics on page 22) Glovebox (Glovebox on page 22) 78  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA Tesla 3 Owner’s Manual at 4. The computing platforms included in HW3 and HW4 are each capable of operating the vehicle controls, including accelerating, braking, and steering. For example, “[the Full Self Driving Function] Navigate on Autopilot [operates the vehicle controls to] guide [the] Model 3 to off-ramps and interchanges based on your navigation route. Along the highway portion of a navigation route, Navigate on Autopilot also [operates the vehicle controls to] change[] lanes to prepare for exits (route-based lane changes) and to minimize the driving time to your destination (speed-based lane changes).” Tesla Model 3 Owner’s Manual at 90. Furthermore, Navigate on Autopilot operates the vehicle controls to do “Speed Based Lane Changes”: * Speed Based Lane Changes: Navigate on Autopilot is designed to perform both route-based and speed- based lane changes. Route-based lane changes are designed to keep you on your navigation route (for example, moving you into an adjacent lane to prepare for an upcoming off-ramp) whereas speed-based lane changes are designed to maintain a driving speed (not to exceed your cruising speed) that allows you to minimize the time it takes to reach your destination (for example, moving into an adjacent lane to pass a vehicle in front of you). Speed-based Tesla Model 3 Owner’s Manual at 90; see also Tesla Model 3 Owner’s Manual at 91 (“If Require Lane Change Confirmation is turned off, Navigate on Autopilot engages the appropriate turn signal, checks for vehicles and objects, and when appropriate, maneuvers Model 3 into the adjacent lane.”). Navigate on Autopilot also has a setting, which if enabled, will operate the vehicle controls to utilize HOV lanes: 79  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA NOTE: When determining navigation routes, and maneuvers at interchanges, Navigate on Autopilot considers whether or not you want to use High Occupancy Vehicle (HOV) lanes. Therefore, ensure the Use HOV Lanes setting is appropriate for your circumstances (see Maps and Navigation on page 142). If the setting is off, Navigate on Autopilot never uses a HOV lane, regardless of time of day. If the setting is on, Navigate on Autopilot uses HOV lanes, whenever applicable. Tesla Model 3 Owner’s Manual at 91. Similarly, HW3 and HW4 performing the Full Self Driving Function “Autosteer on City Streets” (as part of the Full Self Driving Capability package) is capable of operating the vehicle controls, including accelerating, braking, and steering. This video (showing FSD beta 11.3.6) shows how Autosteer on City Streets accelerates, brakes, and steers the vehicle: https://www.youtube.com/watch?v=bH9fD5tB33s Vehicle testing confirmed that this limitation is present in the accused models. [C] determining a competence level of a processor; See Claim 1[H], supra. [D] determining a corrective action; See Claim 1[D], supra. [E] the SDV implementing the corrective action. See Claim 1[E], supra. Claim 10 10. [PRE] The computer program See Claim 7[PRE]-[E], supra. 80  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA product of claim 7, further comprising: [A] determining the competence level of the human driver comprises using a weighted voting system, the weighted voting system comprising: See Claim 5[A], supra. [1] a plurality of inputs and a plurality of weights, the plurality of inputs comprising sensor readings from a camera sensor; and See Claim 1[A] (discussing the cabin camera used to monitor driver behavior), Claim 3[A][1] (discussing sensor inputs and weights), supra. [2] multiplying at least one of the plurality of inputs by a weight from among the plurality of weights; See Claim 3[A][1], supra (“The neural network then applies a first plurality of weights to said inputs.” (citing Claim 3[A], supra)). [3] wherein the plurality of weights are based on active learning data that includes information about a cohort of human drivers of other SDVs. As stated previously, the weights applied to the input by the neural network are determined, at least in part, based on the training using active learning data from other SDVs. See Claim 3[B][1], supra. Claim 11 81  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA 11. [PRE] The computer program product of claim 10, further comprising: See Claim 10[PRE]-[A][3], supra. [A] when the competence level of the human driver is above a first threshold, the SDV autonomously controls the driver controls without requiring the human driver to operate the driver controls; When the competence level of a human driver is above a first threshold, that the driver is paying attention and has his hands on the wheel, the Full Self Driving Programs remain engaged. See Claim 1[B]-[D], supra. [B] the SDV autonomously maintains a buffer of space from other vehicles around the SDV and the SDV autonomously controls the steering of the SDV while the SDV autonomously controls the driver controls, without requiring the human driver to operate the driver controls; See Claim 4[B], supra. [C] when the competence level of the human driver is below a second When the competence level of the human driver falls below a second threshold, such as if they take their eyes off the road to check their phone, or take their hands off the steering wheel, the Full Self Driving Programs determine a fault has occurred. See Claim 1[B] (discussing issuing a warning to the driver), supra. 82  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA threshold, determining that a first fault has occurred; [D] determining the | See Claim 1[D], supra. corrective action comprises determining a first corrective action corresponding to the first fault; and [E] the first See Claim 1[D], supra (discussing corrective actions that do not require the driver to take over, such as issuing corrective action an alert to touch the steering wheel or to pay attention to the road). comprises issuing an alert while the SDV provides autonomous control of the driver controls without requiring the human driver to operate the driver controls. Claim 12 12. [PRE] The See Claim 11[PRE]-[E], supra. computer program product of claim 11, further comprising: [1] when the After the Full Self Driving Programs have issued a warning for the driver to pay attention, see Claim 1 1[A]- competence level of | [E], and the driver remains distracted and/or without hands on the steering wheel, the Tesla determines a the human driver is_| second fault has occurred. See Claim 1[B] (discussing instructing the driver to take over immediately), supra. below a third threshold after taking the first corrective action, 83  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA determining that a second fault has occurred; [2] determining the corrective action further comprises determining a second corrective action corresponding to the second fault; and See Claim 1[D], supra. [3] the second corrective action comprises transferring driver controls to manual control and alerting the human driver to take over. See Claim 1 [I], supra. Claim 13 13, [PRE] The computer program product of claim 7, further comprising: See Claim 7[PRE]-[E], supra. [A] determining the competence level of the processor comprises using a weighted voting system, the weighted voting system comprising: See Claim 3[A], supra. 84  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA [1] a plurality of inputs anda plurality of weights, wherein the plurality of inputs comprise sensor readings from a camera sensor; and See Claim 3[A][1], supra. [B] multiplying at least one of the plurality of inputs by a weight from among the plurality of weights; See Claim 3[A][1], supra (“The neural network then applies a first plurality of weights to said inputs.” (citing Claim 3[A], supra)). [1] wherein the plurality of weights are based on active learning data; and See Claim 3[B][1], supra. [2] wherein the active learning data includes condition data from a cohort of other SDVs, wherein the condition data shares one or more traits with a current condition of a roadway on which the SDV is currently traveling. See Claim 3[B][2] (discussing where the road condition is specifically a weather condition), supra. As previously stated, see Claim 1[D], supra, the Tesla neural network has been previously trained on this active learning data so that it may be able to determine what to do when it encounters a current condition with one or more shared traits. This could be any one permutation of many different traits present in Tesla’s neural network’s training data. Claim 14 14, [PRE] The computer program See Claim 7[PRE]-[E], supra. 85  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA product of claim 7, further comprising: [A] determining the competence level of the human driver comprises using a weighted voting system with a first plurality of inputs and a first plurality of weights, wherein: See Claim 10[A], supra. [1] the first plurality of inputs comprises sensor readings from a first camera sensor; and See Claim 3[A][1], supra. [2] at least one of the first plurality of inputs is multiplied by a weight from among the first plurality of weights, wherein the first plurality of weights are based on first active learning data, the first active learning data comprising information about a cohort of human drivers of other SDVs; As previously stated, the Full Self Driving Functions collect pluralities of inputs from the suite of sensors on the car and has pluralities of weights against which they are multiplied. See Claim 3[A][1]-[B], supra. At least some such weights are based on active learning data collected from other Teslas and included in the training data for Tesla’s neural net. See Claim 3[A], supra. Some of active learning data comprises information about a cohort of human drivers of other Teslas and included in the training data for Tesla’s neural net. Information collected may include whether a driver is paying attention or getting distracted, looking at their phone, staying in the driver’s seat or moving around the cabin, etc. https://electrek.co/202 1/04/08/tesla-driver-monitoring-system-detect-driver-attention-real-time/ This data is used to sharpen the neural net’s ability to differentiate between poor or distracted drivers and drivers who are paying attention and able to operate the vehicle. 86  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA [B] determining the | See Claim 3[A], supra. competence level of the processor comprises using the weighted voting system with a second plurality of inputs and a second plurality of weights, wherein: [1] the second See Claim 5[A][1], supra. plurality of inputs comprises sensor readings from a second camera sensor; and [2] at least one of See Claim 13[B][2], supra. the second plurality of inputs is multiplied by a weight from among the second plurality of weights, wherein the second plurality of weights are based on second active learning data, the second active learning data comprising condition data from a cohort of other SDVs, wherein the condition data shares one or more 87  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA traits with a current condition of a roadway on which the SDV is currently traveling; [C] the first camera sensor is different from the second camera sensor, the first plurality of inputs is different from the second plurality of inputs, and the first plurality of weights is different from the second plurality of weights; The Full Self Driving Functions collect inputs from multiple sensors and apply a variety of weights. See Claim 1[A] (discussing the plurality of sensors), Claim 3[A] (discussing the pluralities of weights), supra. [D] when the competence level of the human driver is above a first threshold, the SDV autonomously controls the driver controls without requiring the human driver to operate the driver controls; See Claim 11[A], supra. [E] the SDV autonomously maintains a buffer of space from other vehicles around the SDV and the SDV See Claim 4[B], supra. 88  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA autonomously controls the steering of the SDV while the SDV autonomously controls the driver controls, without requiring the human driver to operate the driver controls; [F] when the See Claim 11[C], supra. competence level of the human driver is below a second threshold, determining that a first fault has occurred; [G] determining the | See Claim 11[D], supra. corrective action comprises determining a first corrective action corresponding to the first fault; and [H] the first See Claim 11[E], supra. corrective action comprises issuing an alert while the SDV autonomously controls the driver controls without requiring the human driver to operate the driver controls. 89  PRIVILEGED & CONFIDENTIAL WORK PRODUCT SUBJECT TO NDA 90