The report issued on Thursday by the National Transportation Safety Board (NTSB) said Walter Huang, the driver of the 2017 Model X using Autopilot, had been given two visual alerts and one auditory alert to place his hands on the steering wheel during the trip - but those alerts came more than 15 minutes before the March 23 crash.
He died in hospital soon after the crash.
David Friedman, director of Cars and Product Policy and Analysis for Consumers Union, the advocacy arm of Consumer Reports, said the NTSB's "alarming report reinforces why Tesla must respond immediately to previous concerns raised about its driver-assist system."
Friedman said the crash "demonstrates that Tesla's system can't dependably navigate common road situations on its own, and fails to keep the driver engaged exactly when it is needed most."
The NTSB report said the vehicle had sped up from 62 miles per hour (mph) to nearly 71 mph in the three seconds before the crash.
In the minute before the crash, Huang's hands were on the wheel for a total of 34 seconds, but not in the last six seconds before he struck a crash attenuator and concrete barrier on U.S. Highway 101 in Mountain View, California, the report said.
Tesla says drivers are supposed to keep their hands on the wheel at all times when using the system, but many drivers at times opt not to do so.
Tesla declined to comment on the NTSB report and did not immediately comment Friday, but said in March that Huang had not braked or taken actions to avoid the crash in the final seconds before the accident.
A lawyer for Huang's family, Mark Fong, said in a statement the NTSB report supports "our concerns that there was a failure of both the Tesla Autopilot and the automatic braking systems of the car," he said. "The Autopilot system should never have caused this to happen."
Last month, police in Utah said a Tesla in Autopilot crashed into a stopped fire truck. In that incident, the driver of the Model S did not touch the steering wheel in the 80 seconds before the crash.
Tesla's owner's manual warns drivers that the system may not detect stationary objects when traveling at higher speeds.
The safety board is investigating four Tesla crashes since August 2017 and looking at post-crash fire issues and the use of Autopilot.
The NTSB previously faulted Tesla in a 2016 fatal crash in Florida in which Autopilot was engaged, saying: "Tesla allowed the driver to use the system outside of the environment for which it was designed and the system gave far too much leeway to the driver to divert his attention."
Reporting by David Shepardson.