TrueDelta questions quality of JD Power methodology
TrueDelta provides some interesting insight into automotive information and rating
juggernaut JD Power and Associates. The site, which is developing its own vehicle reliability system, points out how JD’s objectives may influence its information provided to clients (auto manufacturers) and the public. TrueDelta believes there are weaknesses in JD’s Initial Quality Study (IQS) when applied to vehicle reliability.
TrueDelta also questions JD’s definitions of problems (e.g., the infamous cup holders and German cars debate, Hummer's miles per gallon); rating system; the use of average; and "What is Best" emphasis. The latter, especially, is eye-opening on buyer behavior.
Get a WordPress.com Blog
Reader Comments (Page 1 of 1)
Terrance Cavendish 7:22PM (2/06/2006)
"quite a few people," "a number of people," "many people" . . . again, if you are a market researcher, where are the hard facts?
Regarding the analysis from J.D. Power that proves a connection between IQS and longer-term reliability, I'm not sure it exists either, but there might not be a direct relationship anyway. You could have a very high quality vehicle (i.e. strong IQS) but low dependability. (i.e. strong VDS). That is the case for many vehicles. I'd be interested in seeing that relationship too, but I'm not as fixated on that and frankly, I'm not sure that the relationship is as big as an issue as you are raising.
If someone from JDPA is viewing this, please chime in.
Reply
Richard Warren 2:44PM (2/03/2006)
I've been saying this since JD Powers came on the scene, that the data many times is flawed.
And yes, it's always been about the money with ole JD.
JD likes to present himself as the definitive quality person that has this huge effect for the consumers, fact is a lot of the changes in the auto business were caused by someting more simple. Competition
This article points out that JD is actually under the power of guess who? The manufacturers, why he even changed the time of the survey based on the clients request.
And you get get any answer you want based on how you setup your survey.
I've spoken before regarding Honda in the late 70's to mid 80's where JD knew and complained of bogus RDR cards that listed dealers, employees, friends as regeistered owners that then filled in the Powers surveys, even knowing that they published the results. Oh how we loved our Hondas as the A pipes failed the head gaskets blew, the syncronizers failed, the mirrors fell off, the headliners fell down, the knuckles were sloppy, exhaust systems failed, yet we rated those cars as just wonderful.
In the end Honda employees went to prison yet JD carries on. And some------Believe him, just another circus barker "step right up folks let me sell you something"
Reply
7of7 2:46PM (2/03/2006)
Who in the hell is "TrueDelta" to question to veracity or quality of anything.
I question the quality of TrueDelta after it said "For all of its faults, the heart of Consumers Union is generally in the right place."
Reply
Lithous 2:51PM (2/03/2006)
"Additional models will be added once there are at least 25 (per model year) in the sample. To help make this happen, please encourage friends and family to join." (source:http://www.truedelta.com/tish.php)
I'm sure they have a huge sample set. Take their word about reliability from their massive sample set they have on hand that is brilliant. Nice marketing on that home business guy though.
Reply
Noah 3:33PM (2/03/2006)
I myself question wether or not JD Power's methods show if a car will be a good car or not, how many small glitches sink good brands while a brand that makes just a couple cars that explode every once and a while killing people flys into a top rank?
Reply
Bob 4:00PM (2/03/2006)
I have never thought JD was perfect, but still seems to have the most scientific basis for their ratings. Namely, the sample. JD surveys the public, while Consumer Reports, only their own readers. The Consumer's approach is like Time magazine asking its readers who they will vote for President and then claiming its a poll. Considering that a company like GM has a 60+ plus percent customer retention rate, I am guessing those people aren't reading or subscribing to consumers reports and therefore participating in its survey. They are satisfied with their choices and don't need third party validation.
If you look at companies like strategic vision, they rate experiences, not reliability so thats not always a good indictaor.
I like the way True is going to ask the question: "times in the shop" which definitely differentiates itself from those survey respondents who will include gas mileage or size of cup holders as a defect. That said, it seems that True's samples may be unrepresentative too.
True can question the motives of JD, but I can do the same for True and Consumer's. All three are there to sell a product.
I really believe that Consumer's Reports is doing poor work. Not only is their sample off, but they often they have "predicted reliability" which seems to have no basis in reality. Their grading system of circles and shapes which amounts to an A through E grading system is so simplistic. This system does not take into account the little differences between cars today. Today the "best" and "worst" cars aren't as far apart as they were just five years ago, but still the "worst" may get the same E grade. Add to that the fact that an E graded car is a more reliable car than the A graded car of only 5 years ago. Where five years ago the difference between the A car and the E car were several more problems per car, and now it may be only a portion of a problem per car, yet we still keep seeing the same A through E grades.
Consumers Reports is doing customers a disservice if they give a car a poor rating when really a person with that car is only 10 % more likely to have a problem with that car than the best cars. If the lower rated car is vastly cheaper, then the consumer would be better off buying the cheaper car. Even if it had a problem, the money saved would more than pay for the minor problem.
Really cars are so close together that we are starting to see the best and average cars with the statistical margin or error, so as time does on, ratings will be less reliable.
When it comes to consumers reports though, I guess exaggerating the differences between cars sells magazines.
Reply
Ani 4:14PM (2/03/2006)
So, if J D Powers main aim is to maximise profits and truedelta's main aim is to help consumers, can I keep truedelta's profits.
Reply
Lithous 4:21PM (2/03/2006)
One of the very first comment posts I made on this site I stated how showing data in a format such that 120 problems per 100 (as an example) is really not a good thing for the domestics. As we have seen the domestics are the ones that have to prove they are not crap now (guilty until proven innocent). What sounds more like something I want to present as a GM marketer we have 190 problems for every 100 cars produced or we have 1.9 defects on average with our vehicles (that is how it breaks down)? Especially if they could then state that their competitor has 1.1 (virtually the same number of problems per car, though rounding would make it closer to 1 more defect per car and keep in mind that these are not car "break down" type problems being reported). So I don't buy that JD Powers is totally a domestic car companies dream. I have mentioned JD Powers as a "don't use CR as your bible, there is other data" type statements but there is a much better solution than CR, JD Powers and especially truedelta IMO...
My personal choice would be going to forums created for the specific car brands and/or models you like and read things there. You could go to a brand/model independent site to get what vehicles are in the same class first and then go hunting for the forums.
This way you can ask questions to the people who are in essence filling out the "survey" so to speak (those who give their opinions and stories on the forum). Get things clarified by asking. Ask for things none of the surveys cover, etc.
Even epinions is too much of a one post type of place, IMO. I mean, people will usually give an opinion there an not go back to post something about the same product for ever or months or whatever. People tend to hang around awhile at the sites specific to what they own.
And you will get the good and the bad. People have no problem when their vehicle has just had a problem to come in pissed (and most of the time rightfully so).
I think the survey companies can be helpful for some things maybe but it is like having a personalized survey company at the tip of your hands if you go to the forums I mentioned.
Reply
Eric L. 4:47PM (2/03/2006)
I am always amazed about the things people say about Consumers Reports. Please tell me, what other organization publishes results not based on advertisements or endorsements? CR's agenda is to review vehicles and judge them in terms of comfort, practicality, and reliability. These reviews are for the most part helpful to the average consumer for making informed decisions. The average consumer doesn't know (and doesn't care) what drop-throttle oversteer is. If I wanted to read about the performance aspect of vehicles, I would read the other auto mags.
Reply
Michael Karesh 5:25PM (2/03/2006)
I learned this article had been posted after noticing a sudden surge of people joining the panel, and noticing where visitors were coming from. Thank you for the mention, Joel. And thank you to everyone who has signed up.
I should probably clarify a few things. In my critiques of CR and JD Power I never state or imply that they are biased towards any particular manufacturer. I also don't believe that they intentionally twist their data, or have an "agenda" as the term is usually understood. Its not necessary for an organization to have bad intentions for it to have unfortunate effects. I don't even argue that the data of either organization is flawed.
What I do assert is that the way CR and JD Power present their results leads to inaccurate and distorted public perceptions, whether they intend this or not. Finally, in the case of JD Power I do not believe they are under the power of the manufacturers. If anything, manufacturers complain about the converse. They do adjust their studies to better serve the needs of manufacturers, but this is what organizations do--serve their primary clients.
The key difference between J.D. Power and TrueDelta is that the first sees manufacturers as its primary clients, while I see TrueDelta's members as my primary clients.
I do hope that the site will be financially viable. But there are certainly other ways that I could earn much more with my time. If you don't believe me, ask my wife...
My sample is currently small, about 3300 vehicles. But I'm just getting started. People who refuse to join the panel because the panel is small--clearly if everyone thought that way nothing would ever get done.
The panel will be far larger before the year is out.
Reply
Terrance Cavendish 5:29PM (2/03/2006)
Sounds like this Micheal guy from TrueDelta has some axes to grind; hence the article.
"I approached J.D. Power with my proposed research a couple of years ago." Oh, so JDPA was good enough to potentially work for/with, but if they say no, then they must have evil motivations. I assume that you would have charged MONEY for your time/service, just like any normal human being would.
"Their focus is on manufacturers." Yes it is, but ultimately that directly benefits the consumers who purchase products. Summary J.D. Power ratings and rankings are available on several top consumer websites free of charge.
"IQS might be a good predictor of long-term reliability. . . " If this guy took 2 minutes over at JDPA's ite he would have discovered that 'IQS measures a broad range of quality problems, heavily weighted toward defects and malfunctions, quality of workmanship, drivability, human factors in engineering (i.e. ease of use) and safety-related problems.' IQS ahs nothing to do with long-term reliability, per se.
" . . . and the VDS might be a good predictor of even longer-term reliability, but to my knowledge J.D. Power has not presented evidence to support this common inference." The JDPA 2004 VDS press release clearly states, "According to actual retail transaction data from the Power Information Network, a division of J.D. Power and Associates, 3-year-old vehicles of brands that perform above the industry average in VDS typically retain $1,000 more of their value than those of brands performing below the industry average."
Sorry Mike. Looks like that PhD was a complete waste of money. It might be tough to find a research job with your attitude
Reply
Lithous 5:59PM (2/03/2006)
"clearly if everyone thought that way nothing would ever get done."
Yes, but forcing people to join the panel and keep up with posting updates or pay money or you don't get to see anything isn't the perfect solution either.
I'd like to know, free of charge, how many models of each year you actually have data on before I commit to anything. A lot of data could be free (like what cars you have data on for what years) without giving away the farm.
I should get a consultant fee for that one. You break down on a page what you do have and (if it is worth a grain of salt, the amount of data you do have now) people will be more apt to at least do the pay option over the panel option, IMO. Actually it may motivate people to want to be some of the first panelist as well, if they knew you only had 3 samples of their car.
Maybe I missed it, I didn't see anything on the site stating that any custom software was available to help analize the data you do have. Or is it just you querying an Access or MySQL database and emailing custom results?
I don't know that putting a few pages up and gathering data from people is 2006 high tech. Again, I could get that and more from forums.
Reply
Michael Karesh 6:08PM (2/03/2006)
Terrance--I'm not sure if you're willfully misreading what I wrote, or just aren't a good reader.
1. I never state or mean to imply that J.D. Power is evil. I even say this in the comment right before your own. I simply feel that their research could be more helpful. They were not interested in improving their research in the way I suggested, so I decided to just implement my ideas myself. How does this make me suspect?
My article addresses the problems with J.D. Power's publicly available ratings and rankings. They save the usefully precise data for the manufacturers.
When I state that IQS and VDS might be good predictors of X and Y, I am merely giving them the benefit of the doubt. I don't have any idea one way or the other.
The problem is, this is how most people interpret these scores, as predictors of long-term durability. It's not as if people only care what happens with cars at the 90-day and 3-year marks, but at no other time. People want to know how reliable a car will be the whole time they plan to own it. So they must make inferences from the J.D. Power ratings, and it is not at all clear that these inferences are valid. People aren't usually trying to find out how much value their car will retain from the VDS.
I have visited J.D. Power's site, and I know what they measure. But this is not what people commonly do. You might argue that J.D. Power is not responsible for how people misinterpret its results. I personally do feel responsible for how people interpret my results, so I intend to present them as clearly as possible. You cannot assume that the average person will read the fine print, because few people do.
Finally, my attitude. What is wrong with my attitude? My tone is civil throughout, and my criticisms thoroughly constructive.
Reply
Michael Karesh 6:09PM (2/03/2006)
Terrance--I'm not sure if you're willfully misreading what I wrote, or just aren't a good reader.
1. I never state or mean to imply that J.D. Power is evil. I even say this in the comment right before your own. I simply feel that their research could be more helpful. They were not interested in improving their research in the way I suggested, so I decided to just implement my ideas myself. How does this make me suspect?
My article addresses the problems with J.D. Power's publicly available ratings and rankings. They save the usefully precise data for the manufacturers.
When I state that IQS and VDS might be good predictors of X and Y, I am merely giving them the benefit of the doubt. I don't have any idea one way or the other.
The problem is, this is how most people interpret these scores, as predictors of long-term durability. It's not as if people only care what happens with cars at the 90-day and 3-year marks, but at no other time. People want to know how reliable a car will be the whole time they plan to own it. So they must make inferences from the J.D. Power ratings, and it is not at all clear that these inferences are valid. People aren't usually trying to find out how much value their car will retain from the VDS.
I have visited J.D. Power's site, and I know what they measure. But this is not what people commonly do. You might argue that J.D. Power is not responsible for how people misinterpret its results. I personally do feel responsible for how people interpret my results, so I intend to present them as clearly as possible. You cannot assume that the average person will read the fine print, because few people do.
Finally, my attitude. What is wrong with my attitude? My tone is civil throughout, and my criticisms thoroughly constructive.
Reply
let me just say this about that... 7:08PM (2/03/2006)
I absolutely feel JD Power is swayed by their customers. I mean, T-Mobile having top ratings in the US as a cellular operator just doesn't work for me as I watched a friend's phone calls get dropped 20 times (in about 2 hours of talking) due to no reception in an area that T-Mo owns the original GSM towers (South Tahoe area, original GSM buildout was 10 years ago by PacBell, sold to T-Mo, my Cingular phone uses AT&T's more recent buildout).
Now, having said this, when a company says JD Power's Initial Quality Survey is no good for predicting reliability, I have to wonder about this other company. Did they not see the word "initial" in there? Did they not know JD Power has a survey with the word "reliability" in the name that is better for predicting reliability?
As to Consumer Reports, they mean well I guess, but for cars, they have done very poorly by me. My car was rated a recommended pick (2000 Audi A6, also was one of C&D's "10 Best") and has been an absolute mechanical nightmare. CR removed Audi from their recommended list the next year largely because of problems with my model car.
How did this problem happen? Because CR gets their results from people who probably don't know much about reliability. Audi wasn't particularly reliable when CR was recommending them, they just were rated well by their customers, probably because they didn't know what real reliability was.
Additionally, CR gives new cars predicted reliability based upon some kind of feeling they pull out of their ass from cars from the same manufacturer at the time. So VW looks great and is recommended in 2000 because they only recently started to make crap, while GM can't get a recommendation even though some of their divisions shellacked companies like VW who were "highly rated" at the time.
In other words, if you bought a car in 2000 based upon CR you would have bought a VW/Audi and had problems with fuel sensors, coil packs and windows dropping into doors, while you would have avoided cars from Buick, Chevy or Cadillac that would likely have done you very well mechanically over those 5-6 years.
I guess I'm saying CR's results are too slanted by the past and people's own egos (damn, I feel smart for buying a VW, I'm going to express how proud I am of myself by marking the car I bought as great in my responses) to be really accurate on reliability.
Reply
Michael Karesh 9:20AM (2/04/2006)
I tried to post responses yesterday, but the confirmation email process wasn't working--they finally rolled in at 8:30 this morning. Yet some duplicate comments got posted last night. Four were there, two still are.
113 new vehicles were added to the panel yesterday. Thank you to those who signed up.
My responses:
Lithous,
I'm not forcing anyone to do anything. I'm doing my best to create a system where the people who put in the time and effort to provide reliability information are the ones who benefit from it. I don't care if no one ever pays the $24.95 fee (though a few have already offered)--I've simply learned that offering to give away everything for free leads to very few people signing up.
In a couple of places on the site it states that data collection only began in November for a few models. Once I have data to provide--never do I say I have data to provide right now, everything is in the future tense--I will do the very things you suggest.
A page showing which models I'm collecting data on already will be up in a few days. I've been emailing this information to members for a couple of months.
I'll just be using standard statistical software to analyze the data.
I don't know if my research is high tech. I only know that no one else is doing anything like it, and it's going to be the best available.
The problem with forums is there's no control over who posts. With my research, I only collect information on problems that occur after people sign up. Collecting repair histories makes problem rates appear higher than they are, as people with many problems are more likely to respond. I like the forums as well, but you cannot judge problem rates from them.
Let me just say that (comment #14)--
I don't fault the IQS for not predicting reliability. I fault its heavy use in ads in a way that will lead people to think it predict reliability. When the average person hears that a car won a "quality" award, they assume this means it will be a reliable car. They don't go to the J.D. Power site and read the details about what the study actually measures.
The VDS, which measures problems at the three-year mark, gets far less attention and play in ads. Now that I think of it, while I believe I've seen model-level IQS ratings in ads, I don't think I've seen model-level VDS ratings in ads.
And even the three-year mark is just one data point. What if someone wants to know how well a car will do at other ages? Can they safely infer that the VDS predicts, say, reliability during the first five years of a car's life? I honestly have no idea currently, but I aim to find out.
Reply
Terrance Cavendish 4:41PM (2/04/2006)
Michael,
You ask if I am willfully misreading what you wrote, or just am not a good reader.
Neither--I am punching holes in your self-aggrandized think piece.
1. You start by saying that "there are reasons for concern" about J.D. Power. You attack J.D. Power's motivation; ". . . far and away the #1 goal at J.D. Power is to make money." What direct evidence do have have that this is their #1 goal (other than them not wanting to work with you that one time)?
2. You go on to say that "All it cares about is how many millions it can pressure the manufacturers into forking over." This is why I question your motives. Again, what firsthand direct evidence do you have that this is all that J.D. Power cares about?
3. You postulate that, "The problem is, this is how most people interpret these scores, as predictors of long-term durability." Most people? Have you conducted market research yourself or do you have secondary research that would prove that? What are your sources? I don't know about most people, but when I hear the words initial quality (the 'I.Q.' in IQS) I assume that that means the product quality of something shortly after I purchased it. In JDPA's case, initial quality is measured within 90 days of ownership.
4. And then you conclude by asking in your response here, "What is wrong with my attitude? My tone is civil throughout, and my criticisms thoroughly constructive. " Criticism is good and J.D. Power deserves all that is thrown at it. However, there is little concrete, hard data to back up your assertions. At most, they are suppositions, generalities and opinions. If you are going to go up against the bigboys, you better bring some good ammunition.
Lastly, you say that you simply feel that their research could be more helpful. Okay, now you're getting somewhere. Why not start your think piece with that and clearly compare and contrast their approach and your approach. By starting by attacking their motivation, it lowers your credibility.
Reply
Michael Karesh 10:06AM (2/06/2006)
I know quite a few people in automotive market research, many of whom have worked at or with J.D. Power at one time or another. I see them at least one or two times a month. It's a very small community where a number of people have moved from firm to firm and they all talk.
My critique was based on what I've heard from many people over the years, and I must admit I haven't divulged all of the details. Don't expect me to name names or to divulge details that might have this effect.
Perhaps the language in that first section isn't the best, but let's put it this way. If you've completed one of their surveys, how about giving them a call and asking to see the results of your effort?
I visited a large number of forums frequently. I note how people discuss these surveys. They fall into two groups:
1. Our car did great/badly in the JD Power survey!!!
2. It's only for the first 90 days, big deal.
The response you won't see:
3. Our car has great/poor quality in the first 90 days of ownership!!!
What you also won't find: any analysis from J.D. Power that proves a connection between IQS and longer-term reliability.
People either notice the initial and thus don't care about the results, or take these results as indicative of how troublesome a car will be and do care. Which group do you think manufacturers are trying to reach by buying the rights to tout IQS results in their ads?
Reply