What happens when a firm feels it simply cannot accept the Gartner opinion of its products and performance?

Sometimes, people get angry. The most extreme case, of course, was ZL Technologies attempt, five years ago, to sue Gartner for $1.7bn. But the most recent example involves Nuix.

Angry ManNuix is a reputable, well-established Australian company in the e-discovery software field, with a good-looking international customer base. But when Nuix managers saw the latest e-discovery Magic Quadrant, they couldn’t believe their eyes. The MQ was so dramatically at odds with their own view of the market that Nuix put out a detailed document to try to counter the Gartner assessment. It’s an interesting read and I would encourage you to look at it. When you’ve done that, though, you might want to sit back for a minute and form your own view as to the relative risks and rewards of taking such a stance.

A plea for justice…

The crux of the matter is that the company feels the Gartner view of the sector is completely out of touch with today’s market realities. Nuix believes the vendors that are praised in the assessment are actually weaker in the market than Nuix and that the methodology used is so opaque that it stops MQ readers from being able to assess the Gartner approach and make up their own minds about its validity. And lastly, of course, Nuix complains that the analysts and the Gartner in-house ombudsman have been highhanded and intransigent in their response to its anguished pleas for change.

Sound familiar? You’re dead right. We must talk to at least one company every week of the year that could come up with a similarly heartfelt complaint about the Gartner analysts, methodology and conclusions.

Or is Nuix cutting off its nose to spite its face?

You may remember my blog earlier this year entitled “If I drink poison, will the Gartner analyst die?” I observed then that one common reaction is for companies to turn their backs on Gartner when this kind of thing occurs. But, as that blog hinted, this approach never hurts the analyst – only the supplier who has had the fit.

So is it any more fruitful to come out fighting and mount an all-out attack on the analyst’s opinion and methodology? Certainly it opens the issue up for debate within the industry. But unfortunately it will always smack of sour grapes, however righteous your indignation may be. Your competitors may well agree with you, but they will be awfully glad it’s you and not them who is out there fighting in vain for truth, justice and a top-right position in the Magic Quadrant.

The specific issues

Several of the issues raised by Nuix are definitely worth closer consideration.

    1. “It’s impossible for customers to evaluate if the research was sound and applicable to their situation,” say Nuix. ”Magic Quadrant…is designed to give the appearance of mathematical precision, but it is much more pseudo than science.” You can’t argue, really. These are statements of fact. the Gartner recent switch to the Interactive MQ format has made the assessment structure slightly more transparent, but the detailed criteria and how the actual scores are arrived at are still not made public. But then this is not academic research. It’s commercial research and, as such, not necessarily bound by the same principles. In reality, if analyst firms were to follow such principles rigidly their output would probably drop by a factor of ten. Sadly, it just is what it is. No amount of complaint over the years has really changed that – and nor is it ever likely to.
    2. “We escalated our concerns through the Gartner ombudsman…The ombudsman concluded that Gartner and Nuix had different views of the market.” This last point is really the key issue, as many companies have completely the wrong idea about what the ombudsman is there to do. The ombudsman’s role is not to judge the fairness of a research position. It is solely to assess whether the analyst has followed the defined process. A view that the analysts don’t “know their arse from their elbow” is not an issue of process – it’s purely opinion, I’m afraid!
    3. “We can’t convince the analysts to change their opinion.” Nuix is far from being the only company to feel that way. But can you afford to stop trying? More than 50% of all purchases are influenced by analysts. Whatever your opinion of their viewpoint, can you afford to just publish a piece saying “They are wrong” and trust that the market will take note of your complaint and ignore the analysts? We both know the answer to that. Today, because of the fuss it has kicked up, Nuix may have some transient visibility. The fact is, however, as they say here in England, “Tomorrow their complaint will be chip paper.” In other words, the Nuix document will no longer be read, and will be good only as a wrapper for Britain’s finest culinary delicacy, our traditional fish and chips. So Nuix, and others that find themselves in the same position, must view a negative MQ assessment as a setback in the struggle, not a reason to lay down their weapons. And there’s only type of weapon that will work here – evidence. If you want to change an analyst’s opinion, you have to prove that it’s wrong. Then you have to prove that it’s wrong over and over again, with incontrovertible proof to back you up. But what if such proof doesn’t exist? If that’s the case, there are only two options – either commission a study to provide the evidence you need or accept the horrible truth that the analyst might actually be right!

Don’t make it personal

Principles won’t get you very far in a situation like this. You’ll have to be pragmatic. So let me leave you with three helpful thoughts:

  1. Be absolutely sure you keep the issues that arise over the Gartner process’s lack of transparency quite separate from your overall relationship with the individual analyst. Remember St Augustine: “Hate the sin, but love the sinner.” After all, it’s the analyst who will always have the last word.
  2. Take a good look at the document prepared by Nuix. It is actually a great example of the type of point by point rebuttal that you are likely to need in a sales situation, after you’ve had a critical assessment in an MQ. If you are in this position, you may want to make sure your sales people are armed with something similar.
  3. Be realistic about the way analysts think. Jim Barksdale, when he was CEO of Netscape, summed it up. “If we have data,” he said, “let’s go with the data. If all we have is opinions, let’s go with mine.” There aren’t many analysts who could put it so pithily, or express it so frankly, but that’s what they’re all thinking. So it makes sense to act accordingly.

No research generates more angst than the Magic Quadrant. People may act tough, but every firm cares how it’s rated by Gartner and how it is covered. As ever, our advice is to put an engagement plan in place and win over the analysts, over a period of months, with a well-thought-out stream of evidence to support your view.

No analyst is going to listen to your arguments one fine bright morning, smite his forehead and declare “Well, I’ll be blowed. You’re absolutely right. What a fool I’ve been. Just let me get on to my boss and tell her I’ve got to rewrite the MQ.” Overnight conversions just don’t happen. Apart from anything else, analysts, like human beings, don’t like admitting they’re wrong. But if you offer enough clear, convincing, consistent evidence, over a period of time, you’re in with a good chance that the analyst will end up shifting position and reassessing the situation when the next MQ comes round.