Before evaluating a martech solution, be ready with a methodical approach and a set of key questions.
When selecting vendors or services firms, martech buyers ask for references often, but check them rarely. Perhaps you believe you already learned enough from your peers, or perhaps you trust your gut instincts on a platform or vendor. However, when you skip this step, you miss an opportunity to enrich your understanding of a solution and supplier, which can hinder your evaluation, but also potentially slow down the actual implementation.
Based on two decades of helping large enterprises select the right martech platforms, I believe martech customers should overcome any reticence and arm themselves with a methodical approach. A savvy reference checker will employ a variety of different approaches and questions to obtain a richer and more transparent view of potential suppliers. Moreover, reference checking is invaluable when it transcends simple lists of strengths and weaknesses to uncover broader issues of supplier “fit” for you. And it’s all about fit!
So I’m happy to share twenty-five specific questions for which my Real Story Group colleagues and I have experienced good results. But first, like any good campaign, you need to set it up right…
Prior to the conversation
The first issue to address is: Who on your team will conduct the reference check? Some buyers remand this service to a procurement specialist, who will bring some useful experience to the task, but will not know the subject matter and is only likely to ask check-the-box questions. Ideally, a subject matter expert from your selection team will lead the effort, perhaps in conjunction with a sourcing specialist.
Likewise, try to learn a little about the interviewee before you call (e.g., a quick LinkedIn check). Recognize that their role will be restricted to what they know and how they approached their own implementation. Moreover, you’ll want to recognize the cultural component. For example, interviewees may only make only subtle criticisms or praise, which you in turn need to decode carefully.
Most vendors will give you a list of customers you can call. Note however, that they are likely only to share their most satisfied customers, and in fact often bestow extra benefits for those who provide such references. To counter this potential bias:
- Leverage the list of questions below for many possible alternative inquiries that may subtly expose weaknesses and problems;
- Don’t hesitate to supplement the vendor’s list with contacts from your own industry or networks;
- When reaching out to the interviewee, give them some reasonable context about your project, how you received their name, and how much time you’ll need (ask for an hour but explain you may not need that much time);
- Remember to reiterate that the conversation is “confidential,” so their answers will contribute to your overall knowledge of the vendor, but you will never cite any identifying information about the individual or the company beyond your team; and
- If the interviewee is busy, ask if someone else in the firm could speak to their experience
Site visits are ideal, though obviously time-consuming for everyone — and still a fraught topic during our lingering pandemic. The questions here work just as well on the phone; but don’t hesitate to set up screen-sharing, and some cases, an interviewee will informally show you useful nuggets. (To make everyone comfy, don’t record the session; just take good notes).
Questions to ask
You almost certainly won’t have time to ask every question below. This is a comprehensive list, so pick and choose based on time available, and the circumstances of your conversation.
- What’s your role within the company?
Titles vary. Find out what the interviewee really does.
- How long have you been using Platform X?
The longer the experience, the more credible the input will be.
- For what have you been using the product? Give a brief overview of your own scenarios/use cases.
This is important context for the relevance of their input for your use cases.
- What made you select the product in the first place?
This is additional useful context.
- Who did the implementation? How long did it take? How much did it cost?
The quality of an implementation can have a huge impact on success and affect the “reputation” of the technology itself.
- What was the most expensive part of the solution and implementation? Were there any unexpected costs?
Interviewees may be reluctant to share financial details, but they may share general TCO advice.
- What are the top three things your organization likes about the platform?
The key here is to encourage the interviewee to think beyond personal pros and cons and speak on behalf of the whole firm.
- If you were the product manager and could change three things about the product, what would those be?
This is a better way to ask about “weaknesses.” The interviewee may have pause and think here; give them time.
- Are you on the current release? How have upgrades gone?
In some cases, the release version can affect answers. If they’re not on a reasonably current release, this is a red flag.
- What do you know about coming versions, timeframes, etc.?
Match their answers against what the vendor tells you. This may also provide further hints at key limitations.
- How has it been working with the vendor? How would you describe their culture? How has your relationship evolved over time?
The idea here is to distinguish between technology fit and organizational fit.
- When have you had to turn to the vendor for support?
This may signal tech-support intensiveness.
- Where do you go when you have problems and you need really good answers? What user groups do you attend, in person and online?
You want to know about the all-important community around a technology and vendor.
- For a future customer, what’s the best fit for this solution? For what types of organizations would it work best/worst?
This asks the same question about strengths and weaknesses in a different way.
- What was the biggest surprise you encountered (positive or negative)?
The answers could forewarn you against potential difficulties.
- If you had to do it all over again, what would you do differently?
This question helps you draw from the lessons they’ve learned.
- Is there anything else that’s significant about your experience with the vendor or the product that we haven’t discussed?
This ensures that you’ve gleaned as much as you could.
- Is there someone else in your organization with whom we could talk?
Sometimes another individual is a more informed resource. Oftentimes, project managers can point you to architects and developers who may have more intimate knowledge of the inner workings of the vendor and technology.
Varying the script for services firms
Thus far, I’ve assumed you’re seeking references for a technology vendor. To gain more insight into services firms (e.g., integrators, agencies, consultancies, or even the consulting arm of the vendor itself) most of the same questions above still apply, albeit with minor tweaks.
For services firms, however, you’ll want to supplement the list above with inquiries on methodologies, project management, experience, and personnel.
- Did the firm follow the methodology they initially proposed?
Services firm often pitch impressive implementation methodologies, but in practice, they may not follow them.
- How many other similar implementations had the integrator done before your enterprise? How well did they know the technology? When did they have to bring in the vendor itself?
These questions hone in on the all-important dimension of practical experience.
- How tight was their program and project management? Did they finish on time and on budget?
Don’t hesitate to be direct about these basics. When things go wrong, it’s often due to projected time frames and budgets.
- Beyond technical and product-specific experience, what other skill sets did they bring to the project?
One of the value-adds of using third-party services firms is the extra capabilities — often “soft” skills — that they can bring. Different firms have different specializations.
- Describe the issue reporting and escalation process. How responsive are they to minor/major issues?
This is analogous to the support question you’d ask any software vendor.
- Who were the most capable individuals with whom you worked?
Services firms are only as strong as the team working on your account. Try to find out who the stars are.
- Would you use them again? If so, for what, specifically?
This is another way to understand the best/worst “fit” for an integrator.
Conclusion
References can prove to be extremely useful — not just for final diligence — but throughout the supplier evaluation process. In other words, treat this differently than an employee recruitment, where you might check references only after an initial offer has been extended. Inquire early and often.
The vendor may hold back and not supply references until you’ve signaled intent to work with them, in which case you’ll want to push back. Indicate your intent to leverage references not as confirmation that the vendor is reasonably competent, but as part of your selection process.
Selecting marketing technology is hard. Implementing it is even harder. You can get smarter at both by integrating comprehensive reference checks into the early phases of your programs.
Real Story on MarTech is presented through a partnership between MarTech and Real Story Group, a vendor-agnostic research and advisory organization that helps organizations make purchasing decisions on marketing technology applications and digital workplace tools.
The post The Real Story on MarTech: Ask the Right Reference Questions appeared first on MarTech.
MarTechAbout The Author
Jarrod Gingras is Managing Director and Analyst at Real Story Group, a customer-focused technology analyst firm. Jarrod specializes in DAM and Content Technologies, as well as helping large enterprises make good decisions around martech of all kinds. Twitter: @jarrodgingras LinkedIn: https://www.linkedin.com/in/jarrodgingras/
(39)
