Etuma Blog

Lists by Topic

see all

Subscribe to Email Updates

12-Point Checklist For Choosing a Customer Feedback Text Analysis Vendor

[fa icon="calendar"] Mar 17, 2016 11:57:43 AM / by Matti Airas

Choosing the right vendor to analyze customer verbatims is a difficult task. The websites of most vendors don’t provide any kind of information on how they accomplish the feat, and finding out what concrete benefits you’ll get out of their services can seem obscure.


In his excellent blog post “12 Criteria for Choosing a Text/Social Analytics Provider, Seth Grimes tries to make the vendor selection process easier by creating a common-sense list of requirements and check points. Because Seth’s list covers the requirements for all kinds of text analysis, I will try to paraphrase it from the point-of-view of verbatim analytics.

His advice–to keep a clear head and set realistic expectations–is great:

“Some preliminary advice: Work back from your business goals. Determine what sorts of indicators, insights, and guidance you’ll need. No business is going to need 98.7% sentiment analysis accuracy in 48 languages across a dozen different business domains. Be reasonable; stay away from over-detailed requirements checklists that rate options based on capabilities you’ll never use. Create search criteria that separate the essentials from the nice-to-haves and leave off the don’t-needs. Then design an evaluation that suits your situation – include proof-of-concept prototyping, if possible – to confirm whether each short-list option can transform data relevant to your business into the outputs you need, with the performance characteristics and at a cost you expect.”

And of course I shouldn't write this blog for obvious reasons (we are a verbatim analytics company!). But Seth's blog was structured so well and the argumentation was quite solid that I couldn't help myself.


Twelve criteria for choosing a verbatim analytics provider

1. Type of text (this is the only point I added to Seth’s list) + Industry & business function adaptation

Your verbatim analysis vendor should have a dedicated solution specially tuned for analyzing customer feedback. The way customers “speak” in their feedback responses is often very different from normal text.

Words can have a different meanings in different contexts. If you are running a hotel chain, it might be valuable to know what people are saying about individual topics like LIGHTS, CORRIDORS, etc., but in a grocery store you might want to pool all those into a topic called STORE LAYOUT. The word SERVICE can be problematic for an airline, for example, because you cannot be sure whether it applies to the service on the airplane or at the airport. Contextual understanding is crucial, which is why keyword extraction doesn’t work. The analysis provider must be able to identify and understand the context of a specific word.


2. Customizability

Each company has its own products, services, and ways of operating. Being able to customize and continuously tune the analysis for each company is a mandatory requirement.

The main consideration is who does the customization: you (or somebody in your staff), a third-party consultant, or the analytics vendor. Most tools and services require you to do the customization work yourself. Make sure that you add the cost of this work into the total cost of ownership.


3. Data source suitability

It is clear that each feedback channel requires a tuning to accommodate the channel specific peculiarities and language. Tweets requires a different kind of analysis from chats. For example, being able to analyze emoticons is important in social media analysis but not required for spontaneous feedback via web form or email.. Ensure that your vendor has the ability to take into a account the specific needs of all your feedback channels.


4. Languages supported

One of the main difference between vendors is whether they can analyze multiple languages. Translating comments loses nuances and often, in the case of smaller languages, gets the entire theme and sentiment wrong. It is important that the analysis is done in the source language.

Presenting the results can be done in a single language if all the topics and touchpoints are “mapped” across languages. This means that the quantifiable information would be in one language, which creates comparable information across languages.


5. Analysis functions provided

Text doesn’t have an absolute benchmark, except what was previously said about the same topic. This is why consistent industry- and company-categorization (point 2) is essential in feedback verbatim analytics. And if each category (topic) can be tagged with sentiment, then we are talking about two dimensional information.

Name or brand detection is crucial, and it would be great if brand mentions could be tagged with sentiment.

Having access to the keywords and phrases that are tagged to a topic can also be valuable but brings up the challenge of handling a large amount of less-structured data (a feedback channel can have millions of keywords and phrases).

The key is to have information that can be used for decision making. What this means in practise is techniques such as clustering (e.g. which topics often occur together), correlation (how a topic correlates with a quantifiable background variable - e.g. NPS score), and trend analysis (how topic volume or average sentiment changes over time).


6. Interfaces, outputs and usability

Verbatim analysis results are not often THE data set. It is information which combined and correlated with other data (e.g. purchase or web behavior) becomes valuable. That is why verbatim analysis visualization in vacuum is not that interesting (even if lot of people still seem to like word clouds). Verbatim analysis should enrich the data analysis transparently (via and API) and visualize the results in the same medium as structured information is visualized.

The verbatim analysis results should be also be usable for data mining (e.g. SPSS and SAS data analysis tools).

And why not use excel to distribute the dashboards. Just make sure it is tuned to the use case and person’s role in the organization.


7. Accuracy: precision, recall, relevance and results granularity

Most KPI’s and financial performance figures are hard facts. When it comes to verbatim analysis results, the results are more obscure (see Seth’s comment before the list). Maybe the more important factor is consistency and that the solution is using machine learning and expert work to improve the analysis all the time.

Results granularity comes from the vendor’s ability to tune the analysis results (topics or categories) according to company’s needs. What this customization means in practise is that each topic can be closed, combined to another topic, broken down to smaller topics or renamed. This is the minimum requirement for meeting the relevance criteria.


8. Performance: speed, throughput and reliability

Verbatim analysis service should run close to real-time. If the vendor solution runs in one of the global cloud platforms (IBM, Microsoft, Amazon) then performance is often not an issue. If the vendor runs their own servers then you need to do more due diligence. Running the analysis in your own servers is an option but in this case you need to make sure that you have the resources to maintain the server environment and teach and tune the analysis framework continuously.


9. Provider track record, market position and financial position

The sad truth is that nobody ever gets fired for buying from IBM. But not considering smaller and often more innovative (and much cheaper) service providers would be a mistake. Cloud platforms have made even the smaller vendors’ services of high quality, secure, reliable and scalable. Therefore the quality cannot be deduced from the size of the company.

These cloud-based services (in which the vendor provides the analysis as a turn-key service) are often much cheaper to own than traditional enterprise software solutions. What kind of set-up makes sense for you depends on your financial and human resources (=how much text analysis platform maintenance you want to do). But make sure that whomever you choose they have relevant references.

The best way to ensure the quality and relevance of the analysis results is to get the vendor to run demo is with your own dataset (=familiar context). That will demonstrate whether the vendor can meet your requirements. It is good to keep in mind with demos, especially when free,  that the quality is not as good as with the commercial implementation. But demo is good way to figure out whether their analysis results meets your relevance and granularity requirements. Those are often depended upon the service architecture and cannot be changed (except if you run a text analysis tool like SPSS yourself).


10. Provider’s alliances and tool and data integrations

If the verbatim analysis service provides a well productized API and you have the infrasctructre (enrichment, analysis and visualization tools) then it doesn’t really matter what kind of alliances they have.

But if you want to integrate your feedback channels directly to the verbatim analysis service and then visualize the results or use an analysis tool to do data mining then it is nice if the vendor has existing connectors to the survey vendor’s service.


11. Cost: Price, licensing terms, and total cost of ownership (TCO)

Seth has a good point here. There is large TCO difference between a pure cloud based service and a platform that you need to maintain and tune. It might be hard to figure out whether the vendor provides a platform or a service. A demo of proof-of-concept will demonstrate this better than thousand words.


12. Proof of concept

The fact that the vendor cannot provide a free demo with your own data tells a lot about the service. If they don’t provide a free demo, it means that their solution architecture requires so much tuning and customization that developing a demo is financially not feasible. What is important to keep in mind is that this type of solutions are also more expensive to buy and the maintenance costs can be substantial. But the analysis results in this type of solutions can be very good.

If the demo is giving you a text analysis platform/tool (e.g. SPSS) to try out for 30 days, then you really need to make sure that you have time to develop the analysis model to assess whether that solution works for you. These tools are excellent but only if you have the knowledge and experience and have lot of time to allocate for this work effort.

If you are not familiar with text analysis tools and don’t have a large budget then analysis-as-a-service (AaaS) is your best option. The nice thing about the cloud services is also that because they are easy to implement. It is also important to keep in mind that because the investment is relatively small, you are not tying yourself to a single vendor forever. Now that the verbatim analysis business has gone mainstream the solutions and services are developing fast and you might want to change vendors even in the near future.


Seth’s advice about working back from business goals is very valid. Decide the business target (what information, granularity, multi-language, real-time) and take the restricting elements (budget, linguistics knowledge, resources) into account, and you will end up making the right decision.


Read this white paper and become a text analytics expert?



Topics: Customer Experience Management, text analysis, NPS, verbatim analytics

Matti Airas

Written by Matti Airas

My passion is to figure out how to turn open-text feedback into well structured usable information.