Voice app testing is hard

User experience testing can be a minefield

A common stumbling block in building an engaging voice app is user experience testing. The freeform nature of a conversational voice application allows the user to input virtually anything. Proper testing must take this into account. There is also no standardization in user experience testing. Each voice assistant platform has a different set of criteria for deployment on their system.

Our experience testing voice applications can substantially improve your voice application development time and final product quality.

Best practices for voice testing

Use automated testing

Automated and regression testing allows the developer to execute hundreds or thousands of tests without manual intervention. Common issues can be easily isolated. Corner cases can be tested and expanded, tracking software improvements. In general automated testing can quickly and effectively cover 90% of potential issues.

Test with actual users

Real users, with varying accents, dialects and speaking mannerisms can uncover unexpected issues using manual tests with actual users. As new issues are uncovered, they can be transferred to automated testing, continually improving test coverage.

Our testing procedure advantages

Get consistent quality with automated regression testing

Automating regression testing ensures software iterations do not invoke unintended consequences. Any newly created bugs can be spotted immediately.

We use many open source and API resources

Open source software modules and API interfaces keeps test procedures adaptable to vendor platform changes and removes dependencies to any single-source vendor applications.

Our test automation is developed using device emulators

We rely on emulators and simulators as well as actual device testing to ensure efficient testing and complete test coverage.

Best practices for voice testing

an orange circle and a gray small icon

Automated testing

Multiple layers of automated testing (model testing, functional testing, end-to-end testing, …) help to find issues as early as possible.

an orange icon and a microphone

Manual testing

A lot of issues can be found only during manual testing because people speak differently in voice and text channels, and various speech-to-text issues may show more issues.

an orange circle and ABB letters

Try unexpected inputs

Users may not always give expected inputs to your app. At each point in you script, try a random speech response. Your application should recover gracefully.

an orange circle, two arrows, and a clock of cubes

Conversation-specific tests

Test various conversation-specific cases, such as the presence of follow-up questions, confirmations, context-switching and more.

an orange circle and a mobile phone icon

Test on all supported platforms

Each platform will react to input differently. Make sure you run your full test suite on all supported platforms.

an orange circle and two arrows

Understand the limitations of each platform

Some issues you may identify in your testing may be an issue with the platform, rather than your application. It is important to understand where the issue may originate.

an orange icon and a shopping cart

Test environment matter

Devices such as Echo or Home should be tested across the room. Test with high ambient background noise. Real life environments can identify issues not normally found in the lab.

an orange circle and a group of people icon

Crowdtesting

So many people, so many possible utterances. Only testing on many users will show the real quality of your application, and it also allows you to collect more data for NLU models.

an orange circle and a check box

Pre-certification testing

Amazon Alexa and Google Assistant have their own quality requirements which we should follow: security, policy, UX.

an orange circle, two arrows, and a clock of cubes

Create conversational maps

Create numerous typical and atypical voice interaction scripts. Map out expected responses to compare against actual responses.

Technologies we use

bespoken logo

Bespoken is the leader in testing and monitoring for voice apps

amazon polly logo

Amazon Polly is a cloud service that converts text into lifelike speech

an icon in difference colors

Allure is a flexible lightweight multi-language test report tool

Test NG logo

TestNG is a testing framework inspired from JUnit and NUnit.

a red icon

Protractor runs tests against your application running in a real browser

Appium logo

Appium is an open source test automation framework for iOS, Android and Windows apps using the WebDriver protocol.

bespoken logo

Bespoken is the leader in testing and monitoring for voice apps

amazon polly logo

Amazon Polly is a cloud service that converts text into lifelike speech

an icon in difference colors

Allure is a flexible lightweight multi-language test report tool

Test NG logo

TestNG is a testing framework inspired from JUnit and NUnit.

a red icon

Protractor runs tests against your application running in a real browser

Appium logo

Appium is an open source test automation framework for iOS, Android and Windows apps using the WebDriver protocol.

bespoken logo

Bespoken is the leader in testing and monitoring for voice apps

amazon polly logo

Amazon Polly is a cloud service that converts text into lifelike speech

an icon in difference colors

Allure is a flexible lightweight multi-language test report tool

Test NG logo

TestNG is a testing framework inspired from JUnit and NUnit.

a red icon

Protractor runs tests against your application running in a real browser

Appium logo

Appium is an open source test automation framework for iOS, Android and Windows apps using the WebDriver protocol.

Grid Dynamics uses state-of-the-art open-source software modules available from industry leaders. By harnessing the knowledge base of the most industry heavyweights, you get world-class results at a fraction of the costs.

Testing voice apps engagement model

Grid Dynamics makes testing an integral part of every voice application project we do.  Our voice application product specification will include plans for comprehensive test coverage.

This includes script testing before coding, unit and system testing during code development, and monitoring after development.

Get in touch

Let's connect! How can we reach you?

    Invalid phone format
    Submitting
    Tools and best practices in testing voice applications

    Thank you!

    It is very important to be in touch with you.
    We will get back to you soon. Have a great day!

    check

    Something went wrong...

    There are possible difficulties with connection or other issues.
    Please try again after some time.

    Retry