Which Devices Should I Test With?
Every time we go to the mall, the kids pull us to the candy store with the floor-to-ceiling tubes of colored candy. The kids quickly grab their bags and scavenge from each of the bins until their bags fill up. They usually don’t even get to the second aisle of candy.
As testers, we do the same thing with new technology, gadgets, and new devices as kids in a candy store. We say to ourselves, “We need some of each device for testing.”
Smart phones have flooded the market; worldwide, Android alone is manufactured by over 23 companies, producing over 250 different device makes and models of phones. The Unites States alone has over 15 manufacturers building over 100 different makes and models. However, that’s not the end; Androids come in over 10 different operating system versions with multiple screen sizes and densities.
While Android and Apple devices are about 92% of the US market, there are many different makes, models, and operating system versions such as Blackberry, Nokia, Sailfish, Microsoft, and Firefox, just to name a few.
Developing a test device selection plan that is updated on a regular basis will help your organization whittle down the list of devices to a manageable number.
Developing the test device selection plan requires you to:
- Know your customer
- Research and look for trends
- Develop a test priority list
The rest of this post gives details for each of the steps above.
Know Your Customer
When you develop your application, keep the customer in mind. Different demographics have different characteristics and devices, and they use these devices quite differently. For example, my kids grew up using the tablet and now do everything on their smart phone. My mom on the other hand just started using a smart phone and mostly uses it to make calls.
One way to help determine your customers is through web analytics tools that create reports of Web site traffic and traffic sources–federal agencies should check out the Digital Analytics Program if they are looking for web analytics programs. From these reports, we are able to see exactly what make, model, and version of mobile device each person is using to access your application. Keeping this in mind, we have a better idea where to focus our testing when we make changes.
What are the demographics, wants, and needs of the users of your mobile application?
Research and Look for Trends
New phone makes, models, and operating systems are continually being released. How can you spot the trends and keep up with the ever-changing combinations of devices so you can plan and acquire the proper devices to use for testing?
Tools such as StatCounter, a free online tool can help to spot trends in data, such as changes in mobile OS, mobile browser, and screen resolution. StatCounter can show you that user trends from the United States and from other areas of the world are very different. This is yet further proof that you need to know your customers and where they are coming from.
Develop a Test Priority List
Since we don’t have time and resources to test every combination of make, model, and operating system, we need to develop a priority list.
Developing a mobile device priority list is like deciding what candy to purchase. Developing the proper list in the correct order can take a bit of time, but it is the best way to most efficiently test on the “most important” devices.
Each organization should conduct a monthly meeting with analysts, developers, testers, and any other stakeholders to agree on what devices their application will need to function properly on and be tested on.
The list should be developed using inputs from:
- Customer information – Your customers’ demographics, special needs, and considerations
- Mobile trends – Using tools such as StatCounter look at the world and US statistics and trends that reflect your needs
- Trends on your site – Using web analytics tools to see what your customers are using to access your application
Next, you should develop your list. Create a chart with the devices you will test with based on the research. Be sure to include make, model, and operating system.
Finally, now that you have your list, make sure the devices are available to the testers.
With the Internet, anyone can use a Web application anywhere in the world, but organizations do not have the time or resources to test every possible device. By developing a mobile device priority list, your organization can develop a standard list of supported devices.
Don’t be like kids in a candy store and think you need to test every make, model, and version of phone on the market. Develop a device selection plan.
What is your strategy for selecting devices to test? Give us your comments on the discussion tab.
- How to Set the Right Strategy for Selecting Devices for your Enterprise’s mobile testing – Perfecto Mobile (PDF – 6 pages, 693 KB)
- 8 Tips for Android App testing – uTest (PDF – 11 pages – 717 KB)
- Strategies for Choosing Test Devices
- Strategies for selecting mobile devices for testing
- Test on Real Mobile Devices without Breaking the Bank
- Android Fragmentation Visualized
- 25 Free Google Analytics Alternatives
- Getting the Most Out of Your Mobile Functional Testing (PDF – 7 pages, 944 KB)
- Leverage Automation to Make Mobile Testing Manageable
- 8 Ways To Test A Mobile Site on Different Mobile Devices
- Testing Mobile Devices is About More than Vendors, It’s About Device Versions
- The Crowd May Have Answers, but Agencies Must Define The Problem
- Using Crowdsourcing in GovernmentThe Mobile Tester Newsletter is sent bi-monthly to the volunteer testers in our Federal Crowdsource Mobile Testing Program. Each newsletter contains a feature article, and interesting trends and statistics from the world of mobile testing. To receive our next edition, sign up to become a tester.