Why Testers Should Use Real Devices Rather Than Simulators?

“Testing on simulators will never replace testing on real handsets as there is no substitute for the real things.”

The mobile market is booming. This is evidenced by the large number of handset manufacturers as they are continuously delivering a wide variety of devices with different mobile operating systems and different sizes. As more companies and brands are turning towards developing mobile applications, their risk exposure is also high. If users don’t have an excellent experience with the app, they may switch to an alternative product. To avoid this switch and to retain users, pre-launch testing on mobile platforms has become a challenge. This may be tricky sometimes because today there is a wide range of handsets with varied specifications. Therefore, the simulator has become an important tool for developers and testers.

Simulators are very powerful tools but they are virtual devices which lack the real target environment. A virtual device is not a real phone but a software which gives the same functionality as the real phone.

But not even 0.01% of the end users use simulators after the app release. So we have to test mobile apps on a real device at some point of time during the development cycle to catch bugs at an early stage. Now the question is: which one should a tester prefer – real devices or simulators! As different kinds of users run the application on different devices, most companies prefer real devices for testing than simulators as it gives accurate results.

Limitations Of A Simulator

Processing Speed

Simulators are often laggy as it takes a lot of time to load and they often depend upon the amount of RAM allocated at the time of designing. Hence the whole process is dependent on the hardware. By using real devices things can be tested directly. So we need at least one real device available during all phases of development to assess the processing speed of the app.

Gesture Testing

Simulators can emulate touch screen gestures, but it’s a bit clunky. Using a mouse or a keyboard to click on a simulator is different from using a finger on the screen of mobile devices. Certain actions like pinching the screen, zooming the image and scrolling the screen are considerably different on touchscreen devices.

Network/WiFi

In terms of network configuration, simulators run on the PC, connect to LAN and access the internet via corporate firewall. Slow network is not possible to be tested on simulator where devices experience network issues.

Screenshots

For marketing the app we need to upload some quality screenshots in multiple sizes for each of the multiple devices. For example, we uploaded nearly 20 screenshots for our Advocare app using multiple devices. (iPhone 6+, iPhone 5S, iPod 5G, iPhone 4S). By using a simulator, we can capture these screenshots but it renders somewhat low quality. Therefore, a real device is a better choice.

Preview Videos

Videos are an excellent way to showcase our app for marketing than screenshots alone. It is possible to screencast in a simulator but simulator’s extra latency won’t show our app in the best light. The best option for video is to take it straight from the device itself, which is well recommended by Apple.

Display

It is not always about the resolution but problems are also related to the quality of a display i.e. pixel density, colors and overall quality of the display used in the device. For example, developers want to have dark blue button for login as per requirement, but due to low display quality of simulator, it is shown as shades of blue.

Real occurring events

Interruptions like incoming calls, SMS and battery consumption i.e., how charger effects overall performance can be performed in simulator but they’re not real. The only option for such events is to do it in real devices.

Hardware Features

• No support for IMEI number (00000 is returned) or device ID
• No support for Mac Address
• No support for GPS testing
• No support for Camera
• No support for Recording and Audio playback
• No support for Sensors (Gyroscope, Acceleration sensor, Gravity sensor)
• No support for USB connections
• No support for attached device headphones
• No support for determining SD card eject/insert

Software Features

• Some of the APIs behave differently or don’t work at all on the simulator (For example – Canvas API like clipPath, drawPath, etc).
• Camera and Flash functionalities cannot be tested.
• Email functionality cannot be tested.
• Installation and configuration cannot be tested.

The Verdict

Although there are a few advantages of simulators like cost- and time-efficiency, covering of more devices, easy integration at early stages of development, we must maintain the quality of the application, and this can be achieved only by testing the app on real devices as it covers different experiences.

As everything has both advantages and disadvantages, there’s no guarantee that the software which works perfectly on a simulator will definitely work in the same way on the real device. A good approach is to use simulator in the early stages of testing when the maturity level of the app is low, and use real devices in the later stages.

A lot of factors should be carefully considered like submission deadlines, costs and customer demands. A healthy mixture of real device and simulator testing can give you the test coverage you need at a reasonable price. If you’re intending to release your application in App/Play Store, or to devices, it’s worth testing it on the device  at least once. Only then can we be sure that it will act and perform as expected on the platform you intend to target.

Vikas Donkeshwara works as QA Engineer at [x]cubeLABS and has 3 years of experience in Mobile and Web Application Testing in different domains like Education, Healthcare and Finance.