Very bad network simulation for testing of mobile applications [PART 1]

Motivation

Mobile internet is a must for smartphones. Most of the apps are somehow connected to the server, syncing every now and then. Whether it is to just show an advertisement, syncing your local changes with your profile somewhere in the cloud or maybe protecting the app from being distributed as cracked one, without paying for it.

But there is also another category of mobile applications, which heavily depend on the Internet connection. One example of such are applications intended for communication. Let's consider for instance PhoneX app.

All its features (secure calls, secure messaging and secure file transfer) require a decent Internet connection to work. But it just begins with its main features, everything from authentication, through contacts presence and server push notifications establish TCP or UDP connections with the servers.

Disadvantages of traditional way

With such applications, QE teams have to devote non-trivial effort to test applications functionality under various network conditions. There are various ways how to simulate real user conditions. Firstly, one can buy SIM cards for all of his devices, enable mobile data and spend lot of time with travelling around city. This method makes the testing environment the most real one, but one has to consider its downsides as well:

• Out of reach of your computer, its more difficult to automate some of the app routines while you are moving, to offload mundane repeating of interactions with the app. In your office, it would be more easy to setup a script or to write a functional test which would send 200 subsequent messages or so.
• Quality of service statistics vary around the globe significantly. And you do not have to go so far. For example 3G bandwidth, latency and jitter is quite different in two towns not far away from each other (100 Km). Needles to say that some places can only dream about LTE, and that these QoS characteristics vary also according to day hour (you would not like testing at 1 AM somewhere in the public transportation). Simulating all these different conditions in laboratory would be indeed more efficient.
• It would be more difficult to intercept the communication e.g. with WireShark. It is sometimes handy, when developers need to see actual transmitted packets, in order to fix the issue.
• It is more reliable to save mobile system logs such as logcat on Android right to the computer. Do now know why, but it is often the case for me, that some of the logs are missing when saving them to the file on the device (maybe some buffer limitation, who knows). I found more reliable to have phones connected to the computer and save such logs right away there.
• Total lost of connection, or lost of some of the packets is more easily to be scripted in your testing laboratory, then in the real world.
• Users use also various WIFI APs, which restrictions (e.g. isolation of clients) can badly affect your application features.
• The most obvious reason is the time spent while moving out there, comparing with the time spent in the comfort of your air conditioned office furnitured with the most ergonomic seats out there.

For sure there are other reasons, why I consider simulating of poor internet connection to be done in the laboratory as better option than trying to reproduce the bugs outside. Please, I am not saying that it can substitute all testing while you are moving with the device. I am just saying that it can replace most of the testing under various network conditions.

Next part

In the next part we we will look into how to setup a WIFI Access Point, and some scripts which would enable simulating of poor internet connection. iOS platform has a solution for this already (Settings -> Developer -> Network Link Conditioner). Our solution would be platform independent, and would solve all of the disadvantages described above. Stay tuned.