Very bad network simulation for testing of mobile applications [PART 1]

Motivation

Mobile internet is a must for smartphones. Most of the apps are somehow connected to the server, syncing every now and then. Whether it is to just show an advertisement, syncing your local changes with your profile somewhere in the cloud or maybe protecting the app from being distributed as cracked one, without paying for it.

But there is also another category of mobile applications, which heavily depend on the Internet connection. One example of such are applications intended for communication. Let's consider for instance PhoneX app.

All its features (secure calls, secure messaging and secure file transfer) require a decent Internet connection to work. But it just begins with its main features, everything from authentication, through contacts presence and server push notifications establish TCP or UDP connections with the servers.

Disadvantages of traditional way

With such applications, QE teams have to devote non-trivial effort to test applications functionality under various network conditions. There are various ways how to simulate real user conditions. Firstly, one can buy SIM cards for all of his devices, enable mobile data and spend lot of time with travelling around city. This method makes the testing environment the most real one, but one has to consider its downsides as well:

• Out of reach of your computer, its more difficult to automate some of the app routines while you are moving, to offload mundane repeating of interactions with the app. In your office, it would be more easy to setup a script or to write a functional test which would send 200 subsequent messages or so.
• Quality of service statistics vary around the globe significantly. And you do not have to go so far. For example 3G bandwidth, latency and jitter is quite different in two towns not far away from each other (100 Km). Needles to say that some places can only dream about LTE, and that these QoS characteristics vary also according to day hour (you would not like testing at 1 AM somewhere in the public transportation). Simulating all these different conditions in laboratory would be indeed more efficient.
• It would be more difficult to intercept the communication e.g. with WireShark. It is sometimes handy, when developers need to see actual transmitted packets, in order to fix the issue.
• It is more reliable to save mobile system logs such as logcat on Android right to the computer. Do now know why, but it is often the case for me, that some of the logs are missing when saving them to the file on the device (maybe some buffer limitation, who knows). I found more reliable to have phones connected to the computer and save such logs right away there.
• Total lost of connection, or lost of some of the packets is more easily to be scripted in your testing laboratory, then in the real world.
• Users use also various WIFI APs, which restrictions (e.g. isolation of clients) can badly affect your application features.
• The most obvious reason is the time spent while moving out there, comparing with the time spent in the comfort of your air conditioned office furnitured with the most ergonomic seats out there.

For sure there are other reasons, why I consider simulating of poor internet connection to be done in the laboratory as better option than trying to reproduce the bugs outside. Please, I am not saying that it can substitute all testing while you are moving with the device. I am just saying that it can replace most of the testing under various network conditions.

Next part

In the next part we we will look into how to setup a WIFI Access Point, and some scripts which would enable simulating of poor internet connection. iOS platform has a solution for this already (Settings -> Developer -> Network Link Conditioner). Our solution would be platform independent, and would solve all of the disadvantages described above. Stay tuned.

Recording tests on Android (neither root, nor KitKat required)

A test suite is good only when provides a good feedback. Testing mobile apps is cumbersome, and far from robust (actually all UI tests are like that). A meaningful test report is inevitable. That is why, I really like to have executions of my tests recorded. Such recordings are great thing to avoid a repeated execution of the test to find out why it failed (repeated execution of tests should be avoided as plague).

It is awesome that Google added a a native support for recording of your Android 4.4.x+ device screen, but what the other folks with lower Android versions. We can not afford to test only on 4.4+, as it is wise to support at least 4.0+. A rooted device is not the answer for me, as we need to test on real devices, devices which are actually used by our customers.

OK, all Android versions are capable to take a screen capture, so why not to use this feature. The following describes small bash scripts, which in simple words create a video (actually a .gif with 2fps) from such screen captures. It is then easy to use them to record your functional UI tests (showcased on Appium tests).

Firstly, the script which takes screenshots until not terminated into a specified directory on your device:

#!/bin/bash
#take-screenshots.sh
DIR=/sdcard/test-screenshots
adb -s $1 shell rm -r $DIR > /dev/null 2>&1
adb -s $1 shell mkdir $DIR > /dev/null 2>&1
for (( i=1; ; i++ ))
do
 name=`date +%s`
 adb -s $1 shell screencap -p "$DIR/$name.png"
done

You can try it by executing ./take-screenshots.sh [serialNumberOfDevice].

Secondly, the script which retrieves taken screenshots from devices into your computer, re-sizes them into smaller resolution, and finally creates an animated .gif:

#!/bin/bash
#pull-screenshots-and-make-video.sh
DIR_REMOTE=/sdcard/test-screenshots
mkdir "$1"
cd "$1"
adb -s $1 pull $DIR_REMOTE
echo "Resizing screenshots to smaller size!"
mogrify -resize 640x480 *.png
echo "Converting to .gif."
convert -delay 50 *.png "$1"-test.gif
echo "Clearing..."
cp "$1"-test.gif ..
cd ..
rm -rf "$1"

Try it by executing ./pull-screenshots-and-make-video.sh [serialNumberOfDevice] [pathToDirectoryIntoWhichSaveScreenshots]. Just to note that it uses the imagemagic and its sub packages.

Here is an example of .gif created by scripts above, while sending encrypted files through the PhoneX app for a secure communication:

So we have some scripts to execute (indeed there are things to improve, a parameter checking etc.). There are various ways how to use them in your tests, all depend on what testing framework you are using, and in what language your tests are written in. We use the Appium, and its Java client. Following shows executing of the first (take-screenshots.sh) script in the beginning of each test class:

public class AbstractTest {
    private Process takeScreenshotsProcess = null;

    @Before
    protected void setupDevice1() throws Exception {
        takeScreenshotsProcess = startTakingOfScreenshots(DEVICE_1_NAME);
        //for readability omitted Appium API calls to setup device for testing
    }

    protected Process startTakingOfScreenshots(String deviceName) throws Exception {
        String[] cmd = { "sh/take-screenshot-on-device.sh", getDeviceSerialNumber(deviceName)};
        return Runtime.getRuntime().exec(cmd);
    }

    @After
    public void tearDown() {
        if(takeScreenshotsProcess != null) {
           takeScreenshotsProcess.destroyForcibly();
        }
    }
}

Hopefully the code above is somehow self explanatory. It starts taking of screenshots before Appium API calls prepare a device for a testing (installs APK, etc.). The same pattern can be used for any number of devices.

Next steps are to use the pull-screenshots-and-make-video.sh script in the end of your CI job (e.g. Jenkins). I prefer fine granular CI jobs, which are short to execute, to provide a quick feedback. Therefore, each job is a one test (or matrix of tests), and that is why, starting of taking screenshots is done in the @Before method, and terminated in the @After method.

Please, bear in mind, that previous are just examples. They need to be polished and altered to ones needs. Enjoy testing.