Sunday, July 19, 2015

Very bad network simulation for testing of mobile applications [PART 2]

In the previous post we talked about the need for platform independent, scriptable solution for testing of your mobile applications in a poor internet conditions. To complement the theory with something executable, this post will introduce scripts (Debian Linux like), and a guide to setup your own WIFI access point, which would simulate slow, unreliable mobile internet. You will be able to connect with your Android, iOS, Windows, whatever devices and see from your office, how your apps adapt.

This tutorial will be divided into following sections:

  1. Failure when firstly attempting to solve this problem.
  2. Obtaining the right USB WIFI dongle.
  3. Tutorial for creating an AP from your Linux based workstation.
  4. Script for changing the Quality of service (QoS) characteristics of your AP.
  5. Script for setting particular QoS, simulating GRPS, EDGE, 3G, LTE, whatever networks.
  6. Example usage

Failure, when firstly attempting to solve this problem.

My first attempt did not end up successfully. I am not saying it is a wrong way, but I was just not able to go this way. The plan was to:

  • Buy WIFI dongle with ability to be in AP mode.
  • Virtualize OpenWRT (a small Linux distro, usually run on routers) in the VirtualBox.
  • Install on that virtual machine a Cellular-Data-Network-Simulator - which is capable running on OpenWRT, and is established on well known technologies: tc, iptables and CloudShark.
  • Connect with devices to that AP, and use CloudShark to sniff the network in order to see particular packets.
It looked promising. It would be just an integration of already existing parts, not reinventing a wheel. A fairy tail. It worked. Even when I found out that it would require some work to do in order to script the way, how the devices are connected to the Cellular-Data-Network-Simulator, and the way how the QoS characteristics are changed in order to switch among 2G, 3G, etc. networks. But it was nothing impossible to overcome. The biggest problem which I encountered after I set it up, was the stability of the AP. It switched off the WIFI dongle at random intervals. I studied various OpenWRT log files, but have not found the root cause, hence I was not able to fix it. I needed to think out a different way. Following describes my second attempt, which finally worked.

Obtaining the right USB WIFI dongle.

First things first. Before buying the WIFI dongle, checkout its chipset, and see, whether it is supported by some Linux driver. I am using TP-Link TL-W22N. Its chipset AR9271 is supported by ath9k_htc driver.

Tutorial for creating an AP from your Linux based workstation.

Next you will need to setup various things properly: hostapd, DHCP server, firewall. I followed this great post (automated in the install script for Debian like systems here). In that install script, you can also spot a part (wifi_access_point), which would enable you to start the AP as a service.

Script for changing of Quality of service (QoS) characteristics of your AP.

Now, you should be able to connect to the created AP with your devices. It should provide you a similar Internet connection quality as you have on your workstation. To simulate various cellular data networks we need to limit it somehow.

Following script does it by setting various firewall rules. You will need to alter it a bit before using it.
  1. Set IF_IN to network interface name which is dedicated to the created AP.
  2. Set IF_OUT to network interface name by which is your workstation connected the Internet.
  3. Set IP_IN to a IP address space which will be assigned to your connected devices (you chose this when setting up a DHCP server).
  4. Set IP_OUT to the IP address of your application backend server.
Save the following script, and named it e.g. qos.sh.
#!/bin/bash
#
#  tc uses the following units when passed as a parameter.
#  kbps: Kilobytes per second 
#  mbps: Megabytes per second
#  kbit: Kilobits per second
#  mbit: Megabits per second
#  bps: Bytes per second 
#       Amounts of data can be specified in:
#       kb or k: Kilobytes
#       mb or m: Megabytes
#       mbit: Megabits
#       kbit: Kilobits
#  To get the byte figure from bits, divide the number by 8 bit
#

#
# Name of the traffic control command.
TC=/sbin/tc

# The network interface we're planning on limiting bandwidth.
IF_IN=wlan1
IF_OUT=wlan0

# IP address of the machine we are controlling
IP_IN=10.10.0.0     # Host IP
IP_OUT=0.0.0.0 #the address of your backend server

# Filter options for limiting the intended interface.
U32_IN="$TC filter add dev $IF_IN protocol ip parent 1: prio 1 u32"
U32_OUT="$TC filter add dev $IF_OUT protocol ip parent 2: prio 1 u32"

start() {
    ping -c 1 $IP_OUT >/dev/null 2>&1
    if [ $? -ne 0 ]; then
 echo "Error:"
        echo "The IP address: $IP_OUT is not reachable!"
 echo "Check out the backend server address!"
 exit -1
    fi

    $TC qdisc add dev $IF_IN root handle 1: htb default 30
    # download bandwidth
    $TC class add dev $IF_IN parent 1: classid 1:1 htb rate "$1"
    $U32_IN match ip dst $IP_IN/24 flowid 1:1
    # in delay
    $TC qdisc add dev $IF_IN parent 1:1 handle 10: netem delay "$3" "$4" distribution normal
    # in packet loss
    $TC qdisc add dev $IF_IN parent 10: netem loss "$7" "$8"

    # upload bandwidth
    $TC qdisc add dev $IF_OUT root handle 2: htb default 20
    $TC class add dev $IF_OUT parent 2: classid 2:1 htb rate "$2"
    $U32_OUT match ip dst $IP_OUT/32 flowid 2:1
    # out delay
    $TC qdisc add dev $IF_OUT parent 2:1 handle 20: netem delay "$5" "$6" distribution normal
    $U32_OUT match ip dst $IP_OUT/32 flowid 20:
}

stop() {

# Stop the bandwidth shaping.
    $TC qdisc del dev $IF_IN root
    $TC qdisc del dev $IF_OUT root
}

show() {

# Display status of traffic control status.
    echo "Interface for download:"
    $TC -s qdisc ls dev $IF_IN
    echo "Interface for upload:"
    $TC -s qdisc ls dev $IF_OUT

}

case "$1" in

  start)
    if [ "$#" -ne 8 ]; then
        echo "ERROR: Illegal number of parameters"
        echo "Usage: ./qos.sh start [downloadLimit] [uploadLimit] [inDelayMax] [inDelayMin] [outDelayMax] [outDelayMin] [packetLossPercentage] "
        echo "[downloadLimit]  See man page of tc command to see supported formats, e.g. 1mbit."
 echo "[uploadLimit] The same as for downloadLimit applies here."
 echo "[inDelayMax] Max delay in miliseconds for requests outgoing from AP."
 echo "[inDelayMin] Min in delay."
 echo "[outDelayMax] Max Delay in miliseconds for requests outgoing to servers."
 echo "[outDelayMin] Min out delay."
 echo "[packetLossPercentage] The percentage of packet lost"
 echo "Example: /qos.sh 1mbit 1mbit 50ms 20ms 30ms 10ms 5%"
        exit -1
    fi

    echo "Starting  shaping quality of service: "
    start $2 $3 $4 $5 $6 $7 $8
    echo "done"
    ;;

  stop)

    echo "Stopping shaping quality of service: "
    stop
    echo "done"
    ;;

  show)

    echo "Shaping quality of service status for $IF_IN and $IF_OUT:"
    show
    echo ""
    ;;

  *)

    pwd=$(pwd)
    echo "Usage: qos.sh {start|stop|show}"
    ;;

esac

exit 0

Script for setting particular QoS, simulating GRPS, EDGE, 3G, LTE, whatever networks.

Now when you have a script to limit the QoS characteristic of your created AP, you will need to do some measurements, in order to have a clue what bandwidth, what latency, and what packet loss various cellular data networks have. You will need to find out a way how to measure these characteristics in the environment where your customers use your application.

The reason is that the same data network type (e.g. 3G) can have different QoS characteristics on different places. There are other factors in play as well: mobile Internet provider, hour of the day, city vs. village, weather and like. For the measurement I used handy mobile applications Speedtest.net (for bandwidth and latency) and Fing for a double check up, as it is able to ping any server you like.

Off-topic: Would not be awesome, if there is a web service which would give me the average QoS characteristics of any place in the world for a particular hour of the day, for a particular data carrier, for particular weather and other conditions? I submitted a bachelor thesis assignment, but so far no enrollment :) And it would be IMO quite easy to setup: a mobile application with gamification characteristics to find out the particular statistics, store them, and make them available via some REST endpoints.

Save the following script as simulate.sh, into the same directory as previous script was saved. The measured values, you can see, are valid for average morning in Brno, Czech republic. The average was made after one week of measuring.

#!/bin/bash

QOS="./qos.sh"
echo -n "Shaping WIFI to "

case "$1" in

    GPRS)
 echo "GRPRS"
 $QOS stop > /dev/null 2>&1
 $QOS start 80kbit 20kbit 200ms 40ms 200ms 40ms 5%
 ;;

    EDGE)
 echo "EDGE"
 $QOS stop > /dev/null 2>&1
 $QOS start 200kbit 260kbit 120ms 40ms 120ms 40ms 5%
 ;;

    HDSPA)
 echo "HDSPA"
 $QOS stop > /dev/null 2>&1
 $QOS start 2400kbit 2400kbit 100ms 100ms 100ms 100ms 5%
 ;;

    LTE)
 echo "LTE"
 ;;

    FULL)
 echo "FULL"
 $QOS stop > /dev/null 2>&1
 ;;

    DISABLED)
 echo "DISABLED"
 $QOS stop > /dev/null 2>&1
 $QOS start 1kbit 1kbit 5000ms 5000ms 5000ms 5000ms 5%
 ;;

    *)
        pwd=$(pwd)
        echo "Usage: simulate.sh {GPRS|EDGE|HDSPA|LTE|FULL|DISABLED}"
        ;;

esac

exit 0

Example usage

So if you followed the steps, you should be able now to:
  1. Start the AP by: service wifi_access_point start.
  2. Simulate e.g. EDGE, by issuing: simulate.sh EDGE
Ideas have no limits. Use these scripts to e.g. stress network test your application (write a bash script which would randomly switch among all types of the network in a random intervals), or use WireShark to go deeper, to see actual packets being transmitted. Your development team would love you, if you attach to your bug report a saved transmission with a packet level information. Fixing of tough, non-deterministic network issues becomes more easy.

Disclaimer: I am still improving the scripts, use them on your own danger :) Any feedback on how you utilized these scripts, or your improvements would be deeply appreciated.

10 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. Thank you for this great post!
    I've been looking for a solution to create a bad network environment at work for our engineers to test applications under different network types.
    Couple questions as I was originally planning on using 'Cellular-Data-Network-Simulator'.
    I was going to buy a new router with rich features of QoS and OpenWRT capabilities.
    However, with your strategy, it seems that I would only need WIFI dongle to create a separate AP for testing purposes only?
    Also can I accomplish this using OSX instead of Linux based machines?
    Thank you again!

    ReplyDelete
    Replies
    1. Hello, thank you for your feedback.

      Yes, a WIFI dongle would be enough.

      Regarding to OSX. IMHO it should work there as well. It would require some tuning, but it should be doable.

      The key parts are: hostapd, DHCP server, firewall (iptables). If there are such components, or there are equivalents you should be OK.

      Recently I have encountered a small problem with this solution. When packet loss is set, then it is sometimes unstable.

      Give it a try and let me know whether you are successful :)

      Delete
    2. hi I purchased a this USB adapter https://www.thinkpenguin.com/gnu-linux/penguin-wireless-n-usb-adapter-gnu-linux-tpe-n150usb
      However, I am having really difficult time setting up AP.
      Whenever I run 'sudo service hostapd start' command, my linux machine internet gets disconnected.
      My android devices are able to connect to the AP with no internet service, but is getting IP address.
      However, my linux machines' internet is completely gone whenever I run service hostapd start command..

      Any ideas why?
      thanks

      Delete
    3. maybe I ask you which Ubuntu you are using as well?
      Are you using TP-LINK TL-WN722N Wireless adaptor?

      Thanks

      Delete
    4. Yes, I am using TL-WN722N (but I guess it should not matter that much if the chipset is the same). I am using Ubuntu 14.10, kernel 3.16.0-29.

      I remember now one problem with hostapd. I had to install lover version because of some bug in it. I locked the version for hostapd, so Ubuntu will not update it. I am using 1:1.0-3ubuntu1 version. If your problems persist try to study /var/log/syslog file, and see the error message if any, once you start the wifi service.

      My second suggestions is:
      * check interface names in the configuration files
      * check whether you have not set any iptables rules which would block Internet for connected devices

      Delete
    5. So it's not normal that 'sudo service hostapd start' disables my internet from linux work station, correct?
      I see the AP name on my devices and able to receive IP address (although can't connect to internet).
      I've been working on this project for a long time now, but I still haven't found a real solution.. :(
      I wonder if it's due to using my old netbook..

      Delete
    6. If everything is configured correctly, then it should not disable Internet from your workstation.

      IHMO you are quite close to the solution if you are able to obtain an IP address. Now you should continue with setting up MASQUERADE.

      Another source to follow, a script which automates all settings for AP:
      https://gist.github.com/dashohoxha/5767262

      Delete
    7. Hi, I think everyone works well until I try to run qos.sh script.
      For example after giving permission to all the files, I simply ran ./qos.sh start 200kbit 260kbit 120ms 40ms 120ms 40ms 5%
      and I keep receiving "Command line is not complete. Try option "help"" message.

      I believe I gave right IP_IN and IP_OUT (ran ip route get 8.8.8.8 | awk '{print $NF; exit}')
      Why am I keep receiving that message?

      Delete
  3. Thanks again!
    I just ordered couple wifi adapters
    Hopefully I can get it work flawlessly!
    I will update you with my progress.
    Really appreciate your post here, thanks

    ReplyDelete