Methodology
This page presents Open Nettest and describes the methodology of the measurement. Open Nettest has adopted the methodology for the measurement based on the open source measurement tool developed by the regulatory authority in Austria, RTR-NetzTest [1]. We measure the connection from your device (we often call it a client device, or just client) and a measurement server (located at an Internet exchange point, to keep the measurement as independent of the operator and the service as possible). The client and the measurement server will send data to each other, which will be used to measure how much data they can transfer in a given amount of time, letting us measure the speed and the latency of your Internet connection. There are several measurement servers available.
[1]
Figure 1. Main components of Open Nettest measurement platform
Figure 1 shows a simplified view of the measurement platform. The design of the system architecture has taken into account BEREC’s recommendations for a measurement tool [2].
Open Nettest measurement
From the factors that can impact the result of the measurement, the methodology for the measurement itself plays a critical role. That is one of the main reasons for different measurement tools (Speedtest, Fast, M-Lab, etc.) to provide different results. However, many of the currently available tools are not open source and there is little disclosure about the specific methodology they follow to provide a result. Open Nettest is open source and aims to provide transparent and comprehensible information. With every measurement, Open Nettest collects information related to the Internet connection. There are two sets of measurements:
The QoS measurement provides downlink and uplink speeds and latency of the connection. If you are running the measurement from a mobile app, the base measurement will also provide jitter (variation in the delay) and packet loss measurements.
Additionally, the user can enable the Net Neutrality and other network tests. This is a set of measurements that checks the viability of other services related to network neutrality, which can provide very insightful information to the user and regulatory authorities about the neutrality policies.
The QoS measurement aims at estimating the performance of the user’s Internet connection by measuring download and upload speeds and latency (jitter and packet loss are measured as well from the mobile applications). In order to do so, every measurement consists of several phases that run sequentially. We provide a brief summary of the different phases.
Phase 1: Initialization
The client tries to establish a secure connection to the control server and they exchange the configuration parameters that are needed to perform the measurement.
Phase 2: Downlink pre-test
This phase aims to get the connection ready to perform the measurements that will follow.
With the information received from the control server, the client can start one or more parallel connections, also called flows, to the measurement server. The number of connections is variable depending on the speed of your Internet connection. In slower connections, a higher number of flows will result on each of them competing with one another for low bandwidth, leading to even worse results and a less accurate measurement. On the other hand, for faster connections, opening several connections simultaneously will allow to reach the maximum capacity of the link faster, improving the accuracy of the result.
In this phase, the client and the server will estimate the optimal number of parallel connections based on the state of the network. They will try to set a higher number of connections if the network conditions allow it. Otherwise, it will be reduced to just one connection.
The number of parallel connections is configurable by the user in Open Nettest. However, if you are not sure how to select this number, you can let the client and server decide based on the current network conditions.
Phase 3: Latency
In this phase, the client will send several short messages to the server in short intervals, wait for the reply from the server and send a confirmation when this reply is received. Then, the client and the server measure the time between the transmission of their message and the reception of the reply. The measurement result Ping shows the median of all the values measured by the server during this phase.
Phase 4: Packet Loss
This phase takes place only when the measurement runs on a mobile application, but not from the web browser. In this phase the jitter (how much the delay varies) and packet loss characteristics of the Internet connection are determined. The measurement is based on the same methodology as the VoIP test - the client sends a series of numbered UDP datagrams with fixed size to the measurement server. On the receiving side, the server calculates the packet loss and jitter.
Phase 5: Downlink
In this phase, the client requests data through each of the parallel connections to the server (according to the result from the downlink pre-test). The server will transmit the data to the client in chunks of the size indicated by the client. The client, upon the reception of each chunk, tracks the volume of data received and the time needed to receive it.
After receiving the last chunk of data, the client will calculate the download speed of the connection taking into account the values measured for all the connections.
Phase 6: Uplink pre-test
Similarly to the downlink pre-test phase, this phase aims to get the connection ready for the speed measurement in the direction from the client to the server. Again, if the state of the network does not allow it, only one connection will be used for the uplink measurement.
Phase 7: Uplink
In this phase, the client will send data to the server through each of the parallel connections (according to the result from the uplink pre-test) in chunks of a size established by the client. The server will measure the time and the amount of data received and send this information back to the client. The client, after receiving all the information from the server will calculate the uplink speed.
Phase 8: Finalization
After finishing the measurements, the client will close the connections to the measurement server and send all the collected data to the control server.
Net Neutrality and other Network tests
Based on BEREC’s recommendations Open Nettest provides the tools to evaluate characteristics of the network connection that have a decisive influence on the quality and the transparency of the Internet access. In the recent years there is a strict regulation under development in Europe to ensure transparency and non-discriminatory treatment of data traffic in the Internet access services, protecting end user’s rights. It consists on a set of 7 test groups, which are carried out after the base measurement, if the option is enabled from the mobile applications for Android and iOS. After all the tests are complete, we compute a score taking into account how many tests failed.
Web content modification
The goal of this test is to detect, whether the content is modified during the transmition of the data between the server and the client in some intermediate network device. The client requests data with a known checksum from the configured web server. Upon reception of the data, the client compares the checksum of the received data to the original checksum. A difference in the checksums indicates the manipulation of the data on the way to the client, and the test is considered not successful.
Website loading
In addition to checking modifications of the content, we monitor the size and the time it takes to download one or more reference web pages. If the pages can be rendered before a defined timeout, the test is successful.
Connection transparency
The use of proxies is a common technique in the current Internet, for instance for caching the most frequently visited webpages, compress content or protect clients from potentially harmful content. In this test, we aim at detecting the presence of proxies between client and server, or "non-transparent" connections. In that case, it cannot be guaranteed that the websites correspond exactly to those on the server. If we detect the presence of such proxy, the test is unsuccessful. In this test the client sends correctly formatted HTTP requests, as well as incorrectly formatted HTTP requests to the measurement server. The measurement server reply by sending the exact copy of the request back to the client. When there is an http proxy between the client and the measurement server, it will react to the incorrectly formatted HTTP requests by sending an error message, indicating the malformed HTTP request. In this case, the test is unsucessful.
DNS
DNS (Domain Name Service) translates the name of the website we visit on the Internet to an IP address, which will indicate the specific location of the content we can access. It is therefore essential to check if the translation provided by the DNS service can be trusted and is compliant with the network neutrality policies. This test is successful if we can confirm that the DNS data is consistent.
TCP and UDP blocked ports
Common protocols used for different Internet services use certain port numbers in their connections. If a certain port is blocked, by the ISP or by the network administrator, any communication attempt through this port will fail. In this test we check whether any TCP or UDP ports are blocked, and therefore certain services are banned, as data cannot be sent or received using those ports. Note that in business environments it is a common practice to block certain ports for security reasons (firewall) or only the relevant ports (and services) are made available.
VoIP
In this test we simulate a Voice over IP type of communication. We monitor the connection and measure the relevant parameters, as delay or jitter. We estimate if the quality of the connection would support a VoIP call, in which case the test is successful.
Route to the server (Traceroute)
In this test we measure the distance (in terms of delay and network hops) between the client and a target location.
Understanding the results
After a measurement is complete, Open Nettest will show you the results, with different level of detail, depending on whether you are using a mobile client or a web browser.
The speed values that you should expect depend not only on the network status, but on your access technology (DSL, WiFi, 3G, optical fibre, etc.) and potential limitations of your client device, among other factors. The download and upload speeds are shown in Megabits per second (Mbps) and the latency (Ping) in milliseconds.
How to interpret if the result is good enough or bad depends on the type of service you would want to use. For web browsing, often 2 Mbps are often good enough. Voice services may not need high bandwidth, but delay and jitter (how much the delay varies) are very important. Therefore, you should not only consider the download speed, but also the upload (especially for cloud services or file transfers) and latency.
To help you, we have included a red/orange/green color code to indicate what is a high/medium/low result. Note that the higher speeds (indicated by green) might only be achievable by certain technologies. Depending on your access network technology, a red speed value could indicate that your connection uses an older technology and might need an upgrade.
When interpreting the results, you should keep in mind that even if the results differ from the advertised speed in your Internet connection contract, this does not necessarily mean that your Internet provider is not fulfilling the agreement. Moreover, operators design their service offer in different ways. Typically, the speed of the data connection is indicated as an upper bound ("up to" values).
If after several measurements the results show a significant difference from the advertised bandwidth, this may indicate that there are problems with the connection, and it should be analysed by a professional. The result can also be impaired by various technological factors like the WLAN router, system configuration, etc. In addition, if your network is shared among several users, the total capacity available will be distributed among them. Moreover, although we work to keep our systems available at all times, it may happen that the test server or its connection are overloaded.
Since operators implement different policies to route the traffic through their networks, it might happen that the result of Open Nettest reaches a higher value than normally accessible by other Internet services. However, the design of the measurement tries to follow common Internet practices to be as close to the user experience as possible. Moreover, if services are treated differently, this might be a hint of the lack of network neutrality.
In any case, the most reliable way to avoid biases on the result and random errors, is to repeat the measurement frequently.
Plausibility checks
Sometimes, due to circumstances out of our control, there are results that are inconsistent, impossible or contradictory. After the completion of the measurement, we postprocess the data collected to identify these measurements and troubleshoot what could have happened to have those results. For instance, we may measure a very high download speed in a 2G connection (higher speed than possible for this type of networks) when there is a change of technology (e.g. from 2G to 4G) during the test. Thanks to the extensive set of data we collect during the measurement, we can identify these situations, among others, which may lead to contradictory results.
Quality of Experience estimation
Based on the obtained measurement results, we offer an estimation of the expected Quality of Experience (QoE) the user would achieve when using the most popular network applications, according to different categories:
- Video streaming
- Web browsing
- Photo sharing/social media
- Online gaming
- VoIP
We combine the results from the measurement to estimate the expected performance of these services, following standard recommendations from different sources (BEREC [2], ITU-T [5,6], content providers [7,8,9] and user surveys [4]).
QoE estimations
Video streaming
| Related metrics | Excellent | Good | Moderate | Poor | Bad |
|---|---|---|---|---|---|
| DL | >=25 Mbps | >=5 Mbps | >=3 Mbps | 3>DL>=1 Mbps | <1 Mbps |
| UL | >=1.5 Mbps | >=500 kbps | >=100 kbps | >=50 kbps | <50 kbps |
| Latency | <50 ms | <100 ms | <150 ms | <200 ms | >=200 ms |
Web browsing
| Related metrics | Excellent | Good | Moderate | Poor | Bad |
|---|---|---|---|---|---|
| DL | >=10 Mbps | >=5 Mbps | >=3 Mbps | >=1 Mbps | <1 Mbps |
| UL | >=0.5 Mbps | >=300 kbps | >=100 kbps | >=50 kbps | <50 kbps |
| Latency | <50 ms | <100 ms | <150 ms | <200 ms | >=200 ms |
| Related metrics | Excellent | Good | Moderate | Poor | Bad |
|---|---|---|---|---|---|
| DL | >=3 Mbps | >=1 Mbps | >=0.5 Mbps | >=0.1 Mbps | <0.1 Mbps |
| UL | >=0.5 Mbps | >=300 kbps | >=100 kbps | >=50 kbps | <50 kbps |
Photo sharing / Social media
| Related metrics | Excellent | Good | Moderate | Poor | Bad |
|---|---|---|---|---|---|
| DL | >=10 Mbps | >=5 Mbps | >=1.5 Mbps | >=0.5 Mbps | <0.5 Mbps |
| UL | >=5 Mbps | >=3 Mbps | >=1.5 Mbps | >=1 Mbps | <1 Mbps |
Online gaming
| Related metrics | Excellent | Good | Moderate | Poor | Bad |
|---|---|---|---|---|---|
| DL | >=10 Mbps | >=5 Mbps | >=3 Mbps | 3>DL>=1 Mbps | <1 Mbps |
| UL | >=5 Mbps | >=3 Mbps | >=1 Mbps | >=500 kbps | <500 kbps |
| Latency | <50 ms | <100 ms | <150 ms | <200 ms | >=200 ms |
VoIP
| Related metrics | Excellent | Good | Moderate | Poor | Bad |
|---|---|---|---|---|---|
| DL | >=1.5 Mbps | >=1 Mbps | >=0.5 Mbps | >=0.2 Mbps | <0.2 Mbps |
| UL | >=1.5 Mbps | >=1 Mbps | >=0.5 Mbps | >=0.2 Mbps | <0.2 Mbps |
| Latency | <50 ms | <100 ms | <150 ms | <200 ms | >=200 ms |
References
[1] BoR (17) 178: BEREC Net Neutrality Regulatory Assessment Methodology. BEREC. Available online [2] BoR(14)117 (document 4602): Monitoring quality of Internet access services in the context of net neutrality [3] BoR(17)179 (document 7296): Net Neutrality measurement tool specification [4] Tutela, analysis of crowdsourced data and network performance recommendations (2019). [5] Rec. ITU-T G.1031 (02/2014) [6] Rec. ITU-T G.1032 (10/2017) [7] Netflix’s speed recommendations [8] Youtube recommendations [9] Skype recommendations