Detailed Information

Drive Test

The campaign covered 50 cities and 10,000 km of Serbian roads and was divided into three area type categories: Large Cities, Small Cities and Roads. The number of tests samples collected during drive tests (in each network over 6000 voice calls and around 3500 of every data service tests were done) allows to receive accuracy better than 3% for standard error with confidence level better than 95%.

* Cities added to the scope in 2019

Drive test routes that were measured are presented in Figure 1.

Fig. 1. Drive test routes

Measurements were performed in a drive test mode, which means that the measurement equipment was installed in moving vehicles. Measurement equipment collects network data by running voice and data tests and using a scanner to obtain radio network parameters. All three mobile networks were measured at the same time and on the same drive test routes using the same smartphones – Samsung Galaxy S10 for voice and Samsung Galaxy S8 for data. The Samsung Galaxy S10 was equipped with operators’ firmware to support all latest networks features (including VoLTE – Voice over LTE calls). The Samsung Galaxy S8 is a Cat. 16 mobile device, which means that it supports 4G (LTE) data speeds up to 1000 Mbps for data receiving and 50 Mbps for data sending. All smartphones worked in automatic technology selection mode. To reflect the latest technical developments in the mobile networks and to examine the benefits from available capabilities, SIM cards with the most comprehensive mobile tariff plans (tariff plans with the highest data rates, highest number of minutes, largest amount of data volume) available from each operator were used.

Mobile Tariff plans used for testing networks are shown in Table 1:

Table 1. Mobile Tariff plans – Drive test

The measurement system consisted of two test cars equipped with identical measurement equipment (SwissQual Diversity Benchmarker II) capable of measuring all network technologies and services simultaneously to a very high accuracy. In order to perform voice tests, the Samsung Galaxy S10 smartphones permanently called each other, within the same mobile network. In Large Cities, voice tests were executed in mobile to mobile scenario between two cars, in Small Cities and on the Roads mobile to mobile calls were performed within the same car. The aforementioned specific setup in Large Cities allowed for effective data collection without performing too much voice traffic within the single radio cell in areas where higher mobile traffic is expected. Voice tests assess network accessibility, retainability and quality of speech. Voice calls with 85 seconds call duration were measured during benchmarking.

Test equipment installed in cars is presented in Figure 2.

Fig. 2. Test Equipment installed in cars

The receiving or sending of additional data during the voice test call was added to the measurement scenario in order to simulate behaviour of a regular subscriber using a smartphone device, for which background data transmissions are typical during the voice call. For each of the voice calls, the quality of speech samples was measured (MOS – Mean Opinion Score) using the standard POLQA P.863 algorithm.

Data tests were performed using Samsung Galaxy S8 smartphones and a dedicated measurement server located at the Serbian Open eXchange (SoX) in Belgrade, which ensured a fair transmission path to all three mobile networks. Data tests assess the network availability, stability, typical performance and highest capabilities. The most representative data services measured during benchmarking were:

• HTTP File Transfer - Download (throughput of a 3 MB file transmission over HTTP protocol from the measurement server to the smartphone). The small file transfer Download test is designed to measure the responsiveness of the network and simulate a user downloading small files such as pictures, mp3 files or email attachments.
• HTTP File Transfer - Upload (throughput of a 1 MB file transmission over HTTP protocol from the smartphone to the measurement server). The small file transfer Upload test is designed to measure the responsiveness of the network and simulate a user uploading small files such as pictures, mp3 files or email attachments.
• FDTT HTTP Capacity Transfer - Download (throughput of a 400 MB file transmission over HTTP protocol from the measurement server to the smartphone during the transmission time of 10 seconds). This test is designed to reach network capabilities to deliver highest continuous throughput which is important for heavy users downloading large files – typical for users using mobile connections for their PC / laptops.
• FDTT HTTP Capacity Transfer - Upload (throughput of a 400 MB file transmission over HTTP protocol, from the smartphone to the measurement server, during the transmission time of 10 seconds). This test is designed to reach network capabilities to deliver highest continuous throughput which is important for heavy users uploading large files – typical for users using mobile connections for their PC / laptops.
• HTTP Browsing Static (testing how fast the reference ETSI Kepler web page is received and opened on smartphones).
• HTTP browsing Live Page (testing how fast the real web pages are received and opened on smartphones).
• YouTube (testing the quality of live stream video transmission)
• Ping (measuring delay between sending and receiving packets inserted by the network).

To simulate the behaviour of average mobile subscriber in Serbia surfing the Internet, a set of websites was tested, based on their popularity amongst Serbian users:

https://www.twitter.com
https://www.facebook.com
https://www.kupujemprodajem.com
https://www.kupindo.com

In the YouTube test, the quality of a live stream was measured (VMOS - Video Mean Opinion Score) using the standard J.343.1 algorithm.

A scanner (Rhode&Schwarz TSME) was used to test radio parameters of the mobile networks. SwissQual NQDI software was used for network data post-processing and reporting. The post-processing activity was supported by the use of the Systemics-PAB proprietary Data Warehouse for a customized analysis.

Hotspot

The campaign covered 4 locations in Belgrade and 1 in Novi Sad.

Hotspot locations in Beograd that were measured are presented in Figure 3

Fig. 3. Hotspot locations in Beograd

Hotspot location in Novi Sad that was measured is presented in Figure 4.

Fig. 4. Hotspot location in Novi Sad

Measurements in hotspot locations were performed in a walk test mode, which means that the measurement equipment was installed in a backpack carried by measurement engineer.

Measurement equipment collects network data by running voice and data tests and using a scanner to obtain radio network parameters. All three mobile networks were measured at the same time and on the same walk test routes using the same smartphones – Samsung Galaxy S10 for voice and Samsung Galaxy S8 for data. The Samsung Galaxy S10 was equipped with operators’ firmware to support all latest networks features (including VoLTE – Voice over LTE calls). The Samsung Galaxy S8 is a Cat. 16 mobile device, which means that it supports 4G (LTE) data speeds up to 1000 Mbps for data receiving and 50 Mbps for data sending. All smartphones worked in automatic technology selection mode. To reflect the latest technical developments in the mobile networks and to examine the benefits from available capabilities, SIM cards with the most comprehensive mobile tariff plans (tariff plans with the highest data rates, highest number of minutes, largest amount of data volume) available from each operator were used.

Mobile Tariff plans used for testing networks are shown in Table 2:

Table 2. Mobile Tariff plans – HotSpots

The measurement system consisted of the backpack (SwissQual Freerider III) equipped with identical measurement terminals capable of measuring all network technologies to a very high accuracy. In order to perform voice tests, the Samsung Galaxy S10 smartphones permanently called each other, within the same mobile network. Voice tests were executed in mobile to mobile scenario between terminals in the same backpack. The Samsung Galaxy S8 was used to perform data tests. The same Voice and Data scenarios, as in the drive test, were used for hotspot measurements.

A scanner (Rhode&Schwarz TSME) was used to test radio parameters of the mobile networks. SwissQual NQDI software was used for network data post-processing and reporting. The post-processing activity was supported by the use of Systemics-PAB proprietary Data Warehouse for a customized analysis.

Test equipment used in Hotspot measurements is presented in Figure 3.

Fig. 3. Test Equipment used in Hotspot measurements

The Hotspot measurements do not influence the scoring results. As the scope of this campaign was limited, it does not provide the number of voice calls and data service tests to support scoring principals.