Trend analysis Please note that in order to distinguish two campaigns’ results compared, the following shade code was used. The lightest bar colour was used to represent 2017 results, mid for 2018 results and darkest for 2019 results. The example is presented in Figure 1.
Fig. 1. Year to year comparison shades codes
Fig. 2. Final scores comparison
Telekom Srbija scored the best in 2019 due to large improvement across all services types and aggregations. Vip mobile was second with improved voice quality across aggregations. In 2019, Telekom Srbija and Vip mobile achieved the advantage due to introduction of VoLTE (Voice over LTE).
The trend analysis of Voice scorings and Data scorings are presented on Figure 3 and Figure 4. Please note that Final Score is the sum of Voice and Data score.
Fig. 3. Final scores comparison
All operators improved the quality of Voice service over the years. Telekom Srbija and Vip mobile made large improvement due to VoLTE introduction in 2019.
Fig. 4. Final scores comparison
Telekom Srbija has improved Data services quality year to year. Telenor and Vip mobile improved the quality only in 2019.
The trend analysis of Large Cities scorings, Small Cities scorings and Roads scorings are presented on Figure 5, Figure 6 and Figure 7. Please note that Final Score is the sum of Large Cities score, Small Cities score and Roads score.
Fig. 5. Large Cities scores comparison
All mobile operators improved quality of service in Large Cities. Telekom Srbija scored the best in this aggregation having the general improvement across most of the KPIs. The biggest improvement has been noticed for Telekom Srbija network with much better Voice quality than in previous campaign in 2018.
Fig. 6. Small Cities scores comparison
The quality of services improved in Small Cities for all mobile operators. Telekom Srbija had the biggest improvement compared to the previous years, with better KPIs across all test types.
Fig. 7. Roads scores comparison
Observing the results of benchmarking of mobile networks on the Roads, Telekom Srbija showed the highest result, which indicates that this operator improves the quality of service year after year. The improvement was seen across most of the KPIs measured. Telenor achieved the worse score on Roads compared to previous campaigns, mainly due to issues with service accessibility.
Voice
Fig. 8. Selected KPIs from voice tests
Call Success Rate: In Large Cities, Telekom Srbija and Vip mobile increased success rate. In Small Cities the results remained on the same level, with only Vip mobile with slight degradation. The results collected in Cities were still on a very good level. On the Roads, all operators decreased their success rates.
Average Call Setup Time: The noticeable difference is visible in Telekom Srbija and Vip mobile, which improved greatly due to VoLTE service introduction. Telenor also improved, compared to the previous years, staying with the same voice call technology (CSFB).
Average MOS score: The perceived voice quality score has improved about 1 point for operators with VoLTE, Telekom Srbija and Vip mobile. Telenor results also slightly improved but within CSFB limitation.
% of MOS score below 2.3: In all aggregations, operators improved noticeably when it comes to the number of samples with bad speech quality. The biggest improvement, by over 6%, was achieved by Telekom Srbija.
FDTT HTTP DL
Fig. 9. Selected KPIs from FDTT HTTP DL tests
Session Success Rate provides information about reliability of Internet access for mobile users. During 2019 benchmarking campaign, all operators provided close to excellent results (>99%) for Large and Small cities. The level of quality of service was maintained from 2018 for Large and Small cities. Only on Roads all operators performed poorer than in previous campaigns.
Average Data Rates measured during 2019 benchmark were improved by all operators. The biggest improvement was noticed for Telenor, with results better by 20-30Mbps than in 2018 campaign. Results achieved during 2019 are satisfactory to provide even those demanding mobile services like media streaming.
Percentage of Tests with Data Rate below 4Mbps is metric which describes portion of the network not satisfying criteria for minimal user satisfaction for Internet connectivity. If data rate is below 4Mbps, users may have troubles with demanding services like downloading video streams, as well as bulk large downloads. Compared to 2018, Telekom Srbija and Telenor improved in all categories, Vip mobile improved in Large cities and Small cities, and deteriorated on Roads.
FDTT HTTP UL
Fig. 10. Selected KPIs from FDTT HTTP UL tests
Session Success Rate provides information about reliability of Internet access for mobile users. Performance of all operators remained stable and on excellent level since 2017. Only on Roads, all operators achieved lower success rate than in previous campaigns.
Average Data Rates measured during 2019 benchmark confirmed improvement in Large Cities and Small Cities for Telekom Srbija (+14Mbps in Large Cities, +5Mbps in Small Cities) and Telenor (+9Mbps in Large Cities, +11Mbps in Small Cities). Vip mobile results improved in Large Cities (+7Mbps), and slightly decreased in Small cities (-3Mbps). On the roads, all operators presented similar results.
Percentage of Tests with Data Rate below 2Mbps is metric which describes portion of the network not satisfying criteria for minimal user satisfaction for Internet connectivity. If data rate is below 2Mbps, users may have troubles with demanding services like uploading photos and video streams to social media platforms, as well as bulk large uploads. All operators improved in Large Cities and Small Cities, reaching target of <2%. Results for Roads category may cause user dissatisfaction and should be improved further
HTTP Transfer DL
Fig. 11. Selected KPIs from HTTP Transfer DL test
Session Success Rate provides information about reliability of Internet access for mobile users. Performance of all operators remained stable and on excellent level, since 2017, in Large and Small Cities. Differences between operators are not likely to be recognized by the users. On Roads, only Telekom Srbija improved success rate compared to the previous years.
Session Time & Average Data Rate are complimentary metrics which should be considered together, as they give indication about network flexibility in delivering instantaneous data rates. During this data test, effectiveness of TCP proxy / acceleration mechanisms are evaluated together with radio network performance. All operators presented improved performance compared to 2018 results. While Telenor typically improved by 0.6s – 0.8s, Telekom Srbija and Vip mobile improved by 0.1s – 0.4s. Vip mobile is leading in Session Time and Average Data Rates. This indicates that Vip mobile network showed higher efficiency of TCP layer platforms as Session Time contains both: data transfer, as well as TCP connection establishment and closure. While downloading multiple small files, users of Vip mobile network would experience the best performance.
Percentage of Tests with Data Rate below 4Mbps is metric which describes portion of the network not satisfying criteria for minimal user satisfaction for Internet connectivity. During 2019 benchmark, Telekom Srbija and Telenor had improvement in all categories. Despite the improvement of Telenor, this operator achieved lower results compared to its competition. Vip mobile improved its performance in the cities.
HTTP Transfer UL
Fig. 12. Selected KPIs from HTTP Transfer UL test
Session Success Rate provides information about reliability of Internet access for mobile users. All operators presented stable and excellent results in Large and Small cities. Roads remained the most challenging area for Uplink data Transfer, as terminals are constrained by transmit power limitations and distance to the base station. In this category, Telekom Srbija was ranking as the best operator, and the only one with improvement to 2018 results. Telenor deteriorated by 3pp, but remained in second place, and Vip mobile ranked as the last one with the results degradation by 6pp. Success Rates on roads confirm that limitations of Internet connectivity remain recognizable to end users and should be improved further.
Session Time & Average Data Rate are complimentary metrics which should be considered together, as they give indication about network flexibility in delivering instantaneous data rates. During this data test effectiveness of TCP proxy / acceleration mechanisms are evaluated together with radio network performance. All operators made improvement for those metrics in Large and Small Cities. Roads stayed as the most demanding areas where only Telekom Srbija and Vip mobile improved both metrics.
Percentage of Tests with Data Rate below 2Mbps is metric which describes portion of the network not satisfying criteria for minimal user satisfaction for Internet access. Overall differences between operators have diminished during 2019 benchmark. Telekom Srbija was leading in Large Cities and on Roads. Telenor was leading in Small Cities. For Roads, above 8% of traffic was not satisfying 2Mbps throughput threshold, which would have an impact on user’s perceived quality of the service (e.g. the small files upload becomes prolonged and possibilities of making video calls would be limited).
Browsing
Fig. 13. Selected KPIs from Browsing tests
Live Page Session Success Rate presents the closest reflection of user perception of web browsing. During 2019 benchmark, all operators showed almost identical performance in Large and Small cities, with differences not recognizable to end user. On the roads, results differentiated, with Telekom Srbija ranking as the best. Telenor and Vip mobile achieved lower success rates than in previous campaigns. While differences on the roads do not seem large, they would be recognizable by end users. Differences between 2018 and 2019 results can be attributed not only to the operators network performance, but also to the changes of live pages content and structure.
Live Page Average Session Time differed between campaigns, due to operators network improvement, as well as live page content and structure – similarly to Session Success Rate. All operators presented very similar performance, with Telenor being slightly better at every aggregation.
Reference Page Success Rate result for this year’s benchmark stayed on the same level for all operators in Large and Small Cities. On Roads, only Telekom Srbija improved, the other operators achieved lower success rates. Differences in performance for Large and Small cities are not likely to be recognized by the end users.
Reference Page Session Time results were deteriorated for all operators. Despite general degradation, Telenor was leading in Cities. On Roads, Telekom Srbija and Telenor achieved similar level of results.
YouTube
Fig. 14. Selected KPIs from YouTube tests
YouTube Session Success Rate improved for all operators, across all aggregations. Differences between operators in those categories are not recognizable by the end users.
YouTube Playout without Interruptions was improved by all operators during 2019 measurements in Large and Small Cities. In these aggregations, all operators achieved results on perfect level and the differences between operators were not recognizable by the end user. On the roads, only Telekom Srbija made the improvement, Telenor stayed at the same level as in 2018, while Vip mobile deteriorated by 1pp.
YouTube VMOS is complex metrics which reflects all aspects of video quality (Resolution, Interruptions, Blurriness, Blockage, etc.). Comparing results between 2018 and 2019, an improvement of 0.5 VMOS points for all operators could be observed. This is caused by different behaviour of YouTube player. YouTube measurements were performed with different versions of YouTube App (v11.49.55 in 2017, v13.10.55 in 2018). During 2019 benchmark, YouTube App v13.48.51 was used. YouTube constantly tests different approaches in adjusting the video resolution to radio conditions. The results achieved by operators in all aggregations are on the same level and differences are unlikely to be recognized by the end user.
Radio Coverage
Compared to the previous benchmark campaign, for GSM technology, ~10% shift from very good to excellent category of the samples could be observed for all operators. Telekom Srbija was still in the lead.
For UMTS technology, Telekom Srbija and Vip mobile presented nearly the same distribution of samples. For Telenor, a ~20% shift from excellent to good category could be noticed. Despite this, the coverage was very good for all the networks, with Telenor still being in the lead.
Compared to the previous year, Telekom Srbija and Telenor had 8% more samples in the excellent category. Vip mobile coverage improved very slightly, with 1% more samples in excellent category.
