‘Neural Networks Can Provide Assessments As Accurate As Humans’
Voice assistants have become part of everyday life. They can plan routes, play music and films, and answer questions. But the quality of their speech requires assessment. To address this, students of the Applied Artificial Intelligence Workshop at the HSE University and VK Engineering and Mathematics Schoolhave developed neural networks capable of evaluating speech synthesis.
Nikita Shevtsov
‘Text-to-speech technologies convert text into speech. They are the voices we hear in satnavs andaudiobooks. Previously, in order to assess how well synthesis worked, manual checking was required: people listened to recordings and gave ratings. We decided to automate this process. The task for students was to create a system that would replace subjective evaluation with a neural network model. Manual labelling is not only costly and time-consuming, but also requires the involvement of numerousspecialists,’ said Nikita Shevtsov, head of the Applied Artificial Intelligence Workshop project at the HSE and VK Engineering and Mathematics School.
The students of the workshop used open English-language SOMOS datasets containing 20,100 audio clips, 200 TTS systems, 987 assessors, and more than 350,000 ratings.
On this basis, two metrics were prepared and implemented: MOS (Mean Opinion Score)—a rating of a single audio file from 1 to 5; SBS (Side-by-Side)—a comparison of two audio clips and subsequentselection of the better one. Five models were also developed: for MOS—MOSNet, MOSNetBert, and WhisperBert; for SBS—NeuralSBS and NeuralSBSBert. The models work both in audio and multimodal formats.
‘We found that neural networks can provide assessments just as accurate as those of humans. The MOS models showed an RMSE value of ≈ 0.4, close to the human error rate of 0.62. The NeuralSBS model identifies the better audio in 73% of cases—comparable to the average subjective listener. This paves the way for faster and cheaper evaluation of TTS quality,’ noted Nikita Shevtsov.
The next step is retraining the models in Russian. The team is also considering integrating the evaluation models into CI/CD pipelines in order to implement automated quality checks in speech generation production processes.
‘Automatic evaluation of speech synthesis is a step towards more reliable and scalable TTS systems. We plan to make it accessible for wider development,’ added Nikita Shevtsov.
See also:
HSE Researchers Assess Creative Industry Losses from Use of GenAI
Speaking at the IPQuorum.Music forum on October 15, Leonid Gokhberg, HSE First Vice Rector, and Daniil Kudrin, an expert at the Centre for Industry and Corporate Projects of HSE ISSEK, presented the findings of the first study in Russia on the economic impact of GenAI on creative professions. The analysis shows that creators’ potential losses could reach one trillion roubles by 2030.
‘Fall into ML Has Firmly Established Itself as a Landmark Event in Russia’s AI Scene’
On October 24–25, 2025, the AI and Digital Science Institute of the HSE Faculty of Computer Science will host the fourth annual Fall into ML 2025 conference at the HSE Cultural Centre. The event is once again supported by its general partner, Sber. The focus this year is on breakthrough research and the future of fundamental AI.
Critique of Obscure Reason: Artificial Intelligence in the Perception of Mathematicians
Mathematicians at HSE University believe that there is no need to fear losing jobs because of the widespread use of AI, while at the same time they warn against uncritical acceptance of works and projects prepared with its help. AI, however, can be a useful tool in research, creating models and processing large volumes of information.
Registration for Russian Olympiad in Artificial Intelligence 2025 Now Open
Registration for the fifth season of the Russian Olympiad in Artificial Intelligence has opened. This year, the competition has gained international status. The event is open to students in the 8–11 grades both in Russia and abroad. The winners will receive benefits when applying to Russian universities.
Global AI Trends Discussed at International Foresight Workshop at HSE University
At an international foresight workshop on artificial intelligence held at HSE University, Russian and foreign scholars discussed the trends and challenges arising from the rapid development of AI.
HSE Students Win International Olympiad in Artificial Intelligence
In the finals of the olympiad, the Russian team competed with 300 talented schoolchildren from 61 countries, including Australia, Brazil, Hungary, China, Mexico, the United Arab Emirates, Poland, Serbia, Singapore, the USA, Sweden, and Japan. The finals included team and individual rounds. In the team round, the Russian team made it into the top 10, winning a silver medal. In the individual competition, Russian schoolchildren won six gold medals, one silver, and one bronze.
HSE to Entrust Routine CPD Programme Development to AI
HSE University, together with the EdTech company CDO Global, is launching AI-based constructors to streamline the design of continuing professional development (CPD) courses. The new service will automate the preparation of teaching materials and assessment tools, significantly reducing the time and resources required of lecturers and instructional designers.
HSE University Becomes Absolute Leader in AI Alliance Ranking of Universities
TheAI Alliance Russia has announced a new ranking of Russian universities based on the quality of education in the field of AI. A total of 203 universities from 68 Russian regions participated in the ranking. HSE University was the first to join the highest A++ group.
HSE University and MTS Join Forces to Combat Deepfakes and Train AI to Create Bespoke Video Content
HSE University and MTS Web Services (MWS) have announced the launch of a series of joint research initiatives in the field of artificial intelligence technologies. These efforts aim to develop innovative solutions in cybersecurity, multimodal content generation, and big data analysis. The project’s leading institution is the HSE Tikhonov Moscow Institute of Electronics and Mathematics (HSE MIEM), under the overall coordination of the HSE AI Research Centre.
HSE University Brings Together Researchers at International AI Summer Institute in Shanghai
In early July 2025, the International Summer Institute on Artificial Intelligence in Education took place in Shanghai. It was organised by the HSE Institute of Education in cooperation with East China Normal University (ECNU). More than 50 early-career researchers and keynote speakers from nine countries—ranging from Russia and China to Canada and Singapore—gathered to share the latest findings from their work and to forge new international partnerships.