A speech-to-text (STT) system is as its name implies: A way of transforming the spoken words via sound into textual files that can be used later for any purpose.
Speech recognition technology is extremely useful. It can be used for a lot of applications such as the automation of transcription, writing books/texts using your own sound only, enabling complicated analyses on information using the generated textual files and a lot of other things.
In the past, the speech-to-text technology was dominated by proprietary software and libraries. Open source speech recognition alternatives didn’t exist or existed with extreme limitations and no community around, just like open source ERPs.
This is changing, today there are a lot of open source speech-to-text tools and libraries that you can use right now.
What is a Speech Recognition Library/System?
They are the software engines responsible for transmitting voice into the actual texts. They are not meant to be used by end users, as developers will first have to adapt these libraries and use them in order to create a program that end users may use later.
Some of them come with a preloaded and trained dataset to recognize the given voices in one language and generate the corresponding texts, while others give just the engine without the dataset and developers will have to build the training models by them selves (Machine learning).
You can think of them as the underlying engines of speech recognition programs.
If you are an ordinary user looking for speech recognition, then none of these will be suitable for you, as they are meant for programmers use only.
What is an Open Source Speech Recognition Library?
The difference between proprietary speech recognition and open source speech recognition, is that the library used to process the voices should be licensed under one of the known open source licenses, such as GPL, MIT and others.
Microsoft and IBM for example have their own speech recognition toolkits that they offer for developers, but they are not open source. Simply because they are not licensed under one of the open source licenses in the market.
What are the Benefits of Using Open Source Speech Recognition?
Mainly, you get few or no restrictions at all on the commercial usage for your application, as the open source speech recognition libraries will allow you to use them for whatever use case you may need.
Also, most – if not all – open source speech recognition toolkits in the market are also free of charge, saving you tons of money instead of using the proprietary ones.
The benefits of using open source speech recognition toolkits are indeed too many to be summarized in one article.
Top Open Source Speech Recognition Systems
In our article we’ll see a couple of them, what are their pros and cons and when they should be used.
1. Project DeepSpeech
This project is made by Mozilla, the organization behind the Firefox browser.
It’s a 100% free and open source speech-to-text library that also implies the machine learning technology using TensorFlow framework to fulfill its mission. In other words, you can use it to build training models by yourself to enhance the underlying speech-to-text technology and get better results, or even to bring it to other languages if you want.
You can also easily integrate it to your other machine learning projects that you are having on TensorFlow. Sadly it sounds like the project is currently only supporting English by default. It’s also available in many languages such as Python (3.6).
However, after the recent Mozilla restructure, the future of the project is unknown, as it may be shut down (or not) depending on what they are going to decide.
You may visit its Project DeepSpeech homepage to learn more.
Kaldi is an open source speech recognition software written in C++, and is released under the Apache public license.
It works on Windows, macOS and Linux. Its development started back in 2009. Kaldi’s main features over some other speech recognition software is that it’s extendable and modular: The community is providing tons of 3rd-party modules that you can use for your tasks.
Kaldi also supports deep neural networks, and offers an excellent documentation on its website. While the code is mainly written in C++, it’s “wrapped” by Bash and Python scripts.
So if you are looking just for the basic usage of converting speech to text, then you’ll find it easy to accomplish that via either Python or Bash. You may also wish to check Kaldi Active Grammar, which is a Python pre-built engine with English trained models already ready for usage.
Learn more about Kaldi speech recognition from its official website.
Probably one of the oldest speech recognition software ever, as its development started in 1991 at the University of Kyoto, and then its ownership was transferred to as an independent project in 2005. A lot of open source applications use it as their engine (Think of KDE Simon).
Julius main features include its ability to perform real-time STT processes, low memory usage (Less than 64MB for 20000 words), ability to produce N-best/Word-graph output, ability to work as a server unit and a lot more.
This software was mainly built for academic and research purposes. It is written in C, and works on Linux, Windows, macOS and even Android (on smartphones). Currently it supports both English and Japanese languages only.
The software is probably available to install easily using your Linux distribution’s repository; Just search for
julius package in your package manager.
You can access Julius source code from GitHub.
If you are looking for something modern, then this one is for you.
Wav2Letter++ is an open source speech recognition software that was released by Facebook’s AI Research Team just 2 months ago. The code is released under the BSD license. Facebook is describing its library as “the fastest state-of-the-art speech recognition system available”.
The concepts on which this tool is built makes it optimized for performance by default; Facebook’s also-new machine learning library FlashLight is used as the underlying core of Wav2Letter++. Wav2Letter++ needs you first to build a training model for the language you desire by yourself in order to train the algorithms on it.
No pre-built support of any language (including English) is available. It’s just a machine-learning-driven tool to convert speech to text.
It was written in C++, hence the name (Wav2Letter++).
You can learn more about Wav2Letter++ from the following link.
Researchers at the Chinese giant Baidu are also working on their own speech-to-text engine, called DeepSpeech2.
It’s an end-to-end open source engine that uses the “PaddlePaddle” deep learning framework for converting both English & Mandarin Chinese languages speeches into text. The code is released under BSD license.
The engine can be trained on any model and for any language you desire. The models are not released with the code. You’ll have to build them yourself, just like the other software.
DeepSpeech2‘s source code is written in Python, so it should be easy for you to get familiar with it if that’s the language you use.
Developed by NVIDIA for sequence-to-sequence models training.
While it can be used for way more than just speech recognition, it is a good engine nonetheless for this use case. You can either build your own training models using it, or use Jasper, Wave2Letter+ and DeepSpeech2 models which are shipped by default. It supports parallel processing using multiple GPUs/Multiple CPUs, besides a heavy support for some NVIDIA technologies like CUDA and its strong graphics cards.
Another sequence-to-sequence toolkit. Developed by Facebook and written in Python and the PyTorch framework. Also supports parallel training. Can be even used for translation and more complicated language processing tasks.
Learn more about Fairseq from Facebook.
One of the newest open source speech recognition systems, as its development just started in 2020.
Unlike other systems in this list, Vosk is quite ready to use after installation, as it supports 10 languages (English, German, French, Turkish…) with portable 50MB-sized models already available for users (There are other larger models up to 1.4GB if you need).
Learn more about Vosk from its official website.
An end-to-end speech recognition engine which implements ASR (Automatic speech recognition). Written in Python and licensed under the Apache 2.0 license. Supports unsupervised pre-training and multi-GPUs processing. Built on the top of TensorFlow.
Visit Athena source code.
Written in Python on the top of PyTorch.
Also supports end-to-end ASR. It follows Kaldi style for data processing, so it would be easier to migrate from it to ESPnet. The main marketing point for ESPnet is the state-of-art performance it gives in many benchmarks, and its support for other language processing tasks such as text-to-speech (STT), machine translation (MT) and speech translation (ST).
Licensed under the Apache 2.0 license.
You can access ESPnet from the following link.
What is the Best Open Source Speech Recognition System?
If you are building a small application which you want to be portable everywhere, then Vosk is your best option, as it is written in Python and works on iOS, android and Raspberry pi too, and supports up to 10 languages. It also provides a huge training dataset if you shall need it, and a smaller one for portable applications.
If, however, you want to train and build your own models for much complex tasks, then any of Fairseq, OpenSeq2Seq, Athena and ESPnet should be more than enough for your needs, and they are the most modern state-of-the-art toolkits.
As for Mozilla’s DeepSpeech, it lacks a lot of features behind its other competitors in this list, and isn’t really cited a lot in speech recognition academic research like the others. And its future is concerning after the recent Mozilla restructure, so one would want to stay away from it for now.
Traditionally, Julius and Kaldi are also very much cited in the academic literature.
Alternatively, you may try these open source speech recognition libraries to see how they work for you in your use case.
The speech recognition category is starting to become mainly driven by open source technologies, a situation which seemed to be very far-fetched few years ago.
The current open source speech recognition software are very modern and bleeding-edge, and one can use them to fulfill any purpose instead of depending on Microsoft’s or IBM’s toolkits.
If you have any other recommendations for this list, or comments in general, we’d love to hear them below!