Frequently Asked Questions (FAQ)

The following is a list of frequently asked questions related to the Dragonfly speech recognition framework.

General Questions

What is Dragonfly?

Dragonfly is a speech recognition framework for Python that makes it convenient to create custom commands to use with speech recognition software. It was written to make it very easy for Python macros, scripts, and applications to interface with speech recognition engines. Its design allows speech commands and grammar objects to be treated as first-class Python objects.

Dragonfly can be used for general programming by voice. It is flexible enough to allow programming in any language, not just Python. It can also be used for speech-enabling applications, automating computer activities and dictating prose.

Which speech recognition software and operating systems are supported?

Dragonfly supports the following speech recognition (SR) engines:

  • Dragon, a product of Nuance. All versions up to 15 (the latest) are supported. Home, Professional Individual and previous similar editions of Dragon are supported. Other editions may work too
  • Windows Speech Recognition (WSR), included with Microsoft Windows Vista, Windows 7+, and freely available for Windows XP
  • Kaldi
  • CMU Pocket Sphinx

Dragonfly has cross platform support for Windows, macOS and Linux (using X11). The following table shows which engines are available on which platforms:

Operating system Available SR engines
Windows DNS, WSR, Kaldi, Sphinx
Linux Kaldi, Sphinx
macOS Kaldi, Sphinx

Windows-only speech recognition software, i.e. DNS and WSR, can be used to control Linux or macOS machines via Aenea, a client-server library for using Dragonfly voice macros on remote hosts.

Dragonfly’s X11 support should work just fine on non-Linux unices, such as FreeBSD. If you are planning to use the Kaldi SR engine backend on a platform like FreeBSD, you will need to compile the Kaldi engine dependencies manually.

Where can I find examples Dragonfly command modules?

There is a list of repositories and other projects containing Dragonfly command modules under the Related resources -> Command modules section of the documentation. There are also example command modules in dragonfly/examples.

What is the difference between dragonfly and dragonfly2?

Dragonfly is the original project written by Christo Butcher (t4ngo). It is no longer actively maintained. Dragonfly2 is a fork of dragonfly that uses a different distribution name in order to upload releases to the Python Package Index, so that the package can be installed by running:

pip install dragonfly2

It is important to note that the import name is still “dragonfly”:

from dragonfly import Grammar, MappingRule, Key, Text, Mouse, Dictation

Dragonfly2 is intended to be backwards-compatible continuation of the original project. Many bugs and other issues are fixed in this version. It supports using additional speech recognition engine backends (e.g. the Kaldi engine). It also works with Python 3 and has cross-platform support for Windows, GNU/Linux and macOS. Dragonfly2 also has many other new features not found in the old version.

See the changelog for the full list of changes between the two versions.

How can I use older Dragonfly scripts with Dragonfly2?

Older dragonfly scripts are mostly written with Python 2.x in mind. Python version 2.7 has reached the end of its life as of January 2020 (see Python 2.7 EOL). For complicated reasons, Dragonfly’s Python 3.x support has come a bit later than most other active projects. You will need to convert older Python 2.x code, to use it with Python 3.x. There are a few ways to convert older code:

  • 2to3 - command-line program that reads Python 2.x source code and applies a series of fixers to transform it into valid Python 3.x code.
  • python-modernize - a command-line program that uses 2to3 to make Python 2 code compatible with Python 3.

You may be interested in the Python 2-3 code porting guide if you prefer to do things manually.

A number of older dragonfly command modules also include the following code:

try:
    import pkg_resources
    pkg_resources.require("dragonfly >= 0.6.5")
except ImportError:
    pass

Since the distribution name has been changed to dragonfly2, you will need to either replace dragonfly with dragonfly2 or remove code like this altogether.

Where are some good resources on learning Python?

If you just want to use Dragonfly for flexible computer control or for programming in other languages and you don’t have much background in Python, then the following resources from the Python Software Foundation might be useful to you:

API Questions

How do I use an “extra” in a Dragonfly spec multiple times?

Sometimes it is desirable to use the same “extra” multiple times in a Dragonfly Compound, CompoundRule or MappingRule specification (or “spec”). You cannot use the same reference name in the same spec. However, there is always an efficient solution available using multiple names. Solutions to two common problems are listed below using the generic compound spec "<X1> and <X2>".

from dragonfly import IntegerRef, Choice, RuleRef, RuleWrap

# For saying and processing two numbers, e.g. "one and ten".
int_extras = [
    IntegerRef("X1", 1, 20),
    IntegerRef("X2", 1, 20)
]

# For saying and processing a Choice element two times,
# e.g. "alpha and bravo".
my_choice = Choice("", {
    "alpha": "a",
    "bravo": "b",
    "charlie": "c"
})
# Use RuleWrap to wrap the Choice element into a private rule only
# pronounceable via references (i.e. with RuleRef elements).
# This is more efficient than using two identical Choice elements.
my_choice_rule = RuleWrap("", my_choice).rule
alpha_extras = [
    RuleRef(my_choice_rule, "X1"),
    RuleRef(my_choice_rule, "X2")
]

All of these example extras lists and their elements can be used with Compound or Choice elements or CompoundRule or MappingRule grammar rules.

Is there a way to re-use a function with different “extra” names?

Dragonfly’s Function action class is normally used to call a Python function when a spoken command is recognized. Function actions pass recognized “extra” values via key word arguments, rather than positional arguments.

Below are two methods to re-use a Python function without redefining it:

from dragonfly import Function

# Define a function to be used by two Function actions.
def add_and_print(x, y):
    print("%d" % (x + y))

# --- Method one ---
# Use a lambda function.
Function(lambda x, z: add_and_print(x, z))

# --- Method two ---
# Use the optional 'remap_data' argument to pass the 'z' argument
# as 'y' internally.
Function(add_and_print, dict(z='y'))

See the Function action’s documentation for more information and code examples.

Is there a way to recognize negative integers with Dragonfly?

Yes. The simplest way of recognizing negative integers is to use IntegerRef and Modifier elements together in a command with an appropriate prefix word such as “negative” or “minus”:

from dragonfly import IntegerRef, Modifier, Text

# Define a MappingRule command for typing a negative integer.
mapping = {
    "(minus|negative) <n>": Text("%(n)d"),
}

# The special Modifier element lets us modify the value of an element.
# Here we use it to specify the "n" extra as a negated integer between 1
# and 50.
extras = [
    Modifier(IntegerRef("n", 1, 50), lambda n: n*-1)
]

Is there a way to construct Dragonfly grammars manually with elements?

Yes. The dragonfly.grammar.rule_basic.BasicRule is the rule class to use for constructing Dragonfly rules and grammars manually with elements instead of with compound specs and extras.

The following is an example of how to use BasicRule and common Dragonfly element and action classes together:

from dragonfly import (BasicRule, Repetition, Alternative, Literal, Text,
                       Grammar)

class ExampleRule(BasicRule):
    # Define a rule element that accepts 1 to 5 (exclusive) repetitions
    # of either 'test one', 'test two' or 'test three'. These commands
    # type their respective numbers in succession using the Text action.
    element = Repetition(
        Alternative((
            Literal("test one", value=Text("1")),
            Literal("test two", value=Text("2")),
            Literal("test three", value=Text("3")),
        )),
        1, 5
    )

# Create a grammar with the example rule and load it.
rule = ExampleRule()
grammar = Grammar("BasicRule Example")
grammar.add_rule(rule)
grammar.load()

Please note that extras in action specification strings (e.g. n in Key("left:%(n)d")) will not work for the BasicRule class. For this functionality, you should use CompoundRule or MappingRule instead. You can also override the _process_recognition() method and use the node object to retrieve the desired extra / element and its value.

Does Dragonfly support using Windows Speech Recognition with the GUI?

Yes. To use WSR with the GUI, you need to initialize the SAPI5 shared process engine in the module loader script file:

from dragonfly import get_engine
get_engine("sapi5shared")

If you are using Dragonfly’s command-line interface, then you need to pass “sapi5shared” as the engine name:

python -m dragonfly load -e sapi5shared _\*.py

There are significant issues with using WSR’s shared recognizer for command-based speech recognition. This is because of the built-in commands and dictation output. Dragonfly defaults to the in-process SAPI5 engine because it doesn’t have these defaults.

Is there an easy way to check which speech recognition engine is in use?

Yes. The current engine can be checked using the dragonfly.engines.get_current_engine() function. The following code prints the name of the current engine if one has been initialized:

from dragonfly import get_current_engine
engine = get_current_engine()
if engine:
    print("Engine name: %r" % engine.name)
else:
    print("No engine has been initialized.")

Troubleshooting Questions

Why are my command modules are not being loaded/detected?

If you have placed Python files into the MacroSystem / user directory (using DNS/Natlink) or the directory where your module loader script is (using another engine) and there is no indication that the files were loaded, then there can be a few reasons why:

  1. Your Python files don’t start with an underscore _ and end with .py.
  2. You’ve put the files in the wrong directory. If you’re using Natlink, then try running the Natlink configuration- program to double check where Natlink loads files from.

In the case that your command modules are being loaded and you’re getting error messages not mentioned in the FAQ, then see the Unanswered Questions section.

How do I fix “No handlers could be found for logger X” error messages?

This error is specific to Python 2.x. It isn’t a Dragonfly error, but as many users still use Python 2.7, it is listed here. This is the most common example of the error:

No handlers could be found for logger "action"

There are two easy methods for to solving this problem:

# --- Method one ---
# Set up a basic logging handler for console output using the 'logging'
# module.
import logging
logging.basicConfig()

# --- Method two ---
# Set up Dragonfly's logging handler from the 'dragonfly.log' module.
# This sets up a logging handler for console output, appends log messages
# to a log file (~/.dragonfly.log) and sets sane defaults for Dragonfly's
# internal loggers.
from dragonfly.log import setup_log
setup_log()

For either method, add the two lines of code near the top of one of your command modules or command module loader script, if you use one.

Cannot load compatibility module support error when starting Dragon

This is a known issue with Natlink. Please see this Natlink troubleshooting page for solutions on how to solve this and other issues that occur before the Natlink messages window appears.

How do I fix “failed to decode recognition” errors?

“Failed to decode recognition” is the error message displayed when Dragonfly is unable to match what was said to a grammar rule. This can occur when saying command words to match a dictation part of a rule.

One way around this to add a top-level grammar rule for dictating other words in your rules:

from dragonfly import Dictation, Text

mapping = {
    "reserved (word|words) <text>": Text("%(text)s")
}

extras = [
    Dictation("text")
]

Another way around the problem is to have an “extra” for reserved words:

from dragonfly import Choice, Text

mapping = {
    "type <reserved>": Text("%(reserved)s")
}

extras = [
    Choice("reserved", {
        "alpha": "alpha",
        "bravo": "bravo",
        "charlie": "charlie",
    })
]

How can I increase the speech recognition accuracy?

Low recognition accuracy is usually caused by either bad-quality audio input or a speech model that isn’t trained to your voice or use case. You might try the following:

  • Re-positioning your microphone.
  • Using a different microphone.
  • Training words or phrases.
  • Change the speech recognition engine settings (e.g. adjust Dragon’s accuracy/speed slider).
  • Using a different engine back-end if possible, e.g. the Kaldi back-end is typically more accurate than CMU Pocket Sphinx and WSR back-ends.

Dragonfly also has programmatic methods for increasing recognition accuracy. They can be used to fine tune accuracy for specific commands or parts of commands:

  1. Kaldi Grammar/Rule/Element Weights (Kaldi-only)
  2. Quoted words in dragonfly.grammar.elements_basic.Literal elements (only applies to Dragon)

Why isn’t Dragonfly code aware of DPI scaling settings on Windows?

There can be problems with Dragonfly’s monitor-related functionality on Windows Vista and above if the system is set up to use one or more monitors with a high number of dots per inch (DPI). For this reason, Dragonfly attempts to set the DPI awareness for the process when it is imported. The SetProcessDpiAwareness() function is used to do this on Windows 8.1 and above.

If you need to set the DPI awareness manually using a different DPI awareness value, do so before importing dragonfly. The following is equivalent to what dragonfly does internally:

import ctypes
ctypes.windll.shcore.SetProcessDpiAwareness(2)  # PROCESS_PER_MONITOR_DPI_AWARE

The SetProcessDpiAware() function can be used instead on older Windows versions (e.g. Vista and 7). The SetProcessDpiAwarenessContext() function can be used on Windows 10 (version 1703) and above. For more information on this topic, please see the following Microsoft documentation pages:

Why aren’t Dragonfly actions working with Windows admin applications?

Since Windows Vista, Windows has built-in security features to isolate processes with different integrity levels. This is done to prevent user-level processes from performing tasks that require administrative authorization via User Account Control. To this end, Windows also prevents user-level processes from sending keystrokes, mouse events or otherwise controlling processes or windows running as the administrator. Since our Python code normally runs as the user, Windows prevents us from interacting with administrative windows.

Windows has a way for accessibility software to run with special privileges. This involves building a signed executable with a special manifest uiAccess="true" attribute set and installing it under either the Program Files or Windows/System32 directories. For more on this, see Microsoft’s Security Considerations for Assistive Technologies documentation page.

Unfortunately, this is not easily achievable with Python programs. Developing a secure UI Access solution for Dragonfly would be quite complicated and, given that it is a small project with only a few developers, present significant security risks. For these reasons, Dragonfly will not be implementing UI Automation support. The following are a few alternative solutions:

  1. Use Dragon’s UI Automation capability

    Dragon runs a UI Automation service in the background and, clearly, uses it to allow users to interact with administrative applications. This may be good enough if you don’t need to perform complex tasks.

  2. Run Python as the administrator

    Running the Python process that loads, recognizes and processes your commands as the administrator should work around the limitations. Be careful if you do this; Windows won’t go as far to stop bugs in your code from doing damage!

  3. Use AutoHotkey’s Run with UI Access feature

    AutoHotkey (AHK) is an automation scripting language for Windows. One of its many features allows running AHK code with UI Access. See the relevant AutoHotkey FAQ on UAC for how to set this up and use it.

    Using this, you can define hotkeys for interacting with administrative windows. This can help if you are able to use a keyboard, even if only for a short time. Unfortunately, Dragonfly’s actions won’t be able to trigger these hotkeys since the AHK code will be running in elevated mode. One way around this is to run a local web server in your AHK script, perhaps using AHKhttp or something similar. Then you can send HTTP requests to the server to run your AHK code.

    If you do use the local web server approach mentioned, it is very important to ensure that requests received by the server are properly authorized so that only your Python code has (indirect) UI access (see Web API security).

Why aren’t Dragonfly’s input actions working on my Linux system?

Dragonfly’s Key, Text and Mouse action classes use the xdotool program on Linux. These actions will not work if it isn’t installed. It can normally be installed through your system’s package manager. On Debian-based or Ubuntu-based systems, this is done by running the following console command:

sudo apt install xdotool

The Window class also requires the wmctrl program:

sudo apt install wmctrl

The keyboard/mouse input classes will only work in an X11 session. You will get the following error if you are using Wayland or something else:

NotImplementedError: Keyboard support is not implemented for this platform!

If you see this message, then you will probably need to switch to X11 or use something like ydotool to have keyboard/mouse input work properly.

If you are using X11 and still see this message, then it means that the DISPLAY environment variable checked by Dragonfly internally is not set.

Unanswered Questions

If your question isn’t listed above, then there are a few ways to get in touch: