KLets Plugins - Developer Guide

KLets provides an easy to use plugin mechanism to allow other applications to provide their own actions in order to make them available through KLets itself.

A plugin must provide the list of actions that it want to handle, providing for each action a list of phrases that the user must say in order to activate them. KLets will then execute the relevant action once an activation phrase belonging to it has been recognized.

Once executed, an action can influence KLets behavior, for example to tell the result of the command to the user, to ask for some sort of data, or other.

KLets works also as a translator between what user is saying and what type of data an action needs to perform its work, handling intrinsic problems and inaccuracies of the voice recognition technology.

To do that, for every data needed by an action, a parameter must be declared inside the action definition itself, specifying to which type that parameter belong, among types supported by KLets. Plugins can also declare their own parameter types, using the well know Providers system of Android.

When an action tells to KLets to ask for a parameter, KLets will then ask for it to the user, and once the user replied, it will perform all the logic needed to translate what the user said to data usable by the action, handling possible ambiguities.

It is also possible to catch any data directly from an activation phrase, in order to allows users to perform action quickly.

KLets interact with plugins through BroadcastReceivers, Intents, Providers and Manifest meta-data.

Each plugin must export one BroadcastReceiver that handle the following Intent:

• INTENT_EXECUTE_ACTION

It must also declare an application meta-data named “voiceControlActions” pointing to an XML containing static actions information.

If needed by the plugin, it can also export a Provider for each custom data type handled by the plugin itself. In this case, the plugin must also declare an application meta-data named “voiceControlTypes” pointing to an XML containing static custom types information.

The comunication protocol between KLets and a plugin is STATELESS (like the http protocol), so, every time a plugin action is executed, KLets will pass to it the complete set of data collected so far from the user, including any additional data requested by previous plugin executions.

The meta-data must be declared inside the AndroidManifest.xml and must point to the xml resource containing actions information:

- AndroidManifest.xml

<manifest>
	<application>
		<meta-data android:name="voiceControlActions" android:resource="@xml/action_definitions"/>
	</application>
</manifest>

The pointed xml file (action_definitions.xml in the sample above) can contain one or more actions, and it is built using the following schema:

- res/xml/action_definitions.xml

<?xml version="1.0" encoding="UTF-8"?>
<!-- pluginSpecVersion attribute must be set to the plugin protocol version. Actually only "1.0" is supported -->
<actions xmlns:vc="http://api.voicecontrolapp.com/actions" vc:pluginSpecVersion="1.0">
	<!-- Each plugin can have one or more actions -->
	<action vc:id="action id" 
		vc:name="action name" 
		vc:icon="(optional) action icon"
		vc:help="action help"
		vc:hint="(optional) action hint"
		vc:commands="action patterns"
		vc:languages="list of supported languages in ISO-639 format and separated by commas">
 
		<!-- Each action can have zero or more parameters -->
		<param vc:id="id of this parameter" 
			vc:type="parameter type" 
			vc:help="information about this parameter"
			vc:sample="example of what can be said for this parameter"
			vc:request="text spoken by KLets when asking for this parameter"
			vc:notValid="text spoken by KLets when it cannot find a value for this parameter with the text said by the user" />
	</action>
</actions>

Every “action” element has these attributes:

When an action has variable parts inside its command templates or it needs additional parameters that could be asked to the user, the plugin must declare them as inner “param” xml elements with these attributes:

A command template consist in a text with zero or more variable parts as defined by “param” elements inside the action definition.

These variable parts are marked by tokens in square brackets containing the parameter id in uppercase (like this [PARAMETER_ONE]). Each token MUST be placed only one time inside a command template and MUST be separated from other tokens AT LEAST by a single character, otherwise KLets will ignore the command template and will not use it to activate the action.

Commands template doesn't require to contains tokens for parameters declared in an action, as they could be asked directly by the plugin at execution time, but to use a token it is required to declare the relative parameter in the action definition, otherwise the token will be handled as simple text of the command template.

Command templates, also, don't have to respect a specific case, because KLets will convert them in lowercase at runtime (except for tokens).

Parameter types provided by KLets out-of-the-box are:

If a plugin needs a custom parameter type not supported by KLets, then it need to implement and export a Provider, and to configure it inside the custom parameter types Meta-Data.

The meta-data must be declared inside the AndroidManifest.xml and must point to the xml resource containing custom types information:

- AndroidManifest.xml

<manifest>
	<application>
		<meta-data android:name="voiceControlTypes" android:resource="@xml/type_definitions"/>
	</application>
</manifest>

The pointed xml file (type_definitions.xml in the sample above) can contains one or more custom types, and it is built using the following schema:

- res/xml/type_definitions.xml

<?xml version="1.0" encoding="UTF-8"?>
<!-- pluginSpecVersion attribute must be set to the plugin protocol version. Actually only "1.0" is supported -->
<dataTypes xmlns:vc="http://api.voicecontrolapp.com/data_types" vc:pluginSpecVersion="1.0">
	<dataType vc:id="data type id" 
	    vc:providerAuthority="provider autority that provides data for this type"
	    vc:valueField="the field containing the value to pass to the action (IT MUST BE A String TYPE FIELD)"
	    vc:textField="the field containing the text to check against what the user said (IT MUST BE A String TYPE FIELD)"/>
</dataTypes>

When KLets will need to get available values for this type, due an action activation phrase, or an explicit request by the plugin, it will query the configured provider autority and will select the value with the text that best match what the user said, setting its value in the map of the extra EXTRA_PARAMETERS when invoking the action.

KLets will fire an intent for INTENT_EXECUTE_ACTION every time an action must be executed, and it must be handled by a BroadcastReceiver.

This intent is fired in the following situations:

• When the action has been successfully identified by one of its templates
• When the action has made a request for a parameter and it has been said by the user
• When the action has made a request for a user choice and it has been performed

The intent contains all extras needed by the action to perform its logic:

• EXTRA_ACTION_ID: The action id for wich KLets is asking the execution, as configured inside the action definition. Useful when a plugin provide more than a single action.
• EXTRA_PARAMETERS: A Bundle with a map with a key/value pair for every parameter handled so far (both parameters parsed from the activation phrase and parameters explicitly asked by the action itself).
• EXTRA_CHOICES: A Bundle containing a map with a key/value pair for every choice performed by the user so far.
• EXTRA_ASYNC_PENDING_INTENT: A pending intent that must be used for asynchronous tasks completion.

Once executed, the action must set a result code from the BroadcastReceiver handling the INTENT_EXECUTE_ACTION Intent, in order to tell to KLets what should be done.

Accepted result codes are:

• ACTION_RESULT_OK: to be used when the action completes its work.
• ACTION_RESULT_ASK_PARAM: to be used when the action needs to ask the user for a parameter.
• ACTION_RESULT_ASK_CHOICE: to be used when the action needs to ask the user to perform a selection among a list of values.
• ACTION_RESULT_ASYNC: to be used when the action needs to perform its logic asynchronously.

The result ACTION_RESULT_OK must be used when the action has completed its work.

Optionally, the BroadcastReceiver can set the result extras to a bundle with the following extras:

• EXTRA_MESSAGE: Optional; Can be set to a message that must be said before KLets quits.
• EXTRA_PENDING_INTENT: Optional; Can be set to a PentingIntent that must be fired just before KLets quits and after that any bluetooth and audio connections have been closed.

Once received this result code, KLets will quits terminating the vocal session.

The result ACTION_RESULT_ASK_PARAM must be used when the action needs to ask the user for a parameter.

When setting this result code, the BroadcastReceiver must also set the result extras to a bundle with the extra EXTRA_PARAMETER_ID set to the id of the parameter to ask.

Once the requested parameter has been said by the user and looked-up by KLets, KLets will fire a new INTENT_EXECUTE_ACTION intent, with the map inside the extra EXTRA_PARAMETERS updated to contains the entry, identified by the parameter id, set to the value said by the user accordy to the parameter type.

The result ACTION_RESULT_ASK_CHOICE must be used when the action needs to ask to the user to select a value from a list of elements.

When setting this result code, the BroadcastReceiver must set the result extras to a bundle with following extras:

• EXTRA_MESSAGE: must be set to a message that must be said to the user.
• EXTRA_CHOICE_LABELS: must be set to an array of String with labels to use for the choice list.
• EXTRA_CHOICE_VALUES: must be set to an array of Parcelable object to use for the choice list. The element selected by the user will be set on the extra EXTRA_CHOICES.
• EXTRA_CHOICE_ID: must be set to an identifier that the broadcast receiver can use to identify the choice performed by the user on the next EXECUTE_ACTION intent fired.

Once the requested choice has been performed by the user, KLets will fire a new EXECUTE_ACTION intent, with the map inside the extra EXTRA_CHOICES updated to contains the entry, identified by the previously set EXTRA_CHOICE_ID, set to the value selected by the user.

The result ACTION_RESULT_ASYNC must be used when the action execution is asynchronous or cannot be completed inside a broadcast receiver, due resources constraints, timing constraints or other reasons.

In this case, once the action completed its asynchronous logic, it must fire the PendingIntent originally passed to the BroadcastReceiver in the extra EXTRA_ASYNC_PENDING_INTENT, using the PendingIntent.html.send(Context, int, Intent) Android API, setting:

• the context parameter to the current plugin context.
• the code parameter to the same value that would be set using the “BroadcastReceiver.setResultCode(int)”, with the exception of the ACTION_RESULT_ASYNC value that isn't supported in this mode. (this parameter is supported starting from KLets 2.4.0, on previous versions, set this parameter to 0 and put the resultCode value in an additional EXTRA_RESULT_CODE extra on the Intent passed to the intent parameter)
• the intent parameter to an intent instance with extras that would be set using the “setResultBundle()” of the the action BroadcastReceiver

Ensure to always execute the PendingIntent in case of error, otherwhise KLets will remain in a “working in progress” state infinitely.

This is a sample plugin that will repeat what the user said.

Lets defining which are the activation texts of this plugin.

We can say that the action will be called “parrot” and that it could be activated using “tell”, “tell to me” or “say” phrases. We will also setup two quick activation phrases, that will allows the user to say the text to repead within the activation phrase itslef.

So lets add them to the string.xml resource file, with other resources needed for the action.

- res/values/strings.xml

<?xml version="1.0" encoding="utf-8"?>
<resources>
	<!-- Parrot action resources -->
	<string name="action_name">Parrot</string>
	<string name="action_hint">tell, tell to me or say [words to repeat]</string>
	<string name="action_help">This action repeats what you say.\n
\n
Examples:\n
\"<b>tell</b> Hi! I\'m an Android phone!\"\n
\"<b>tell to me</b> You are the best!\"\n
\"<b>say</b> I\'m the best voice command app of the world!\"</string>
 
	<string-array name="activation_texts">
		<item>tell</item>
		<item>tell to me</item>
		<item>say</item>
		<item>tell to me [TEXT_TO_REPEAT]</item>
		<item>say [TEXT_TO_REPEAT]</item>
	</string-array>
 
	<string name="action_param_text_help">The text to repeat</string>
	<string name="action_param_text_sample">You are the best!</string>
	<string name="action_param_text_request">Please, tell me what I need to say.</string>
</resources>

Then, we define the actio metadata in an XML file called parrot_action_definition.xml. We will also define the action parameter TEXT_TO_REPEAT, used inside two of the activation phrases, and that will be asked by the plugin when missing.

- res/xml/parrot_action_definition.xml

<?xml version="1.0" encoding="UTF-8"?>
<actions xmlns:vc="http://api.voicecontrolapp.com/actions" vc:pluginSpecVersion="1.0">
	<action vc:id="parrotAction" 
			vc:name="@string/action_name" 
			vc:help="@string/action_help"
			vc:hint="@string/action_hint"
			vc:commands="@array/activation_texts"
			vc:languages="en,it">
		<param vc:id="TEXT_TO_REPEAT" 
				vc:type="TEXT" 
				vc:help="@string/action_param_text_help"
				vc:sample="@string/action_param_text_sample"
				vc:request="@string/action_param_text_request" />
	</action>
</actions>

After, we will write the BroadcastReceiver that will handle the action execution. As you can see, the action will check if there is the text to repeat, and if it is missing, the action will ask for it.

- src/com/example/testandroid/ExecuteActionBroadcastReceiver.java

package com.example.testandroid;
 
import com.voicecontrolapp.klets.api.KletsPluginApi;
 
import android.content.BroadcastReceiver;
import android.content.Context;
import android.content.Intent;
import android.os.Bundle;
 
public class PluginActionReceiver extends BroadcastReceiver {
	@Override
	public void onReceive(Context context, Intent intent) {
		Bundle resultExtras = getResultExtras(true);
 
		Bundle parameters = intent.getBundleExtra(KletsPluginApi.EXTRA_PARAMETERS);
 
		// Lets check if we have something to repeat.
		if (!parameters.containsKey("TEXT_TO_REPEAT")) {
			// The user didn't said the text to repeat. We tell to KLets to ask for it.
			setResultCode(KletsPluginApi.ACTION_RESULT_ASK_PARAM);
 
 
			resultExtras.putString(KletsPluginApi.EXTRA_PARAMETER_ID, "TEXT_TO_REPEAT");
 
			// Quit and wait for the next invocation, that will have the needed parameter.
			return;
		}
 
		String textToRepeat = parameters.getString("TEXT_TO_REPEAT");
 
		// Setting the result Bundle and result code for this BroadcastReceiver execution.
		setResultCode(KletsPluginApi.ACTION_RESULT_OK);
 
		// We just tell to KLets to say back what the user said, setting it to the message in the result bundle.
		resultExtras.putString(KletsPluginApi.EXTRA_MESSAGE, textToRepeat);
	}
}

As latest stepwe need to wire everyting up inside the Application Manifest:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android" package="com.example.testandroid"
	android:versionCode="1" android:versionName="1.0">
 
	<uses-sdk android:minSdkVersion="8" android:targetSdkVersion="16" />
 
	<application android:allowBackup="true" android:icon="@drawable/ic_launcher" android:label="@string/app_name">
 
		<receiver android:name="com.example.testandroid.PluginActionReceiver" android:exported="true">
			<intent-filter>
				<action android:name="com.voicecontrolapp.klets.api.INTENT_EXECUTE_ACTION" />
			</intent-filter>
		</receiver>
 
		<meta-data android:name="voiceControlActions" android:resource="@xml/parrot_action_definition" />
	</application>
 
</manifest>

That's it. Now the action will be usable from KLets, with every feature of standard actions (like activation texts customization, confirmation requests and so on).

For a complete sample plugin project, containing also an example for custom data types, config intents atc, download it from the topic Downloads, JavaDocs and Support.

The current API tries to provide a protection against malicious applications that could try to use KLets plugin system as a “backdoor” to trigger plugin actions: to do that, KLets provides and holds two permissions with a “signature” protection level (in order to deny other application to hold them), PERMISSION_EXECUTE_TASK for plugin execution and PERMISSION_QUERY_TYPE for plugin data access. Plugins could use this permissions to protect their own receivers and providers, in order to allow only KLets to access them.

Now, there is a problem on how permissions provided by applications are handled inside Android: they can't be trusted, because another application, if installed before KLets, can provide the same permissions, in order to hold them and get access to plugins receivers and providers. However, plugins can protect themselves from this sort of permission-hijacking using a simple application signature check: Android allows only one application at time to provide a specific permission, so, plugins can check if the public signature of the application that is providing a KLets permissions match with the KLets public signature. This way, a plugin can be reasonably confident that who is calling it is KLets and not a malicious app.

To check the signature, use the utility method KletsPluginApi.isPermissionAuthentic(...). It can be used to check both PERMISSION_EXECUTE_TASK and PERMISSION_QUERY_TYPE permissions.

1.1.0

• Refactored API classes base package from com.lukasoft.android.voicecontrol to com.voicecontrolapp.klets
• Renamed class com.lukasoft.android.voicecontrol.api.VoiceControlPluginApi to com.voicecontrolapp.klets.KletsPluginApi
• Added two new INTENT usable by plugins to monitor KLets vocal session starts and shutdowns: INTENT_SESSION_START and INTENT_SESSION_STOP

1.0.0

• Initial release

Plugin API JavaDocs can be found here: Api JavaDocs

Plugin API jar can be downloaded here: voicecontrol-plugin-api.jar

Plugin API sources can be downloaded here: voicecontrol-plugin-api-sources.zip

(OUTDATED: refers API version 1.0.0) A sample plugin project can be downloaded here: voicecontrol-plugin-sample-project.zip

Plugin Development support forums can be found here: Plugin Development forums