Author Topic: OLD Generation 1 - Verbal Requests (HAL)  (Read 3609 times)

Offline RichardTSchaefer

  • Moderator
  • Master Member
  • *****
  • Posts: 10090
  • Karma: +762/-142
OLD Generation 1 - Verbal Requests (HAL)
« on: January 22, 2012, 06:05:53 pm »
Folks,
   I have a private android application called Hal that it similar to the Google Search widget (i.e. you press a button and make a request). The voice stream is sent to my windows server where it is parsed and sends commands to my Vera2.

At this time the grammar is hard coded for my devices. In order to release this as a general application to the Vera community it needs to be driven by the specific Vera configuration. I would like to solicit some input from the community. My current strategy is to first handle a limited set of device types, then to increase the scope. In the first step I believe I can hard code the type of actions and have a grammar as described below using a form of extended BNF syntax.

First I need to define some spoken symbols:
Code: [Select]
<RoomName> := Roon Name Options defined in Vera;
<SectionName> := Section Names defined in Vera;
<SceneName> := Scene Names defined in Vera;
<BinaryLightDeviceName> := Device Names defined in Vera of type "urn:schemas-upnp-org:device:BinaryLight:1";
<DimmableLightDeviceName> := Device Names defined in Vera of type "urn:schemas-upnp-org:device:DimmableLight:1";
<LockDeviceName> :=  Device Names defined in Vera of type "urn:schemas-micasaverde-com:device:DoorLock:1";
<SensorDeviceName> := Device Names defined in Vera of type "urn:schemas-micasaverde-com:device:*Sensor:1"
<ThermostateDeviceName> := Device Names defined in Vera of type "urn:schemas-upnp-org:device:HVAC_ZoneThermostat:1";
<SceneName> : Scene Names defined in Vera;
Then a spoken command is of the form:
Code: [Select]
<top-level> :=  <command-prefix> <command> <command-suffix>;

<command-prefix> := Please|Hal|Computer;
<command-suffix> := Please|Thanks|Thank You|Over|OK|Hal|Computer;
<command> := <specific-command> [In|On <RoomName>] [Section <SectionName>];

Where The room clauses and section clauses will be enforced when specified, and required if device or scene names are ambiguous;


<specific-command> := <BinaryLightCommand> | <DimmableLightCommand> | <DoorLockCommand> | <ThermostatCommand> <SceneCommand> <StatusCommand>;

<BinaryLightCommand> := {Turn {On |Off} [the] <BinaryLightDeviceName> } |
        {Turn [the] <BinaryLightDeviceName> {On |Off} };

<DimmableLightCommand> := {Turn <On|Off> [the] <DimmableLightDeviceName>} |
          {Turn [the] <DimmableLightDeviceName> <On|Off> } |
  {Set [the] <DimmableLightDeviceName> <SliderDeviceValue>} | ;

<SliderDeviceValue> := <SliderDeviceValueWithNumber> | High |  Medium |  Low | {Last [value]};

<SliderDeviceValueWithNumber> := {To | At | {Increase [to]}| {Lower [to]}} <Digits> [percent];

<Digits> :=   <Digit> | <Digit> < Digit>;
<Digit> := One | Two | Three | Four | Five | Six | Seven | Eight | Nine | Zero;

<DoorLockCommand> := {Lock | Unlock} [the] <LockDeviceName> [lock];

<StatusCommand>:= {What is [the] <StatusDevices> Status} |
  {What is [the] Status of [the] <StatusDevices>} ;

<ThermostatCommand> := <ThermostatModeCommand> | <ThermostatFanModeCommand> | <ThermostatTemperature>;

<ThermostatModeCommand> := Set the <ThermostatDeviceName> [Mode] to {Off |Heat | Cool | Auto};

<ThermostatFanModeCommand> := Set the <ThermostatDeviceName> FanMode to {On | Auto};

<ThermostatTemperature> := Set the <ThermostatDeviceName> [Temperature] to <Digits>;

<SceneCommand> := [Run | Activate] [the] <SceneName> [Scene];

<StatusDevices> := <BinaryLightDeviceName> | <DimmableLightDeviceName> | <LockDeviceName> | <SensorDeviceName> | <ThermostatDeviceName>;

Assuming a configuration that included the following devices:
Quote
BinaryLightName: {Hallway Light}
DimmableLightName:  {Great Room Light} | {Office Fan}
LockName: {Front Door}
ThermostatName: {Great Room Thermostat}
SceneName: {Secure the House} | {All Lights On}

Spoken examples would look like:
Quote
   Hal turn on the Great Room Light OK
   Please turn of the Great Room Light Hal
   Please Lock the Front Door thanks
   Computer unlock the Front Door lock thanks
   Hal what is the status of the Front Door over
   Please set the Office Fan to medium thanks
   Computer set the Great Room Light to two four percent thanks
   Computer set the Great Room Light lower two five percent thanks
   Computer set the Great Room Light lower two five percent thanks
   Hal set the Great Room Light to last value please
   Hal Set the Great Room Thermostat Mode to auto thanks
Hal Secure The House Please
        Computer Activate All Lights On Thanks

My questions include:
  • Is it OK to assume a set of operations based on a devices device_type ?
  • I will need some help on syntax for other device_types, the above is the limit of devices I own.
  • Any suggestions for a more robust way to get actions and define a spoken grammar for a device ?

A few more comments on my Android App:
  • Has a help option that displays a textual representation of the specific spoken grammar
  • Has the ability to save the last command as a gesture
  • Has a region to issue a gesture command, not just a verbal command
  • Plan to add an option to make a screen widget for the last command.

A few more comments on my Windows Server App:
  • Written in C#
  • Status are returned to the android application as text messages
  • Uses Windows System.Speech Libraries for recognition
  • Has a windows installer
  • Plan to add Speech Plugins ... i.e. Grammar + actions
  • Has a Windows Control Panel for Configuration
  • Acts as a Windows Service

« Last Edit: April 26, 2013, 11:12:28 am by RichardTSchaefer »

Offline garrettwp

  • Beta Testers
  • Master Member
  • *****
  • Posts: 6371
  • Karma: +227/-128
  • Vera 3, Lite, ISY994
Re: Verbal Requests (HAL)
« Reply #1 on: January 23, 2012, 11:58:32 am »
Folks,
   I have a private android application called Hal that it similar to the Google Search widget (i.e. you press a button and make a request). The voice stream is sent to my windows server where it is parsed and sends commands to my Vera2.

At this time the grammar is hard coded for my devices. In order to release this as a general application to the Vera community it needs to be driven by the specific Vera configuration. I would like to solicit some input from the community. My current strategy is to first handle a limited set of device types, then to increase the scope. In the first step I believe I can hard code the type of actions and have a grammar as described below using a form of extended BNF syntax.

First I need to define some spoken symbols:
Code: [Select]
<RoomName> := Roon Name Options defined in Vera;
<SectionName> := Section Names defined in Vera;
<SceneName> := Scene Names defined in Vera;
<BinaryLightDeviceName> := Device Names defined in Vera of type "urn:schemas-upnp-org:device:BinaryLight:1";
<DimmableLightDeviceName> := Device Names defined in Vera of type "urn:schemas-upnp-org:device:DimmableLight:1";
<LockDeviceName> :=  Device Names defined in Vera of type "urn:schemas-micasaverde-com:device:DoorLock:1";
<SensorDeviceName> := Device Names defined in Vera of type "urn:schemas-micasaverde-com:device:*Sensor:1"
<ThermostateDeviceName> := Device Names defined in Vera of type "urn:schemas-upnp-org:device:HVAC_ZoneThermostat:1";
<SceneName> : Scene Names defined in Vera;
Then a spoken command is of the form:
Code: [Select]
<top-level> :=  <command-prefix> <command> <command-suffix>;

<command-prefix> := Please|Hal|Computer;
<command-suffix> := Please|Thanks|Thank You|Over|OK|Hal|Computer;
<command> := <specific-command> [In|On <RoomName>] [Section <SectionName>];

Where The room clauses and section clauses will be enforced when specified, and required if device or scene names are ambiguous;


<specific-command> := <BinaryLightCommand> | <DimmableLightCommand> | <DoorLockCommand> | <ThermostatCommand> <SceneCommand> <StatusCommand>;

<BinaryLightCommand> := {Turn {On |Off} [the] <BinaryLightDeviceName> } |
        {Turn [the] <BinaryLightDeviceName> {On |Off} };

<DimmableLightCommand> := {Turn <On|Off> [the] <DimmableLightDeviceName>} |
          {Turn [the] <DimmableLightDeviceName> <On|Off> } |
  {Set [the] <DimmableLightDeviceName> <SliderDeviceValue>} | ;

<SliderDeviceValue> := <SliderDeviceValueWithNumber> | High |  Medium |  Low | {Last [value]};

<SliderDeviceValueWithNumber> := {To | At | {Increase [to]}| {Lower [to]}} <Digits> [percent];

<Digits> :=   <Digit> | <Digit> < Digit>;
<Digit> := One | Two | Three | Four | Five | Six | Seven | Eight | Nine | Zero;

<DoorLockCommand> := {Lock | Unlock} [the] <LockDeviceName> [lock];

<StatusCommand>:= {What is [the] <StatusDevices> Status} |
  {What is [the] Status of [the] <StatusDevices>} ;

<ThermostatCommand> := <ThermostatModeCommand> | <ThermostatFanModeCommand> | <ThermostatTemperature>;

<ThermostatModeCommand> := Set the <ThermostatDeviceName> [Mode] to {Off |Heat | Cool | Auto};

<ThermostatFanModeCommand> := Set the <ThermostatDeviceName> FanMode to {On | Auto};

<ThermostatTemperature> := Set the <ThermostatDeviceName> [Temperature] to <Digits>;

<SceneCommand> := [Run | Activate] [the] <SceneName> [Scene];

<StatusDevices> := <BinaryLightDeviceName> | <DimmableLightDeviceName> | <LockDeviceName> | <SensorDeviceName> | <ThermostatDeviceName>;

Assuming a configuration that included the following devices:
Quote
BinaryLightName: {Hallway Light}
DimmableLightName:  {Great Room Light} | {Office Fan}
LockName: {Front Door}
ThermostatName: {Great Room Thermostat}
SceneName: {Secure the House} | {All Lights On}

Spoken examples would look like:
Quote
   Hal turn on the Great Room Light OK
   Please turn of the Great Room Light Hal
   Please Lock the Front Door thanks
   Computer unlock the Front Door lock thanks
   Hal what is the status of the Front Door over
   Please set the Office Fan to medium thanks
   Computer set the Great Room Light to two four percent thanks
   Computer set the Great Room Light lower two five percent thanks
   Computer set the Great Room Light lower two five percent thanks
   Hal set the Great Room Light to last value please
   Hal Set the Great Room Thermostat Mode to auto thanks
Hal Secure The House Please
        Computer Activate All Lights On Thanks

My questions include:
  • Is it OK to assume a set of operations based on a devices device_type ?
  • I will need some help on syntax for other device_types, the above is the limit of devices I own.
  • Any suggestions for a more robust way to get actions and define a spoken grammar for a device ?

A few more comments on my Android App:
  • Has a help option that displays a textual representation of the specific spoken grammar
  • Has the ability to save the last command as a gesture
  • Has a region to issue a gesture command, not just a verbal command
  • Plan to add an option to make a screen widget for the last command.

A few more comments on my Windows Server App:
  • Written in C#
  • Status are returned to the android application as text messages
  • Uses Windows System.Speech Libraries for recognition
  • Has a windows installer
  • Plan to add Speech Plugins ... i.e. Grammar + actions
  • Has a Windows Control Panel for Configuration
  • Acts as a Windows Service

Richard,

I am surprised no one has answered you yet as this has caught my attention. Hopefully I can answer some of your questions.

Quote
Is it OK to assume a set of operations based on a devices device_type ?

There is noting wrong with this as every device in Vera is categorized by device type and that will not change. Going by the type of device makes it easy to determine what type of controls it can take.

Quote
I will need some help on syntax for other device_types, the above is the limit of devices I own.

Let me know if I can be of any help on the android side of things.

Quote
Any suggestions for a more robust way to get actions and define a spoken grammar for a device ?

Google is suppose to have an updated version of the speech recognition. Not sure when this will be available. I have seen other apps that use Google's speech recognition as well. Might want to look into that and see how much work that will require. May allow for the removal of having a man in the middle server e.g. the windows machine.

What you are describing sounds exactly what the hacking community is doing with Siri.

- Garrett

Offline RichardTSchaefer

  • Moderator
  • Master Member
  • *****
  • Posts: 10090
  • Karma: +762/-142
Re: Verbal Requests (HAL)
« Reply #2 on: January 23, 2012, 05:13:59 pm »
The Free Form syntax from the Google and Siri servers are nice but come with a higher error rate. With a defined syntax the error rate goes down quite a bit and works in a higher noise environment.

The work I have to do is on the windows side ... generating the syntax and inserting the operations by reading the data from Vera. The android stuff is in pretty good shape ... all though I never pushed anything to the market.

I assume I will get more response once I put it out there for folks to try ...

Offline parkerc

  • Sr. Hero Member
  • ******
  • Posts: 2425
  • Karma: +33/-45
  • Life Moves Pretty Fast....
Re: Verbal Requests (HAL)
« Reply #3 on: June 06, 2012, 06:15:22 pm »
Hi Richard,

'love your work' :)

For what it is worth, I think you've got a pretty good starting point there, like you say more will come out as it is used. One future proof way could be to create a DIY section, where you have list of every possible operation on one side and the user links that to the device they add. (with a default section too, to keep it simple for some)

Not being an Android users, I'm still crossing my fingers for an iOS version one day.