# Robot Arm with Voice Control - Tutorial (part 2)

Here is part 2 of the robotic arm tutorial. Click here for the first part.

### Running julius

If you've managed to perform Part 1 of the tutorial successfully, then in the command line in the 'voxforge/auto' directory, run:

julius -input mic -C julian.jconf

Julius should then wait for microphone input, and if you speak into the microphone, for example 'elbow up' then it should print output similar to the following:

<<< please speak >>>------### read waveform inputStat: capture audio at 48000HzStat: adin_alsa: latency set to 32 msec (chunk = 1536 bytes)Error: adin_alsa: unable to get pcm info from card controlWarning: adin_alsa: skip output of detailed audio device infoSTAT: AD-in thread createdpass1_best: <s> ELBOW UP </s>pass1_best_wordseq: 0 2 7 1pass1_best_phonemeseq: sil | eh l b ow | ah p | silpass1_best_score: -9641.213867### Recognition: 2nd pass (RL heuristic best-first)STAT: 00 _default: 13 generated, 13 pushed, 5 nodes popped in 388sentence1: <s> ELBOW UP </s>wseq1: 0 2 7 1phseq1: sil | eh l b ow | ah p | silcmscore1: 1.000 1.000 1.000 1.000score1: -9581.207031<<< please speak >>>

Note the decoder output

sentence1: <s> ELBOW UP </s>

As well as the confidence scores:

cmscore1: 1.000 1.000 1.000 1.000score1: -9581.207031

Where cmscore1 indicates the confidences for the sentence, in this case, 'silence word word silence'. score1 shows the Viterbi score as calculated by Julius. All this is printed to the tty's stdout, and all we need to do is to filter out these lines and use them to control the robotic arm.

### The program

I chose Python for the ease of programming as well as its wide range of modules. Since I'm not too worried about speed or memory usage, I managed to build the robotic arm and program it within a week despite my right hand being unable to type (yes, python's that great!). I assume by doing this tutorial you already have basic knowledge of Python. At the beginning of the script we import the required modules:

#!/usr/bin/python
import pexpect
...

Pexpect is a module for creating a pseudo-tty that we can connect with the Julius LVCSR decoder to obtain its output. Once we obtain the output, we then 1. filter out the lines we need (mainly sentence1 and cmscore1, but I've also found the Viterbi score can be useful in some cases), and 2. filter out low-confidence results, and finally 3. send the commands via USB to the robotic arm.

### Obtaining the output

There are many ways to obtain the output - I initially tried the subprocess module, but settled on the pexpect module as it didn't have the pipe buffering issues that subprocess encountered. With pexpect, spawning julius and getting its output in blocks was very simple:

child = pexpect.spawn ('julius -input mic -C julian.jconf')

while True:
try:
process_julius(child.before)
except KeyboardInterrupt:
child.close(force=True)
break

As before, there are many ways to filter out the desired output and confidence scores. First, I need to determine if an output was generated at all:

def process_julius(out_text):
match_res = re.match(r'(.*)sentence1(\.*)', out_text, re.S)
if match_res:
get_confidence(out_text)
else:
pass

Here we use the python regular expressions (re) module, so put at the beginning of the python script:

import re

After making sure we have a sentence, we can then process the text block and extract sentence1, cmscore1 and score1:

def get_confidence(out_text):
linearray = out_text.split("\n")
for line in linearray:
if line.find('sentence1') != -1:
sentence1 = line
elif line.find('cmscore1') != -1:
cmscore1 = line
elif line.find('score1') != -1:
score1 = line
cmscore_array = cmscore1.split()
#process sentence
err_flag = False
for score in cmscore_array:
try:
ns = float(score)
except ValueError:
continue
if (ns < 0.999):
err_flag = True
print "confidence error:", ns, ":", sentence1
score1_val = float(score1.split()[1])
if score1_val < -13000:
err_flag = True
print "score1 error:", score1_val, sentence1
if (not err_flag):
print sentence1
print score1
#process sentence
process_sentence(sentence1)
else:
pass

In this function, I also set the criterion of 0.999 for cmscore1, and -13000 for score1. You can tweak these values until you get good accuracy and robustness, and training the acoustic model more would also help. If the output passes all these tests, we then pass the command to the process_sentence() function to move the robot arm. To proceed, let's put the python script aside for a bit and examine the robotic arm USB protocol.

### Robot Arm USB Protocol

According to notbrainsurgery's deconstruction of the robotic arm's USB protocol, the motors are controlled by 3-byte USB control transfers. Since these are ordinary electric motors, we can only control their direction in addition to turning it on or off.

The first byte can be divided into four half-nibbles. A half-nibble is two bits i.e. 00 or 01.

Reading the first byte left to right,

1. the first half-nibble controls the shoulder,
2. the second controls the elbow,
3. the third controls the wrist,
4. the fourth controls the grip.

For the second byte, the fourth half-nibble controls the base rotation.

Finally, for the third byte, the last bit turns the light on or off.

The value of the half-nibble (01 or 10) commands the motor to run one direction or the other, while "00" will stop the motor.

So the python script needs to set the USB control bits appropriately to move the motors. I chose to send a 1-second command to move the motors and then stop automatically. This spares me from having to frantically say "stop" while the robot arm grinds against its safety gear when the movement limit is reached, but the disadvantage is that I may have to issue commands repeatedly to get the desired movement. To do this, we need to import 2 modules: the python time module, and the pyusb module (which requires libusb) for interacting with USB devices:

import time
import usb.core

Now on to the final part of the tutorial!

Tags:

 Traceback (most recent call last): File "./core_julius.py", line 50, in child.expect('please speak', timeout=60) File "/usr/lib/python2.7/dist-packages/pexpect.py", line 1316, in expect return self.expect_list(compiled_pattern_list, timeout, searchwindowsize) File "/usr/lib/python2.7/dist-packages/pexpect.py", line 1330, in expect_list return self.expect_loop(searcher_re(pattern_list), timeout, searchwindowsize) File "/usr/lib/python2.7/dist-packages/pexpect.py", line 1414, in expect_loop raise TIMEOUT (str(e) + '\n' + str(self)) pexpect.TIMEOUT: Timeout exceeded in read_nonblocking().
 
version: 2.4 ($Revision: 516$) command: /opt/julius/bin/julius args: ['/opt/julius/bin/julius', '-input', 'mic', '-C', 'julius.jconf'] searcher: searcher_re: 0: re.compile("please speak") buffer (last 100 chars): >>> before (last 100 chars): >>> after: match: None match_index: None exitstatus: None flag_eof: False pid: 2972 child_fd: 3 closed: False timeout: 30 delimiter: logfile: None logfile_read: None logfile_send: None maxread: 2000 ignorecase: False searchwindowsize: None delaybeforesend: 0.05 delayafterclose: 0.1 delayafterterminate: 0.1