Prototyping a Better Virtual Keyboard

Table of Contents

1. Introduction

1.1 Goal

1.2 Aspiration

2. Research

2.1 Experiments and studies into chorded keyboards

3. Prototype

3.1 Implementation

3.2 Application

3.3 Layout

3.4 Character Mapping

3.4.1 Observations

3.4.2 My method’s mapping

4. User testing

4.1 Participants

4.2 Procedure

5. Analysis

5.1 Usability

5.1.1 Key optimisation

5.1.2 User satisfaction

5.2 Memorability

5.3 Typing Speeds

5.3 Future work

6. Conclusion

7. References

1. Introduction

Touch displays have been an important part of how we communicate with technology since the widespread popularisation of smartphones and tablets. Although the touch screen makes for a more convenient and straightforward interface with software in many ways, the practise of typing text has remained largely unchanged from its physical origins which is the computer keyboard.

The mobile and tablet market is still growing, attracting new people with little to no computing expertise. Since current text input implementations depend on anticipated experience with computer keyboards, it's worth looking at alternate and better-suited solutions for text input on touch devices.

1.1 Goal

Certain compromises had to be taken in the conversion of the physical keyboard to touch screens. Today's touch screens are typically far shorter than a traditional computer keyboard, forcing the keys to be shrunk to fit on the panel. This necessitates very careful finger placement while typing, which often results in several errors.

Another disadvantage to using a keyboard on a touch screen is that it loses its tactility. Most computer experts rely on their ability to sense where their fingers are placed in order to easily shift their fingers to the desired keys. The ability to type easily and without looking at the keys has seriously decreased without being able to sense where the fingertips are in comparison to other keys.

With these disadvantages to today's standard touch input devices, the only benefit being that it is in a style that users with prior computing experience are familiar with. However, as a growing number of inexperienced computer users begin to use touch technologies, this benefit is becoming increasingly obsolete.

Experienced users would find it simple to start typing on touch devices without any instructions thanks to the familiar format of the keyboard. However, if an attempt is made to learn to use an alternate text input method, a person will be able to type more quickly. This alternative system will be best developed exclusively for touch screens, welcoming rather than fighting against their limitations.

This takes us to the project’s main goal: to provide a feasible text input option for touch devices that is not based on the poor design of the QWERTY keyboard.

1.2 Aspiration

As previously said, the design of today's standard touch keyboards has a range of serious flaws. This project investigates alternative text input methods for touch devices in order to provide insight and ideas for replacing the existing common, but flawed, interface keyboard. The ultimate aim of this area's study and exploration is to obtain valuable insight into how to design a better text input mechanism for both advanced and beginner computer keyboard users on touch devices.

The desired realistic outcome of this area of interface design research is not to be able to create a device that automatically increases our typing capacity on existing touch keyboards. Instead, the ultimate aim is to be able to design input mechanisms that fully exploit the advantages of touch screens and, once mastered, have the ability to provide a more effective and enjoyable typing experience than current systems.

2. Research

Since the invention of the QWERTY keyboard, a range of attempts at developing alternative text input systems have been developed, all of them aimed to create a mechanism with less keys(buttons). The principles discussed in these applications have since been incorporated into alternative touch keyboard implementations. This chapter is devoted to providing the reader with an outline of significant physical and virtual entries in this field.

2.1 Experiments and studies into chorded keyboards

A chorded keyboard (also known as a chord keyboard or chording keyboard) is a computer input system that helps users to insert characters or commands by pressing several keys simultaneously, similar to playing a chord on a piano. The oldest known chord keyboard was part of Wheatstone and Cooke's 1836 "five-needle" telegraph operator station [1]. A keyer is a chorded keyboard without the board that is normally designed to be used when holding in the hand. One-handed and two-handed versions of modern chorded keyboards are available.

The idea of a chorded keyboard is not new; the SHIFT, CTRL, and ALT keys on a computer are all designed to be used in chords. A chord keyboard, on the other hand, makes more use of chords, allowing a larger number of characters to be generated with a smaller number of keys. A collection of n keys will generate the number of characters given by the formula (2^n-1) without using any other mechanism than recording keys pressed down at the same time.

Microwriter

The Microwriter (Figure 1) is an example of an old commercial chorded keyboard that was designed in 1980 by Enfield Cie in the United States and marketed in the early 1980s by Microwriter Ltd in the United Kingdom [2]. Microwriter is a chording keyboard-equipped handheld word processor. With only six buttons, you can type all alphabet letters, numerals, and punctuation marks. Microwriter was not a commercial success, owing to a problem that characterized all chorded keyboards: learning to type is painful at first. On the other hand, several people who took the time to study the device were pleasantly surprised by its capabilities.

Figure 1: A Microwriter MW4 (circa1980)

NLS

Douglas Engelbart's NLS, was a multi-user system that he demonstrated in 1968. The machine came with a first-of-its-kind electronic mouse and a five-key chorded keyset that could be used at the same time. The system's aim was not to make programming more available to the general public, but to help expert users increase their effectiveness by learning a better system.

The machine also included a standard QWERTY keyboard alongside the set of mouse and chorded keyset. The set would be used to more quickly edit text whilst the keyboard to type large amounts of text. Characters could be added and extracted from the document more efficiently by pointing the mouse at various portions of the text and entering commands on the keyset. The mouse had three buttons, two of which could be used to create chords with the keyset to expand the number of characters and commands available.

In late 1950, tests found that the machine could be learned in less than two hours, and that although standard keyboards are quicker for normal typing, Engelbart's mouse and keyset system are far more effective for editing and text manoeuvring.

ASETNIOP

Pointesa LLC created ASETNIOP, a chorded virtual keyboard. It has ten set keys that are intended to be used for all ten fingers at the same time. The space bar and move are regulated with the thumbs, while the rest are used to form chords. A total of 28 chords, with 18 for letters and 10 for punctuation marks that are widely used. There is also a different interface that can be changed easily to allow access to numbers and other icons.

ASETNIOP is the name for the eight letters that can be typed without chords and are assigned to the fingers in order from left to right. The positioning refers to the fingers that will be used on a standard QWERTY keyboard while typing. These letters were chosen because they contain the most commonly used characters in the English language.

UpSense

Another newly released touch keyboard substitute for smartphones and tablets is UpSense by Inpris Ltd. Swipes with one or two fingers in four different directions are used in addition to chords to shape letters. [3]

Four, five, or ten fingers can be used for the keyboard. Any letter can be personalised to the user's desired chord or gesture, but the defaults are designed to be as intuitive as possible, with movements that match the letters' form.

To differentiate between the user's various fingers, the feedback must first be calibrated by putting both of the user's fingers on the screen. This causes circles to appear on the screen, denoting the contact area for each finger. Swipe movements must begin within the designated circles and pass outside before raising the fingers.

Many chorded keyboard styles are based on the Braille scheme, which is a commonly used way of reading and writing for blind people. It was founded in 1825 by Louis Braille. The Braille scheme is based on the “night writing” mode of communication. Each Braille character consists of six dots arranged in a rectangle of two columns of three dots each (Figure 2).

Figure 2: Braille alphabet

As opposed to other text entry types, such as QWERTY, chorded keyboards provide many benefits. Best one of all is that these keyboards are mainly used with little visual feedback. Most chorded keyboards, on the other hand, cannot be played using the hunt and peck system (looking for each key separately before striking it). As a result, chorded keyboards necessitate further instruction until the text entry process is fully grasped and typing speed is appropriate. As a result, chorded keyboards are often regarded as only suitable for certain activities or situations in which conventional typing methods or technologies are unavailable.

Gopher and Raij introduced some of the findings of their experiments on a two-handed chorded keyboard in 1988. Two panels with five buttons, one for each side, make up their keyboard. After 20 hours of instruction, users were able to achieve speeds of 30-35 words per minute, demonstrating that the device allows for rapid skill acquisition. Users achieved admission speeds of close to 60 words per minute after 60 hours of preparation. When compared to QWERTY, the observed learning rate is substantially higher. The new chording ability had no detrimental effects on participant’s QWERTY keyboard typing performance, according to Gopher and Raij. [4]

3. Prototype

A comparatively unusual prototype was introduced for this project to investigate the capabilities of alternate virtual keyboards in order to make it easier for the user to tap the correct button. The method is an app-based implementation that uses chords on twelve touch keys to generate the letters of the English alphabet.

3.1 Implementation

The prototype is build in Android Studio and therefore is made specifically for android devices. The main language used is Java. Because of my prior experience and level of familiarity dealing with Android Studio, developing the prototype in this context was a natural option, but it also offered a number of other advantages:

• Inbuilt Android Virtual Device to test Android applications.
• Android Debug Bridge.
• The load of effort required to set up and try the prototype for other users is considerably reduced. All I have to do is send the apk file to them.
• It comes with 24x7 Google support and JetBrains support to resolve problems, etc.

The prototype was mainly tested on Android 7 and higher, but it should work on any Android phone or tablet version.

3.2 Application

The following is a rundown of the various features and aspects of my prototype. Figure 3 can also give you a clear idea of how the software looks when its running.

Figure 3:Working prototype of the keyboard

3.3 Layout

The prototype keyboard starts like most virtual keyboards do. As soon as you enable it from “Manage keyboards” in settings, you are good to go. The design is twelve separate keys split in two between the left and right side of the screen. Instead of being limited to a set position while typing, this makes for a more relaxed hand placement. No delay of any kind between each character to be able to produce the best accurate results as possible.

3.4 Character Mapping

Finding a strong balance between memorability and efficiency is a major challenge when developing a character mapping scheme. Consider a device that reads input by allowing the user to write letters on the screen in handwritten form. Since the user obviously already knows how to write by hand, this would be very simple and unforgettable, but the degree of efficiency would be very poor since large gestures are required for each letter entry. The complexity of the gesture is also unrelated to the frequency at which the letter is typed.

In the other hand, an optimised scheme will map commonly used letters to basic commands that can be executed quickly, while assigning less frequently used letters to more complex inputs. The disadvantage of this scheme is that the letter movements are more difficult to recall because they are not dependent on what the person has previously learned.

3.4.1 Observations

Let us look at the input structures from the research section before moving on to the prototype’s character mapping.

NLS

The binary mapping scheme of Douglas Engelbart was not created with efficiency in mind. The mapping is simple to understand for someone who is familiar with the binary numeral system, as many of the participants in the NLS target group were.

Understanding the mapping scheme, on the other hand, does not always make it easier for users to recall the chord for a given letter when typing. Apart from the first and probably last couple of letters in the alphabet, most people are not capable of easily understanding the exact positional order of a letter in the middle of the alphabet without any preparation, and even less so when the number has to be in binary form.

Microwriter

The Microwriter's letter mapping was a mix of efficient and memorable. The letter E, the most widely used letter in the English language, was efficiently mapped to a single press with the index finger.

Even if many of the mnemonics seemed contrived or far-fetched, they were useful in learning the chords. For instance, the mnemonic "most common finger for most common letter E" is illogical – aren't all fingers equally common?

The aim of mnemonics, regardless of their absurdity, is to assist the brain in converting knowledge into a more retainable form. And if the recipient is offended by the mnemonic's lack of sensibility, it has already accomplished the goal of making the particular letter mapping unforgettable.

ASETNIOP

Though ASETNIOP is primarily focused on improving typing speed, the mapping of the eight single-finger letters is based on a standard QWERTY keyboard, making it more intuitive and unforgettable for users with experience. The system's name also functions as a mnemonic for recalling the sequence of these eight characters.

UpSense

UpSense's character mapping is based on a simple and memorable scheme. It's easy to see that swiping up with the same fingers generates the letter W until the user discovers that a downwards swipe with three fingers produces the letter M. Despite the fact that using swipes in addition to chords allows for more flexibility in assigning movements to letters based on their forms, the form tracing technique does not work on all letters in the alphabet.

3.4.2 My method’s mapping

The frequency of the letter in the English language was a major factor in deciding the number of keys to use in the chord when marking out the characters to chords. A chord with fewer keys should correspond to a letter that is used more often. Figure 4 depicts all of the letter chords. And these are not all: all the quotes, grouping symbols, numbers and shifted numbers are attached in the appendix.

Figure 4: All mapped letters of the alphabet.

The letter forms are taken into account in addition to the extent of use of letters in the mapping. The user would find the method more straightforward and easier to remember if chords are mapped to the form of the character.

A Researcher at Google Inc, created the following organised series of letters based on their frequency using data from English books scanned by Google: ETAOIN SHRLDCU MFPGWYB VKXJQZ. [5] The eight most common letters are mapped to single keys: E, T, A, O, I, N, S, and H; as well as space and repress button. The rest are two-key chords.

Although not all chords have greater meaning than some, there are a few that stand out:

• The backspace key uses sliding motion to the left over two-key chords from the middle rightmost keys to simulate moving back when removing characters.
• The numbers keys are created on the right pad by sliding just as like you would write them by hand. Attempted a second try for left-handed people (see appendix for all key mappings)

4. User testing

4.1 Participants

I used social media to find ten people, three of whom were female. Their average age was 24. (range: 18-30). One was left-handed. Students made up 80% of the group, with 50% of them having a computer science background. The majority of them were experienced using a touch keyboard.

4.2 Procedure

I used my app and two questionnaires, one at the beginning and the other at the end of the test. Users used their own smartphones. Independently, I explained the analysis, the logged info, and the privacy security scheme to participants. I also familiarised them with the ethics forms, processes, and procedures that I will need to use and conform with them when conducting user involvement / data collection tasks during the project. I enabled the software on their personal smartphones and configured it to fit their usual settings. I sent out an email seven days later inviting participants to complete the final questionnaire and instructing them to uninstall the software.

The participants had different levels of understanding of smartphone text entry techniques. The initial questionnaire was used to determine the participants' degree of familiarity with mobile text entry methods. The multi-tap process was mostly known to everyone. Other methods were less well-known. 5 out of 10 participants had a positive experience with T9, 4 had tried it, and only one had no experience with T9. 10 out of 10 participants were familiar with the virtual QWERTY keyboard.

Many of the participants were able to follow the instruction during the test and feel comfortable using the app.

The first phase of the test was an individual one. It was a two-day learning phase of the character map that users performed individually. Although the prototype should be trained and checked with users over a long period of time to fully determine its ability, the project's intended scale has limited the scope of the user research. Despite this, the restricted user testing has proven to be a useful tool for determining the design's flaws and strengths. The second phase consists from each user typing 10 sentences each of which 5 pangrams (sentences using every letter of the alphabet at least once).

The elapsed period of typing two different sentences was used as a benchmark to test the user's typing performance on the device. First one was taken from the pangram sentences "Pack my box with five dozen liquor jugs”. This elegant sentence assesses the user's skill and productivity in locating chords for all of the alphabet's characters. This sentence can also be used in chord memorization preparation.

"Lie down with dogs and wake up with fleas," is the second benchmarking non-pangram sentence, which is made up entirely of letters generated by chords with no more than two keys. Because of the similarity between the number of keys in a chord and the letter's frequency of appearance in the English language, this sentence was used to more precisely represent how a person would type in a real-world situation.

During the trials, users were allowed to use the alphabet mapping system (figure4) but only as a last resort if they couldn't recall the chord for the current letter. Both sentences were written and timed several times, with the average speed and best attempts being registered. The users were then asked to type the same sentences using the device's normal QWERTY keyboard, with the average speed and best entry being recorded once more.

Autocorrection was switched off while calculating typing speed on the virtual QWERTY keyboard. Although this may be a function that the user has enabled when clicking on the keyboard on a daily basis, it is not a feature that is exclusive to the input system's design. Without changing the design of the method, an auto-correction feature may be applied on the prototype (on future work).

5. Analysis

This chapter focuses on the study of the experiences obtained from the prototype development and user feedback. Both evidence and observations are explored to provide a foundation for drawing conclusions in the final chapter.

5.1 Usability

The overall usability of the prototype is a difficult to quantify yet critically significant factor. Although memorability and efficiency are both aspects of keyboard’s usability, they will be discussed in greater detail in a later section. The interaction method of pressing the keys of the prototype is the subject of this segment.

5.1.1 Key optimisation

As described earlier, the keyboard is displayed as twelve buttons split in two between the left and right side of the screen which makes it easy for users to find convenient hand positions for typing.

With 12 buttons there are millions of possible combinations so a logical thinking at optimal keys was critical. As well as the size of the key markers, which on the other hand, allowed users to leave enough space between each finger, lowering the chance of pressing the wrong keys.

Since there are only six keys on both sides of the phone, the user is much more likely to concentrate on the written text rather than the buttons. Typing errors are caught and corrected more easily because the user is actively looking at the written text. When clicking on a physical keyboard, this is something that an expert computer user would benefit from, but it's something that everyday touch keyboards struggle with. On touch keyboards, auto-correction often assists in the correction of small typing errors without prompting the user to take any action; however, the auto-correction mechanism will sometimes yield unexpected effects that entirely alter the context of the written sentences.

5.1.2 User satisfaction

One feature of usability that is often debated is the level of happiness that the user experiences when using the prototype. Certain keys can feel uncomfortable to type depending on the user's dexterity (left or right-handed, hand posture, etc.). Long periods of usage, particularly when compared to using the device's standard keyboard, may be taxing. This is most likely due to the user's lack of experience with repeating certain types of movements, something that the user will learn to do over time.

Despite the outlined inconveniences of using the prototype, users characterised the virtual keyboard as "fun" to use. When the users first started practising the “chords”, he or she felt a sense of pride and accomplishment similar to the emotions encountered when learning to play a new instrument. Although the pleasure derived from using a text input device might not be the first thing on a user's mind, it is an interesting factor to consider.

5.2 Memorability

Seeking a good balance between efficiency and memorability character mapping, as described in the previous chapter, is a difficult challenge. This prototype is more concerned with mapping letters to effective chords than with making them memorable.

However, the few mnemonics employed proved to be effective in aiding the user's memory of the accompanying chords. The chords that have no meaning, such as the chords for the letters x and z (sliding thumbs), are the ones that users have the most trouble with.

When a user forgot a certain chord, the character mapping was made open to them so that they could remember easily. However, the users almost always forgot the chords within a short period of time, forcing them to look it up again. More widespread use of mnemonics is likely to help mitigate this issue. If the user forgets a chord, these mnemonics could be shown alongside a set of highlighted keys to more effectively inform the user; a different highlighting color for different chords.

All 10 participants needed one to two hours to learn and memorise all of the chords, which is close to the NLS keyset's recorded learning time. While this does not seem to be a long time to learn a brand new text input, it may create friction for new users, which can ideally be minimised. Friction reductions do not, though, come at the expense of performance or other facets of usability.

There is still a lot of space to boost the user's efficiency of the keyboard even if they have memorised all of the keys. This gains can only be made by repeated practise, which allows the chords to become muscle memory and improves hand dexterity, making for more graceful formation of the various movements. Since I only had one week with the participants due to the project duration, I was unable to explore the prototype's long-term aspects.

5.3 Typing Speeds

Both an average and best entry were registered when users' typing speeds on the prototype were compared to the device's normal QWERTY keyboard. The best entries act as a benchmark for the maximum typing rates that can be achieved. Figure 5.1 shows the effects of the first sentence, which indicate significant speed variations between users on the prototype, but more stable results on the QWERTY touch keyboard.

"Pack my box with five dozen liquor jugs"

Prototype QWERTY Keyboard

Figure 5.1: Results for the average and best typing speeds of the prototype and the usual virtual keyboard for the sentence: “Pack my box with five dozen liquor jugs”

The second sentence, whose effects are seen in figure 5.2, is a more precise description of the letters that a user would type in standard English text. The findings indicate strong speed gains for most users, which is to be anticipated given the sentence's simplified complexity. It is clear that the typing speeds of the two sentences on the device's QWERTY keyboard do not differ significantly, and the range of letters used has little effect on the attainable typing speeds.

"Lie down with dogs and wake up with fleas”

Prototype QWERTY Keyboard

Figure 5.2: Results for the average and best typing speeds of the prototype and the usual virtual keyboard for the sentence: “Lie down with dogs and wake up with fleas”

Based on these findings, as well as the other 8 sentences (of which 4 pangrams) a possible prototype typing speed of 35 - 55 percent of QWERTY touch keyboard speeds can be estimated. On the prototype, this translates to about 23-28 WPM, and 34-51 WPM on the regular touch keyboard.

With the limited amount of time with the prototype compared to the users' several years of experience with QWERTY keyboards, this has a lot of promise.

5.3 Future work

In addition to the goal of effective letter placement and smaller keys, I wished to implement a collection of other features that would’ve simplified the learning process, increased typing speed and minimized typing errors; but unfortunately the limited time schedule didn’t allow me to finish. Some of these are:

Ambiguity

When first getting started with the app, a disambiguation scheme is used with the goal of allowing users to ignore the chording mechanism. For eg, instead of typing the word “three”, which requires the letter R, which is not among the eight non-chorded letters, the user might type “thnee”, and the machine will immediately replace it with the correct word.

Autocorrection/Prediction

The machine attempts to autocorrect terms that could have been mistyped as a result of using a chord instead of two subsequent letters, or vice versa. When a user types in a word that isn't in a regular English dictionary, the machine attempts to substitute pairs of single letters with chord letters or chord letters with their component parts to see whether a word is created.

Word combinations

Although all of the letters of the alphabet can be entered with just two fingers pressed together, digraphs, trigraphs, and even whole sentences can be entered immediately with three finger chords or more. The word “the” is generated by pressing down the fingers for typing T, H, and E at the same time, regardless of order.

5.4 Project Management

Since the start of the project, a few things went off track from my original plan, eg. participants were asked to only test the prototype instead of helping me build one by completing surveys. Because of that, new project milestones had to be set and that made me derail from my mid-term report in terms of priority scheduling and features. However that only gave me the motivation to work that much harder to achieve all the possible milestones in the time left.

6. Conclusion

Returning to the introduction, the study attempts to determine if this prototype touch keyboard solution, one that is not dependent on the QWERTY keyboard and has the potential to be more effective, can be built. ASETNIOP, NLS, and the prototype, all are based on programming paradigms that are very different from the design of a physical keyboard. These options are built so that hitting the right keys on the keyboard would not necessitate exact finger placement, and with only 12 chords (buttons), this makes it much easier. This allows the user to concentrate their vision on the written text rather than their fingertips, which is important for easily correcting possible typing errors.

It's impossible to assess the feasibility of a text input device critically. The feasibility of the keyboard style is highly reliant on the user's commitment, but this is also so for newcomers learning to use a QWERTY keyboard. The user can effectively type on the prototype at a steady pace until they have memorised all of the chords, which takes about two hours. Despite the fact that the user's typing pace is likely to be much slower than on the device's QWERTY keyboard, it should be deemed a suitable keyboard option.

Potential typing speeds of up to half the speed on the device's QWERTY keyboard have been found based on measurements taken with the prototype. These speeds were reached reasonably easily, and with more experience, even faster speeds could be possible. This has a lot of potential in terms of getting users to become more effective with the device.

The prototype's design may not be ideal in terms of alternate keyboard designs, but it demonstrates what can be accomplished with a small amount of effort and study. The ASETNIOP and NLS keyboards demonstrate that there is a desire to learn more about this subject, and I just wanted to share mine and hopefully expect to see more virtual keyboards with unique designs on the market in the future.

7. References

[1] Wikipedia, Chorded keyboard, Dec 2020 https://en.wikipedia.org/wiki/Chorded_keyboard

[2] Wikipedia, Microwriter, Dec 2020 https://en.wikipedia.org/wiki/Microwriter

[3] Bgr, Upsense is an invisible Braille-friendly keyboard for your tablet, Feb 2021 https://www.bgr.in/news/upsense-is-an-invisible-braille-friendly-keyboard-for-your-tablet-293521/

[4] IEEEXplore, Typing with a two-hand chord keyboard: will the QWERTY become obsolete?, JAN 2021 https://ieeexplore.ieee.org/document/17378

[5] Cornell, English Letter Frequency, JAN 2021 http://pi.math.cornell.edu/~mec/2003-2004/cryptography/subs/frequencies.html