Proximal User Interface - Now What?

Steven Solie sent me the following email:

After reading your well written article (on The Proximal User Interface) I am left with the question, "Now what?"

I like developing software and enjoy applying software engineering principles to my work. GUI's have always been a problem for me with my users often saying "what the?" when they are attempting to use my software.

Your ideas about a more proximal computer interface makes a lot of sense and I'd like to start applying it. I already know that I will run into problems when I try to make Intuition, BOOPSI and ClassAct perform in a more proximal way. I'm going to end up writing a lot of code because the original programmers did not think the same way.

I hope, in the future, there will be extensions to Intuition which allow it to be much more proximal and take a load off of the developers.

and I sent him a brief reply telling him how I went about implementing some of the ideas. An email discussion then ensued and this web page is the collation of that discussion. Steven's words are indented in italics as the above is.

Towards an Implementation of the Proximal User Interface

Now what? The problem we have is that Intuition etc. are built out of the conventional paradigm. So it is very difficult to build proximal UIs on top of them: we not only have to ignore many of the useful facilities they offer us but we have to work against others. We also find they impose oveheads that make proximal UIs only feasible with fast machines - overheads we do not need. Windoze is even worse. Though *maybe* BOOPSI might be a bit better; I don't know.

I've built a software foundation for proximal UI from scratch in C. (I could let you have it if you wish, though I should warn you that it is not really yet ready for public release to the timid!) It is modular and comprises a number of independent modules arranged in three sets, approximalely corresponding to the time-honoured model-view-controller triple. There are data-structuring modules, which set up complex data structures (the 'model'):

Then there are modules that operate the 'view': Then there are modules that operate the 'controller' and which are most concerned with proximal user interfaces: The plan is to have several parallel to BA for specific styles of drawing. These modules embody: There is also one management module, which manages the lot: Then there are two 'whole program' modules so far which integrate these three sets of modules:

I have found that the controller set seem to be reasonably easy to use for creating program-layer things, though one always goes through a frustrating couple of days when one finds things you *want* to happen do not happen (e.g. DW takes its default action rather than the one you have tried to modify it to take) because one has not set up all the inter-module links necessary. But after that it seems to go OK. But I suppose even I am a learner with using this stuff.

I suspect that others in computer science somewhere are far ahead of this type of program structure, and it could do with a huge revamp - but that is not what my research and other work is all about.

Discussion of the Modules (Layers)

RB - Rings and Blocks Module

Start with the concept of a knowledge base, implemented as a single piece of memory of any size needed. Blocks of memory are allocated from this that have
I'm confused on what you mean by grouping to any level. Are you meaning that you may have as many rings of blocks as you want? That brings up the question of whether a block is allowed to be in many rings or not
Yes and yes. It is designed for flexibility. The rings part of a block comprises: Each ring (start and link) (normally) has a purpose. e.g.: Thus, so far, we have the following ring words in an Easelpiece block: but in actual practice there are also a few more. But that hopefully gives an example to clarify.

EA - Easels and EaselPieces Module

Use some blocks to hold details of what to draw (and therefore what has been drawn) on an easel (as I call it; from your point of view treat it as a glorified Rastport). The holding details of what has been drawn (EaselPieces) is important since it allows you to find out what the mouse is over. Now, all easelpieces in a given easel are linked by a single ring of pointers, which terminates on a block used for the easel itself. Note that grouping to any level is allowed.

One major design feature on EA module is that the EaselPieces each hold information about what type of graphic they are - line, closed figure, whether poly or simple, bitmap, group, textpiece, etc. This means that you can hav any mix of them in a ring. It also offers a kind of polymorphism, so beloved of the OO community.

UA - User Action Module

Now take IDCMP messages and process them in a more convenient manner, detecting gestures like clicks, multiclicks, and complex mouse operations. While any of this is proceeding, an optional routine will tell you what easelpieces are affected by what types of operation. e.g. click over an easelpiece, a mousemove over the background, etc. (Also integrates keyboard with mouse.)

One large class of user actions is that of clicking on certain things - and indeed this is common in WB etc. Another is that of picking up something and dragging it - such as an icon in WB or the handle of a slider gadget. Yet another class of user actions is to draw or manipulate things, such as drawing a line, as in many drawing packages. Now, for each class of user actions we need to be able to do the appropriate things in response to mouse gestures.

SK - Sketching Module

Take the latter: drawing. As mouse moves various lines stretch with the mouse, and/or things move. But what? So in the SK layer, (sketching) we create a block for each 'sketchpiece' - a graphic primitive such a line or rectangle that moves with the mouse. These sketchpieces are created at the start of the mouse operation, normally, and are deleted at the end, or when it aborts. A sketchpiece holds data to say: Thus in a box and arrows diagram, to move a box and all its arrows we create a skp for the box itself (rectangle) and one for each arrow (lines) with purposes 'box' and 'arrow-end'. What skpieces are created depends on the next outer layer. This layer deals with the moving and then deleting of whatever skpieces are created, and other administrative matters. It is designed to allow complex multi-piece drawings.

That leads me to a question about the mouse pointer itself. Is the pointer not a SketchPiece? You may want the pointer to change shape as it is moved over EaselPieces or sections of EaselPieces. For example, near the end points of a line you may want it to look like a rubber band for "stretching" the line. After it leaves one EaselPiece, it is destroyed and a new SketchPiece is created. These actions would take place just before and just after an operation.

1. No, the mouse poiniter is not itself a sketchpiece. Two benefits of this. a) It actually allows the mouse pointer to move slightly offset from the 'active' point, so as not to obscure it such as when doing fine drawing and positioning. b) allows the outer layers to supply whatever mouse pointer is appropriate.

2. But you can make up sets of skps of arbitrary complexity. At present they can be lines and rectangles, but it would be easy to (and has always been planned to, but no time to do so) they could be a Bob as well.

DW - Drawing Module

Next layer: DW, drawing. Knowledge of what easelpieces an operation started over, and where in the easelpiece is started (in the middle, on an edge, on a vertex, etc.) can be used, depending on routines supplied by an outer layer, to decide what type of visual action is to be carried out. e.g. in B+A, press LMB over box edge and we start drawing a link (one skp: a line for the new arrow). Press LMB over middle of box and we start to move it and its arrows as above. What happens at end of operation? In DW, if we are drawing a new thing then an easelpiece block is created for the new thing and given data of position and size derived from the latest data found in the relevant sketchpiece. If moving a box and its arrows then at the end, all the relevant easelpieces are updated re. their positions and end points according to the latest data in the corresponding sketchpieces. the DW layer preforms almost all the functions of a standard vector drawing program during the act of drawing. It allows grouping, duplicating, as well as sizing, moving, creating new poly vertices, etc. But it has a certain style that is proximal, which means the user does not have to select the easelpiece to operate upon first (as when normal drawing software gives you handles) and then perform the operation. Rather, it 'intelligently' detects what part of an easelpiece is pulled with the mouse and acts accordingly. So, to bend a line, simply pull it in its middle. To stretch a box simply pull one ot its edges or corners. To move it simply pull it in the middle, etc.

Next, two important questions:

By styles we mean e.g. box and arrows drawing is different from contour lines, and each has its own distinct semantics.

This is where a user interface style guide is a must. The context would also be very important for the user (eg. drawing versus erasing).
Exactly. But since there, at my last count, aroudn 15 different styles of drawing, we need 15 parts to the style guide, and all different.

Styles

I'm not sure what you mean by styles. A picture of the famous march you referred to may help. I understand the need for distinct semantics but I don't see how you fit that in.

Here are some examples:

Combining styles in one picture

How then do you solve the problem of giving the user the correct context? I usually handle it now by poping up a requester full of buttons with the title indicating the context. Perhaps a row of indicators which tell the user what is going on? For example, when I'm "drawing" in my paint package I see a paint brush icon highlighted. When I'm "image processing" I see the corresponding icon highlighted. I'm thinking about modes.

If one allows several 'styles' in a single diagram or easel (e.g. a map with contour lines and also network of roads, rivers, railways, etc - then we have two styles (5,6). If also we have little shapes to show churches, etc. then we have style 1 too. We might also have a wee bar chart in each area of map to show proportions of pupolations etc. - style 4.

Now, how to combine them? Requester (or toolbar)? That is needed for adding new elements to the drawing. But not for modifying existing elements. Each Easelpiece (a) holds an indication of its purpose and (b) has links ('Expression') to the item or relationship or attribute it expresses, and by following that link we can find out what it is. So, by either (a) or (b) we can find out what style of drawing action to start on this thing.

BA - Box and Arrows Module

So for instance one next layer out combines simple drawing actions into the style of boxes and arrows - such as ensuring that all arrows move with the box to which they are attached. There are planned to be a number of modules, each for one style. And note that the whole system is designed to allow several styles in a single easel, rather than each easel having a single style. The aim is to allow us to return to the complex types of diagram that our forefathers used, such as the famous one about Napoleon's march to and from Russia.

KT - Knowledge Tools Module

The next layer out is perhaps the most important one. It deals with the purpose of drawing, in whatever style we use. We can simply draw things, and print them out - but what then? To be useful we must attach each easelpiece to some meaningful object that it expresses. Each such object occupies one or more memory blocks, and is linked to the easelpiece by ring of pointers. In this way any user action on an easelpiece can be interpreted semantically. e.g. in a B+A diagram start drawing a new line from edge of one box and release over another, and once the operation finished we create a block for a relationship and attach it to the two item blocks expressed by the two boxes. Or, in a different style, if we draw a wall across a field this forces the creation of two fields where there had been one.

Next layer out uses the results of layer 7. And so on. But as you can see we are nearly at the outermost layer, that of the main program.

MOD - Module Management Module

Then there is a module-management module that keeps all the above working just fine, and also allows flexibility. The flexibility is provided by structures for each layer (=module). e.g. xxOpControl usually holds, for layer xx, parameters to define the precise shape of the operation and also optional routines to be called at certain key points such as just after it has started and just before it ends. It does get fairly complex, but there is I hope an overall standrad structure to it.

About the Proximal User Interface

I'm not entirely convinced that the idea of a proximal interface is that far of a leap forward. I'm beginning to see it as a methodology for redesigning your GUI and extending it to be more proximal. Add a layer onto Intuition to handle the vector objects versus the bitmapped ones. But how far ahead is this idea?

Exactly. A methodology. Or, rather an approach. Not so much a technically different thing. Except that the GUI APIs we have at present are not flexible enough to fully and efficiently support it.

How far ahead? Well, CLI people in 1980 asked that about GUI. The main difference is that standard GUI focuses on the BL-SL (lexical-syntactic) mapping while proximal UI focuses on the BL-KL (lexical-semantic) mapping. So, to those who are used to thinking and discussing and researching mainly the syntax (SL) the semantics (KL) seems of little concern to them. It is, they feel, merely a concern for the users themselves to sort out. PUI says "No, there are good and bad ways, and we must link in the KL demantics directly.)

Andrew Basden.