UI Element Editor

The UI Element Editor is used to define UI elements associated with certain actions and matching criteria that is used to locate the element when the action is to be performed.

When selecting a UI element in the Macro Editor default match criteria will apply and in many cases this will be sufficient to locate the UI element when running the macro and invoking the action. However, although it is not always necessary to use it, the UI Element Editor allows setting up direct and detailed specifications that can pinpoint precisely which UI element you want the action to manipulate. When used properly, the UI Element Editor can unlock the full power of Flexi-deck!

When a UI element is selected in the Macro Editor it will be shown in UI element field as in this example:

It will either hold the name of the element, as in the example above, or indicate it has not as yet been selected '<No UI element selected>' or show '<Mulitiple UI elements>' if more than one element is defined within the UI element matching process.

Selecting the drop arrow will invoke the UI element editor.

The editor is divided into three sections: the "UI element list"; the "Selected UI element options" and the "Selected UI element hierarchy".

UI Element list

The "UI element list" section specifies a list of possible UI elements that can be matched to an on-screen element. When Flexi-deck searches for a matching on-screen UI element, it will first try to find one that matches the first UI element in the list. Failing that, it will move on to the second in the list, and so on. If none of the UI elements in the list have a matching on-screen UI element, then the action will fail.

The reason to define more than one UI element in the list is for when the on-screen UI element you want to interact with isn't always in the same portion of the screen, or appears in multiple different forms. Here are some examples of this:

• You may want to create an action that clicks a button on a website, but depending on where on the website you are, the button may take on multiple forms. Each version of the button can be added to the list so that the click action will work regardless of which specific page of the website you are viewing.
• You are creating a macro that you want to work with more than one version of an application, and the different versions of the application change around the arrangement of its user interface. You can add a UI element obtained from each version of the application to ensure that the macro will work for everyone who tries it, regardless of which version of the application they're using.
• A specific example in Logic Pro X: when creating a macro that adds a plug-in to a track, you may want to click the on-screen button that opens the plug-ins pop-up menu. However, this button changes depending on how many plug-ins are already inserted into the track. You can therefore add multiple UI elements to list, one for an empty track and one for a track that already has plug-ins added to it. This video shows how to define this in the UI element editor and hopefully shows the power of this feature.

Once you have defined a list of UI elements, you can use the "Highlight Overall Match" button to have Flexi-deck and search for the current matching on-screen UI element given what you've specified in the list. This is the same on-screen UI element that would be found when running the action normally. When there is no match, a red question mark ? will be shown near the cursor. While this button is highlighted, you can work in and manipulate the application containing the matching UI element to make sure that it continues to match correctly in all of the circumstances you need it to.

Selected UI element options

This section allows fine-tuning both how a UI element will be matched, and which point within the UI element will be used for the action, if applicable.

Option
Offset	This option is only available when a UI element is being used to define a point on screen. This specifies this target point within the on-screen UI element by offsetting from the top-left corner of the UI element by a certain number of pixels horizontally and vertically. An offset of (0, 0) indicates the top-left corner of the UI element, while (9,7) indicates 9 pixels to the right and 7 pixels down from the top-left corner. Use the "Set" button to visually set this offset. The currently selected UI element in the UI element list must have an on-screen match. If it is not, a red question mark ? will be shown near the cursor. Similarly to when selecting a UI element, use the same keystrokes to select an offset or cancel as when selecting a UI element.
Use default matching	When checked, Flexi-deck will use a general purpose and lenient method for finding matching on-screen UI elements. Generally, this is done by looking for on-screen UI elements of the same type and that sit in the same position in the application's UI element hierarchy. Occasionally this method may find the wrong UI element due to its leniency, and in that case, two more methods are available for finding a match.
Use strict matching	When checked, Flexi-deck will be more strict when finding matching on-screen UI elements. With this method, your target on-screen UI element and the on-screen UI elements containing it must have the same title, text, description, or any other identifying characteristics that may apply. For example, if a UI element specifies an application window, while default matching may always find the topmost window in the application regardless of its title, this method will make sure the title of the window matches the UI element you specified in the UI element list.
Use custom matching	When checked, Flexi-deck will use a customisable method for finding matching on-screen UI elements. This method allows specifying precisely which attributes must match for each sub-element listed in the UI element hierarchy, and is described in more detail in the "Selected UI element hierarchy" section below.
Highlight Match From List	Click this button to have Flexi-deck search for an on-screen UI element that matches the currently selected UI element in the UI element list, using that UI element's selected matching method. This is useful for individually testing whether each UI element in the list matches the on-screen element it's supposed to.

Selected UI element hierarchy

This section will display the full hierarchy of the currently selected UI element in the UI element list. A UI element's hierarchy starts with the application that contains it, followed by the sub-element of the application that contains it, and so on until it reaches the UI element you originally selected.

For example, if the UI element is a button, it could have a hierarchy that starts with the application, followed by one of its window, followed by a group box, and then the button itself. Alternatively, if the UI element is a menu item, it would start with the application, followed by the application's menu bar, then the menu containing the menu item, and then finally the menu item itself. Every sub-element listed in the hierarchy corresponds to a specific on-screen UI element.

Here is an example of what the hierarchy can look like when the selected UI element is a menu item:

Note that if the UI element is degenerate, it will not start with the application that contains it. See the section on degenerate UI elements for more information.​
When you select a sub-element in the hierarchy list, it will display all of the attributes of that sub-element along with those attributes' values in the attribute list in the bottom-right portion of the UI Element Editor.

If the current matching method for the selected UI element is "Use custom matching", then the attribute list allows specifying precisely which attributes of each sub-element are important and must match when Flexi-deck is searching for matching on-screen UI elements.

When "Use custom matching" is checked and a sub-element in the hierarchy list is selected, the attribute list will become enabled and some of the attributes will be highlighted in blue. These are the attributes that currently set to be important and must match. To toggle this state for an attribute, click the attribute's name in the "Attribute Name" column.

You can also modify the value an important attribute must have by double-clicking the attribute's value in the "Value" column.

This allows you to specify exactly which on-screen UI element you want to match, down to the finest detail.

The purpose of many of the attributes is self-explanatory or can be discovered by experimenting with their values, but there are many that have obtuse meanings that may not be useful for use with Flexi-deck. There are four attributes that are always present: local rect, child index, and homogeneous child index.

Child index is most frequently used by Flexi-deck when finding matches and is often marked as important by default.

You can click the "Highlight Match From Hierarchy" button to have Flexi-deck search for an on-screen UI element that matches the currently selected sub-element in the hierarchy list. This is useful for testing to see if the attributes you've defined as important are correct and allows Flexi-deck to find the correct on-screen UI element for each level in the hierarchy. If no matching on-screen element is found, then a red question mark ? will be displayed next to the cursor.

Video Example

This one shows how to use the UI element editor indicating its ability to locate the correct UI element in an ever changing UI environment:

In the above video we utilise a sequence of click UI element macros to ensure we can locate the correct UI element as required despite the ever changing dynamics of the screen layout.

You may ask why you can't use Send command to UI element option to select the correct UI element, even as it moves. However, the button wasn't just changing position on screen, it was changing its position in the overall hierarchy of UI elements. In fact, it would be more accurate to say that, as plugins are added to a track in Logic Pro X, the first popup button is removed and a new popup button is shown. Flexi-deck must find UI elements based on both their position in the hierarchy and their attributes (such as names) since without both finding the right element won't be reliable, so if button can exist in multiple places in the hierarchy, it becomes necessary to add each version of it.

So in this case, using "Send command" wouldn't have made things simpler because the UI element that the command should be sent to is changing. There are also some issues with the Send command operation such that it can cause problems when used in Logic Pro.

You may also notice that the initial name given to the UI element when first selected is 'Pop up button - Logic Pro x'. In addition to the above explanation, it's also the case that the element we're looking for (the popup button) doesn't actually have a name. What's displayed in the video when it is first select it is a convenient shorthand for displaying what button is selected picked (i.e. a popup button in the application Logic Pro X) but the specific button is actually identified by a whole sequence of UI elements starting from the top of the hierarchy and drilling down until it reaches that specific button. A more complete version of that is displayed in the "UI element list" in the UI element editor.

So that said, since there are multitudes of popup buttons in Logic Pro X, we need all of that information to find the right button to select.

Inevitably it appears complex but the flexibility of the UI element editor allows you to set up some very powerful playback controls in some challenging situations as shown in the above video.