Degenerate UI elements


Sometimes when selecting a UI element, you may see a warning that reads "This UI Element is degenerate":

These are UI elements that, for reasons beyond Flexi-deck's control, are not properly included in an application's hierarchy of UI elements

The Apple API used to traverse the UI elements in an application structures all of the UI elements with a top down hierarchy, where each UI element has a parent, except for the one at the top representing the application itself. Understanding how these elements are structured allows Flexi-deck to search through the tree of elements until it finds the required one on which to perform the action.

However, degenerate UI elements are not accessible from this hierarchy. Either they or one of their ancestors is not properly listed as a child of one of the elements that is in the hierarchy. Hence them being degenerate -- that's not supposed to be the case! There's a number of reasons this can happen, but generally it's due to bugs in another application that we have no control over, and the upshot is that there's no standard way for Flexi-deck to find the element when performing an action.

Fortunately there's one other method provided by Apple for finding UI elements, which is to query which element is sitting under a point on the screen. This is the only method that can find degenerate UI elements. So every time FD looks for a UI element, first it tries using the application's hierarchy, but if that fails, it uses this second method to look at the UI element within proximity of the last point of interaction and searches all of the UI elements connected to that one through its parent / child relationships.

The last point of interaction refers to the last point that Flexi-deck tried to click or interact with in the current macro. It includes:

  • The point that was clicked in last the a "Click" action
  • The ending position of a drag from the last "Click and drag" action
  • The point that was scrolled in the last "Scroll" action
  • The position the cursor was moved to in the last "Move Mouse Cursor" action
  • The center most point of the menu item from the last "Pick Menu Item" action
  • The center most point of the UI element from the last "Send Command to UI Element" action​

Whichever one of these actions was performed last determines the last point of interaction. If none of the previous actions in a macro were one of these, then there is no last point of interaction and it will not be possible for Flexi-deck to find a degenerate UI element.

This means that when a degenerate UI element is selected, it is important that the previous actions in the macro have resulted in the last point of interaction being contained within the desired UI element. Flexi-deck will search directly under the last point of interaction, as well as slightly above, below, to the left, and to the right.

Here's an example of a real world case of this and how it can be dealt with:

A common degenerate UI element is a menu item in a pop-up menu that appears when clicking a button. Logic Pro X, for example, has many of these. Because it's degenerate, it's not possible to simply use the "Pick Menu Item" action like it generally is with a menu item in the application's main menu bar. However, if you use a "Click" action to click the button that produces the pop-up menu, the pop-up menu will usually appear on top of the button. This means that the last point of interaction is contained within the pop-up menu, and Flexi-deck will be able to find the degenerate menu item.

In the case that the pop-up menu appears to the side of or below the button (so that the last point of interaction from clicking the button is not contained within the pop-up menu), you can add a "Move Mouse Cursor" action to position the mouse cursor within the menu. Set the "Move mouse to" setting of the "Move Mouse Cursor" action to "UI Element", set the "UI Element" setting to the button that produces the pop-up, and then adjust its offset to a point that will be contained by the pop-up menu. Now the last point of interaction is contained with the pop-up menu and Flexi-deck will be able to find the degenerate menu item.

Search