AppleScript – How can I _get_ UI elements' names, attributes, properties, classes programmatically without "guessing" via AccessibilityInspector?
When I started to search for an AppleScript's generic way to "click onto Safari's context menu" of a selected item, there were a lot of questions on "how-to" on several levels of nomenclature.
How can I, without having to use Accessibility Inspector, investigate what "categories" of AppleScript UI elements an application is offering and what additional details AppleScript knows about them?
This is a "study" –an attempt– to find pragmatic ways to identify and "use" AS's UI elements:
Code below works in principle but must be adapted to individ. case (e.g.: find second image).
I am not an IT professional so I apologize for a distinct lack of "on error" code – Sorry!
… nor is this meant to be a "Comprehensive Guide" on AppleScript, rather a "Travel Journal".
(BTW: AirBook, i7-3740QM, 8GB. Repeat-loops on "entire contents" may take 70+ sec's.
System: MacOS X High Sierra 10.13.6, Applescript 2.7. Commands may vary with versions.)
I started my search here at Ask Different, my below "results" mostly apply to thread:
How to know the name of UI elements using Accessibility inspector (or any other tool)
1) How many˜˜ UI elements are there in a "Ask Different" window? …
"Frame": tell application "System Events" to tell process "Safari"
set frontmost to true
set i to 0
Start: set listItems to (entire contents of window 1 as list)
Next: repeat with thisItem in listItems
set i to i + 1
end repeat
end tell
"Result" in
ScriptEditor: 951 -- display dialog ("UIElems:" & i)
¯¯¯¯¯
How many˜˜ elements are there in which different classes? …
Classes: if (class of item i of listItems is static text) then
set classCount to classCount +1
end if
¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯
"Results" in ScriptEditor:
static text (376), group˜˜ (337), UI element (166), button (51),
radio button (12), menu button (6), image (2)
(1 inst. each:) toolbar, scroll area, scroll bar, text area, text field,
tab, checkbox, splitter group, splitter
(˜˜ The "static text" you are just now reading is at the end of this "upper" hierarchy:Group> WebArea> ScrollArea> Group> Group> TabGroup> SplitterGroup> Window
Several levels contain ≈400 real elements: every paragraph & bullet point is a StaticText !)
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈
2) How to identify e.g. those 2 images within 951 UI elements & what to do:
(From here on only differing code-parts, e.g. "Next" are posted; _/¯ = line break.)
Next: set imgNums to {}
repeat with thisItem in listItems
set i to i + 1
Then: if (class of item i of listItems is image) then
set imgNums to imgNums & i
end repeat
return imgNums _/¯ end tell
¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯
"Result" in
ScriptEditor: {111, 255} (≈ 70 sec.! Searching 951 items is tedious…)
¯¯¯¯¯¯¯¯¯¯
… or just "context" the first … (≈ 7 sec.!)
Next: if (class of item i of listItems is image) then exit repeat
end repeat
Then: tell thisItem to perform action "AXShowMenu" _/¯ end tell
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈
3) How to get at AppleScript's "verbal description" of a UI element?
Next: repeat with thisItem in listItems _/¯ set i to i + 1
Then: if (class of item i of listItems is image) then
return (item i of listItems as list)
exit repeat
end if _/¯ end repeat _/¯ end tell
¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯
"Result" for image 1 in ScriptEditor:
image 1 of group 15 of group 6 of UI element 1 of scroll area 1 ¬
of group 1 of group 1 of tab group 1 of splitter group 1 of window ¬
"applescript - How to know the name of UI elements using Accessibility ¬
inspector (or any other tool) - Ask Different" of application process ¬
"Safari" of application "System Events"
¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯
[Last three lines may be shortened to "window 1" – it's still in front …]
– which is this image … 
… of the above mentioned thread.
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈
4) Which attributes are possible for an element, which are factual properties?
Then: if (class of item i of listItems is image) then ¬
return (name of attributes of thisItem as list)
¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯
"Result" for "attributes" in ScriptEditor:
"AXFocused", "AXAccessKey", "AXTitle", "AXElementBusy", "AXPosition", ¬
"AXLinkedUIElements", "AXSelected", "AXLanguage", "AXStartTextMarker", ¬
"AXEnabled", "AXEditableAncestor", "AXHighestEditableAncestor", ¬
"AXVisited", "AXDOMIdentifier", "AXHelp", "AXURL", "AXChildren", "AXRole", ¬
"AXParent", "AXSelectedTextMarkerRange", "AXTopLevelUIElement", ¬
"AXDOMClassList", "AXSubrole", "AXDescription", "AXFocusableAncestor", ¬
"AXValue", "AXBlockQuoteLevel", "AXRoleDescription", "AXSize", "AXWindow", ¬
"AXEndTextMarker", "AXFrame"
Which are the factual properties of "image 1"?
Next: if (class of item i of listItems is image) then ¬
return (properties of thisItem as list)
¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯
"Result" for "properties" in ScriptEditor:
missing value, m.v, {382, 790}, image, "enter image description here", ¬
"Bild", false, "", {241, 118}, "", {}, true, m.v., "AXImage", "", m.v., ¬
false, m.v., "enter image description here"
Obviously factual properties are not corresponding directly to attributes (19 to 32).
But some attributes actually can return values if asked directly:
Then: end repeat
return value of attribute "AXFrame" of thisItem _/¯ end tell
¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯
"Result" for "AXFrame" in ScriptEditor: {382, 790, 623, 908}
Which property is assigned to which attribute? (For exact "AX…" name, see appendix!)
... easy solution, found on Apple's developer pages:
Next: if (class of item i of listItems is image) then
tell item i of listItems to properties
¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯
"Result" for "Bild" in ScriptEditor:
minimum value:missing value, orientation:m. v., position:{382, 790}, ¬
class:image, accessibility description:"enter image description here", ¬
role description:"Bild", focused:false, title:"", size:{241, 118}, ¬
help:"", entire contents:{}, enabled:true, maximum value:m. v., ¬
role:"AXImage", value:"", subrole:m. v., selected:false, name:m. v., ¬
description:"enter image description here"
Well, apparently Apple's own simple command "properties" also returns terms that are actually higher level, like "class:image" and "entire contents:{}".
If you are interested in learning which properties are assigned to which actual attributes try the "appendix solution" (e.g.> "AXFrame:{382, 790, 623, 908}").
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈
- Which "actions" belong to an element (AXPress, AXShowMenu…) ?
I DuckDuckGo[ogle]'d (DD'went?) and found a list of "AXActionConstants" on Apple's site:
"AXShowMenu" "AXPress" "AXCancel" "AXConfirm"
"AXDecrement" "AXIncrement" "AXShowAlternateUI" "AXShowDefaultUI"
"AXPick" Choose a UIElement, like menu item
"AXRaise" Cause a window to become as frontmost as possible
"AXZoomWindow" Zoom a window to "desktop dimensions" but NOT to "fullscreen".
NEW! … found action "AXZoomWindow" through UIElementInspector°° © Apple Inc. 2010
.
- How are menu items addressed best?
(I will soon add my view/findings on these remaining issues.)
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈
- How "different" are classes –and UI elements– in various apps?
- "TextEdit" is an example for minimalistic "functions" of an application window. (Be aware that
its tools windows (Fonts etc.) have UI elements of their own, many more than the main app!) - Its top "bar" consists of 3 coloured buttons, an image representing the document, its title as static text and a tiny down-arrow, a menu button.
-
Apart from these there are (potentially) 2 scroll areas (right & bottom) with scroll bars, also value indicators (their size), and last not least its "contents", a text area.
Classes of window 1 of TextEdit / Finder / iTunes: set classNum to classNum & (class of item i of listItems) – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – "Result" for "TextEdit" (16): button (7), scroll bar (2), scroll area, text area, value indicator, menu button, image, static text – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – "Result" for "Finder" window in list view with left area with "devices" etc. , 11 files (208): UI element (49), static text (49), image (27), row (26), button (19), text field (12), group (6), radio button (4), column (4), ... – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – "Result" for "iTunes" (26): button (12), radio button (3), static text (3), pop up button (2), radio group (2), slider, value indicator, scroll area, text field "Finder" in list view shows icons & tools (= images), files have names, date etc. (static text), tools are also buttons, file names are also text fields (rename).
- "iTunes" (with no TV items "installed") has got 2 pop up buttons (1: music, movies, TV / 2: next, history, song text) plus several sorts of navigation buttons.
- All of these applications, though, also have hundreds of menu items: menus, sub-menus, sub-sub-…menus (TextEdit ≈ 350, Finder ≈ 430).
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈
A P P E N D I X
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈
Older code to get at "assignations" – but returns exact name! ... THAT took some testing:*
Next: if (class of item i of listItems is image) then
repeat with name in (attributes of thisItem as list)
try
if value of name is "Bild" then exit repeat
end try
end repeat _/¯ exit repeat _/¯ end if
try
i & k & item k of (name of attributes of thisItem as list)
end try
¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯
"Result" for "Bild" in ScriptEditor: {111, 28, "AXRoleDescription"}
(Incl.: # of UI element, # of attribute, name of attribute)
¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯ ¯
({382, 790} => AXPosition, {241, 118} => AXSize, "AXImage" => AXRole …)
The try-block is necessary since some value-less attributes otherwise cause errors.
This search on sample page took ≈ 9s. (#111) on THIS page ≈ 135s. (#767) !
≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈≈
Although this is only an
Alternative Way[s]°° to get at a UI element's AS description
I hold it as important enough to justify an answer of its own:
If "normal" procedures as described above do not get you a satisfactory result (e.g: description "paths") or if you are in a hurry, one effective way is to use Automator as an "in-between-agent". Do this:
Open Automator > Click its red-bullet "Record" button > *DO YOUR THING* >
Stop "Recording" > Copy relevant "action icons" > Open ScriptEditor and
Paste those into a new script.
You'll get a lot of error-catching and time-out-routines but also "paths" to used UI elements etc.
Actually Automator is mightier than ScriptEditor in some ways; if s.th. works via Automator scripts/apps it does NOT necessarily also work if pasted into a "traditional" applescript.
E.g., clicking on (right-side) menu bar icons of non-scriptable menu apps WILL work if recorded and played-back by Automator. Sadly it does NOT do so in ScriptEditor.
Interestingly this example will "physically" move your mouse cursor to the menu icon … something that normal scripts will not achieve.
(°° basic description by wch1zpink, posted in "AppleScript UI scripting and click" )
([s]°° I also recommend an older version of Accessibility Inspector named UIElementInspector (version 1.4 from 2010) that works with OS 10.13.6 AND, though graphically less attractive, gives you "names" in addition to classes.