Unscriptable apps using AppleScript. What can Applescript see?

At the moment, I can control menu bar items and keypresses, which is excellent, but i'd like to to be able to know everything that ApleScript can see inside an app's window.

get properties returns some good things, but is there anything more powerful?

tell application "System Events" to tell application process "Clearview"
  get properties of window 1
end tell

tried this:

delay 2
tell application "System Events"
  tell front window of (first application process whose frontmost is true)
    set uiElems to entire contents
  end tell
end tell

but it times out for Clearview and even Terminal.

Solution 1:

AppleScript can see pretty much all the UI elements via System Events, but to get items in a manageable form, instead of just throwing everything into a box, you really should traverse the hierarchy of the desired element(s). For example, there may be several objects (such as splitter group, scroll area, etc) in an application window, but those can each contain other elements, which can contain other elements....

This is one of the reasons that GUI scripting is so problematic. You need to know the specific hierarchy of the element or control, but that can change based on the application version and the particular layout at any given time, so it is helpful to know how it was found in the first place.

For spelunking the UI hierarchy, Xcode includes an Accessibility Inspector application, but it can also be done manually in the Script Editor, where the object references can be viewed in the Log window. For an example, start with the UI elements of an application window, here the front window in Terminal v2.11 (Big Sur):

tell application "System Events" -- peek into the rabbit hole
    tell application process "Terminal" to tell window 1
        set elements to its UI elements -- get elements at the current location
        if elements is {} then -- no more
            return its value
        else
            return elements
        end if
    end tell
end tell

Then if elements are shown in the results, pick one and add it to the tell statement to get its UI elements, and so on, until you get to the control you are interested in. Continuing from the above example (the ¬ is a line continuation symbol from using option-return (or option-l) to try to keep the formatting reasonable):

tell application "System Events" -- kick the rabbit out
    tell application process "Terminal" to tell window 1 ¬
        to tell splitter group 1 ¬
        to tell scroll area 1 ¬
        to tell text area 1
        set elements to its UI elements -- get elements at the current location
        if elements is {} then -- no more
            return its value
        else
            return elements
        end if
    end tell
end tell

The final object reference for the above would be something like text area 1 of scroll area 1 of splitter group 1 of window 1 of application process "Terminal" which can be copied from the Script Editor event window to use or keep around.

Solution 2:

Accessibility Inspector

Apple publish a graphical tool called Accessibility Inspector. It is designed to explore and test the accessibility values associated with an interface. This is the same information AppleScript uses for Graphical User Interface (GUI) Scripting:

Accessibility Inspector makes it even easier to identify user interface element information. This app is included with Xcode. To use it, open Xcode and select Xcode > Open Developer Tool > Accessibility Inspector.

See Apple's Automating the User Interface documentation for examples.

Solution 3:

tell application "System Events" to return value of every attribute of UI elements of windows of application process (name of first process whose frontmost is true)