How to record and replay mouse activity on automator in the background?

Solution 1:

Update Note:

My original answer, shown further below, was written using a generic example and not the actual target because it had not yet been disclosed. Once it was disclosed, in a comment, I downloaded CopyClip 2 and wrote the AppleScript code using UI Scripting to bring up the Advanced Preferences pane, click the Delete All Clips button, the subsequent Delete All button, close the Preferences window, and close the UI that shows the clips. It works, however it's UI Scripting and you see it as though manually done, but it happens much faster and without seeing visual mouse movements across the screen. (Not really a background solution.)

That said though, I found CopyClip 2 to be very unstable and crashed every other or few times I ran the script, and why I'm not posting the code unless you really want to see it. So I don't recommend the UI Scripting method. Note that the instability expressed was just in conjunction with using AppleScript on the app and not the overall use of it. (I actually have not used it other then to answer the question.)

Looking now for a background solution, I investigated how CopyClip 2 stores the clips, and found where the clip datastore was and in what form. Not surprisingly, it's stored in an SQLite database. So with that information, I see two different options:

The first involves manipulating the data stored in the SQLite database. The problem there is, I don't do SQL programming, so you'd have to pursue that option on your own.

The second option is more of a quasi-brute-force method but it works well and did't crash the app in the process. It involves programmatically quitting CopyClip 2, deleting the clip database, and restarting CopyClip 2.

Either of these option would be considered the background method, especially when compared to using UI Scripting.

The following example BASH script code is what I used for the second option mentioned. It can be run in a number of different ways.

#!/bin/bash

if [ -n "$(pgrep "CopyClip 2")" ]; then
    pkill "CopyClip 2"
    sleep 1.5
    for i in $HOME/Library/Containers/com.fiplab.copyclip2/Data/Library/Application\ Support/CopyClip/*; do
        rm "$i" 2>/dev/null
    done
    open -a "CopyClip 2"
fi

I first tested this a half dozen or so times in Terminal as an executable BASH script before I used the code in a Run Shell Script action in an Automator application, and tested another half dozen or so times. For my 2 cents worth, this is the way I'd go over trying to do the SQL programing option even if I knew SQL.

Saved as an Automator application, e.g., Delete All CopyClip Clips, I used Spotlight to find and execute it, i.e.:

CommandSpacedelEnter

The application could also be placed in the Dock for a quick mouse click to delete all the clips too.

By the way, I also have Hide window as startup set to ON on the General tab in Preferences for CopyClip 2, so as to make this as background as possible in appearance. This way, unless you're looking directly at the Menu bar icon for CopyClip 2, you don't really notice it's been closed and reopened.

Also, note that the clip database is normally created upon first run and only changes as clips are stored or deleted through CopyClip 2. I examined the clip database using SQLiteBrowser when first created and after deleting all the clips and its structure is essentially no different then when first created¹, so repeatedly deleting the clip database really isn't a problem to achieve the goal.
_{(¹ While technically there are some differences, the deleted data had not be removed from the database by the VACUUM command yet, nonetheless this option is extremely safe and should not be problematic with repeated use.)}

Original Answer:

Using the Record button in Automator, you've recorded some mouse clicks on the Menu bar and this produced a Watch Me Do action in the workflow. Subsequently running the Automator workflow containing the Watch Me Do action, to use your words, it plays like a movie.

So you are asking if this can run in the background more like an Excel macro then play like a movie?

Generally speaking, the short answer is both yes and no... yes more like a macro (sort of), but no not in the background.

That said, note however and depending on the target application and what needs to be done, there may be the possibility of coding something to perform in the background but without knowing the explicit specifics it's beyond the scope of this generally speaking short answer.

The longer answer is, to convert the Watch Me Do action into appropriate AppleScript code. However, because it uses UI Scripting it cannot run in the background, but it can run much more efficiently and faster then a Watch Me Do action played back and you do not see the visual mouse movements across the screen. This means that when the AppleScript code is run, by whichever means one choose, one has to allow the script to process its code uninterrupted, thus being one of the drawbacks of UI Scripting.

However, I believe you'll find this more desirable then playing like a movie and seeing the mouse visually move across the screen.

As an example, recording my actions while using the Clock on the Menu bar in this example, I'm going to click the Clock menu item, then click the Open Date & Time Preferences… menu item, then click the Close button on the Date & Time Preferences pane of System Preferences, which is opened as a result of the two previous clicks.

In Automator, I select the three Events in the Watch Me Do action, then press CommandC to copy the events to the clipboard.

Now I open Script Editor and in a new document, press CommandV to paste the copied events from the clipboard.

This produces the AppleScript code shown at the very end of this answer, which is placed there because not much of it will actually be used, but it will have the key pieces of code, so not having to manually type all of the finished code and is only included herein for completeness of this example case.

Within the pasted code there are three commands that represent the actual click events, and since this is using UI Scripting it's being done by System Events.

So, in another new document, in Script Editor, I add a tell application "System Events" block, e.g.:

tell application "System Events"

end tell

Within that tell block, I copied the appropriate portion of the three click events from the code pasted in the other document while removing the unnecessary code and backslashes for this use case, e.g.:

tell application "System Events"
    click menu bar item 7 of menu bar 1 of application process "SystemUIServer"
    click menu item 6 of menu 1 of menu bar item 7 of menu bar 1 of application process "SystemUIServer"
    click UI element 1 of window "Date & Time" of application process "System Preferences"
end tell

Now because this is UI Scripting, appropriate delay commands need to be added, e.g.:

tell application "System Events"
    click menu bar item 7 of menu bar 1 of application process "SystemUIServer"
    delay 0.5
    click menu item 6 of menu 1 of menu bar item 7 of menu bar 1 of application process "SystemUIServer"
    delay 2.5
    click UI element 1 of window "Date & Time" of application process "System Preferences"
end tell

Now while I could run the code above (or even the originally pasted code) and it should function properly at this time, the problem is that menu bar item 7, which is currently the Clock on my system, can change. So too in some cases, can the menu item n change as well, although not in this example use case. However, knowing this, lets rewrite the code so it's not hard coded for the menu bar item number or the menu item number:

tell application "System Events"
    -- click menu bar item 7 of menu bar 1 of application process "SystemUIServer"
    click (every menu bar item of menu bar 1 of application process "SystemUIServer" whose description is "clock")
    delay 0.5
    -- click menu item 6 of menu 1 of menu bar item 7 of menu bar 1 of application process "SystemUIServer"
    click (every menu item of menu 1 of (every menu bar item whose description is "clock") ¬
        of menu bar 1 of application process "SystemUIServer" whose title is "Open Date & Time Preferences…")
    delay 2.5
    -- click UI element 1 of window "Date & Time" of application process "System Preferences"
    -- click button 1 of window 1 of application process "System Preferences"
    quit application "System Preferences"
end tell

In the code above, the original three lines of code have been commented out, i.e. -- was added in front of each line. Also, the original third line has an additional line under it commented out, but was added to show how to generalize the original third line. However, since its purpose in this example was to close System Preferences, just tell it to quit instead.

So the final example version of the code to handle the three events that took place in the example Watch Me Do action in Automator is:

try
    tell application "System Events"
        click (every menu bar item of menu bar 1 of application process "SystemUIServer" whose description is "clock")
        delay 0.5
        click (every menu item of menu 1 of (every menu bar item whose description is "clock") ¬
            of menu bar 1 of application process "SystemUIServer" whose title is "Open Date & Time Preferences…")
        delay 2.5
        quit application "System Preferences"
    end tell
end try

Now comparing this final example version of the code above to the original code shown below, one can see the difference and what's necessary to cleanup the original code while not hard coding the menu bar item number or the menu item number, thus making it more intelligent then using the Watch Me Do action in Automator.

Note: The example AppleScript code above is just that, and sans wrapping it in a try statement, does not include any other error handling as may be appropriate/needed/wanted, the onus is upon the user to add any appropriate error handling for any example code presented and or code written by the oneself. This includes setting the value of the delay commands as appropriate for the timing on ones own system.

Additionally, not all menu items are as easily scripted as this example case and in some cases may not work well to even use this method, it just depends and is one of the reasons I was trying to get more explicit and specific information from you in my comments. That is, I might have been able to give you explicit and specific code for your immediate need, rather then having to write an answer using an example. Although, hopefully this example with be useful to may others as well.

Original AppleScript code generated when events were copied and pasted from the Watch Me Do action in Automator for this example use case:

-- Click the “clock” menu bar item.
delay 0.249747
set timeoutSeconds to 0.500000
set uiScript to "click menu bar item 7 of menu bar 1 of application process \"SystemUIServer\""
my doWithTimeout( uiScript, timeoutSeconds )

-- Open Date & Time Preferences…
delay 0.186764
set timeoutSeconds to 0.500000
set uiScript to "click menu item 6 of menu 1 of menu bar item 7 of menu bar 1 of application process \"SystemUIServer\""
my doWithTimeout( uiScript, timeoutSeconds )

-- Click the “<fill in title>” button.
delay 0.509640
set timeoutSeconds to 0.500000
set uiScript to "click UI Element 1 of window \"Date & Time\" of application process \"System Preferences\""
my doWithTimeout( uiScript, timeoutSeconds )

on doWithTimeout(uiScript, timeoutSeconds)
    set endDate to (current date) + timeoutSeconds
    repeat
        try
            run script "tell application \"System Events\"
" & uiScript & "
end tell"
            exit repeat
        on error errorMessage
            if ((current date) > endDate) then
                error "Can not " & uiScript
            end if
        end try
    end repeat
end doWithTimeout