Why can't I access File Sharing when Open Directory is enabled in macOS Mojave?

Here's the procedure I have followed:

  1. Install a fresh copy of macOS Mojave to an APFS volume

  2. Perform initial OS configuration and create 'admin' user at first launch. Assign a static IP from 192.168.168.0/24 private network. Use a DNS server located in private network. Ensure IP resolves to a FQDN ('test.mydomain.com') and vice versa.

  3. Download macOS Server application (5.7) from App Store
  4. Open macOS Server application
  5. Create a new Open Directory domain with default options
  6. Create a new user 'testuser' to Local Network Directory
  7. Create a new group 'testgroup' to Local Network Directory
  8. Assign newly created 'testuser' to 'testgroup'
  9. Open System Preferences application
  10. Open Sharing preferences
  11. Enable File Sharing
  12. Create a Shared Folder 'myshare' and assign 'testgroup' and 'admin' Read&Write access to it
  13. Select 'myshare' and click Options button to ensure SMB sharing is enabled for it

  14. Attempt to connect to the file server from a client computer within the same subnet via smb://test.mydomain.com/myshare or alternatively smb://192.168.168.X/myshare either using 'admin' or 'testuser' credentials

In the last step connection fails for both 'admin' and 'testuser' accounts. If I turn Open Directory to Off, I can connect with 'admin' user. Restarts in any phase of the procedure make no difference.

Why can't I access SMB when Open Directory is enabled?

Here are the opendirectoryd log entries from creating the OD master (step 5): https://pastebin.com/uQm8b8NM

Here are the opendirectoryd and smbd log entries from login attempt (step 14): https://pastebin.com/U2RS3LYC & https://pastebin.com/7bFNfd8V


The issue is the ACLs are not set up in the local directory for SMB and AFP. These used to be created in the older Server apps that had File Sharing in them. I've written an AppleScript that takes care of all this. It creates the appropriate ACL groups in the directory (/Local/Default/Groups/com.apple.access_smb and com.apple.access_afp), then adds all the users to it. The script is below. I threw it together today trying to solve this very issue. Hopefully it will help others.

-- Script to sort out ACLs for file sharing
set savedDelimiters to AppleScript's text item delimiters

display alert "Setup File Sharing ACLs" message "This script will set up the appropriate ACLs in the local directory to allow users to connect to file sharing on a macOS 10.14 server with OpenDirectory.

WARNING: Changes will be made to your local directory. Administrator privileges are required (you will be prompted for a password).

USE AT YOUR OWN RISK!

Set for all users, or only a single user?" buttons {"Cancel", "All Users", "Single User"} default button "Single User" cancel button "Cancel"

if button returned of result = "All Users" then
    set progress description to "Loading User List..."
    -- Load all directory users from the server
    -- (identified by UserShell value of '/bin/bash'; most likely to be normal users)
    -- The delimiter is horrible, but it's the only way to do it
    set delimiter to tab & tab & "UserShell = (" & return & "    \"/bin/bash\"" & return & ")"
    set AppleScript's text item delimiters to {delimiter & return, delimiter}
    set users to every text item of (do shell script "dscl /LDAPv3/127.0.0.1 search /Users UserShell \"/bin/bash\"")
else if button returned of result = "Single User" then
    repeat
        set username to the text returned of (display dialog "Enter Username:" default answer "" with icon note)
        if username is "" then
            display alert "Please enter username, or click cancel to end"
        else
            exit repeat
        end if
    end repeat
    -- Add blank element to end, as this happens with output from dscl above
    set users to {username, ""}
end if

-- Create the SMB & AFP ACL groups if necessary (this may be the first user)
createACLGroup("afp", 250)
createACLGroup("smb", 110)
-- Go through all the users now
set total to (length of users) - 1
set progress total steps to total
set progress description to "Adding Users to ACLs..."
set current to 0
repeat with idx from 1 to total
    -- Need to use indexed repeat because of issue with missing username in list from dscl
    set username to item idx of users
    try
        set progress completed steps to current
        set progress additional description to "User " & (current + 1) & " of " & total & " (" & username & ")"
        -- Now, check to see if the user is already in the file sharing lists
        set AppleScript's text item delimiters to {" "} -- Split words, not letters!
        set currList to every text item of (do shell script "dscl /Local/Default read Groups/com.apple.access_smb GroupMembership")
        if username is in currList and length of users is 1 then
            -- Only alert if in single user mode
            display alert "Username already set up"
        else
            -- Go ahead and set it up
            -- Firstly, get the user's GeneratedUID from the LDAP directory
            set isError to false
            try
                set guid to second item of (every text item of (do shell script "dscl /LDAPv3/127.0.0.1 read Users/" & username & " GeneratedUID"))
            on error
                display alert "Error" message "User " & username & " is not a directory user"
                set isError to true
            end try
            if not isError then
                -- Add the user to the group
                addUserToACL("afp", username, guid)
                addUserToACL("smb", username, guid)
            end if
        end if
        set current to current + 1
    on error
        -- Likely an empty username from the delimiters tokenising the list from dscl
    end try
end repeat
set current to total
display alert "Process completed!"

set AppleScript's text item delimiters to savedDelimiters

on createACLGroup(acltype, groupid)
    try
        do shell script "dscl /Local/Default read Groups/com.apple.access_smb"
    on error
        -- Doesn't exist, so we need to create it!
        do shell script "dscl /Local/Default create Groups/com.apple.access_" & acltype with administrator privileges
        do shell script "dscl /Local/Default create Groups/com.apple.access_" & acltype & " RealName \"" & changeCaseOfText(acltype, "upper") & " ACL\"" with administrator privileges
        do shell script "dscl /Local/Default create Groups/com.apple.access_" & acltype & " PrimaryGroupID " & groupid with administrator privileges
    end try
end createACLGroup

on addUserToACL(acltype, username, guid)
    do shell script "dscl /Local/Default append Groups/com.apple.access_" & acltype & "  GroupMembership " & username with administrator privileges
    do shell script "dscl /Local/Default append Groups/com.apple.access_" & acltype & " GroupMembers " & guid with administrator privileges
end addUserToACL

on changeCaseOfText(theText, theCaseToSwitchTo)
    if theCaseToSwitchTo contains "lower" then
        set theComparisonCharacters to "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
        set theSourceCharacters to "abcdefghijklmnopqrstuvwxyz"
    else if theCaseToSwitchTo contains "upper" then
        set theComparisonCharacters to "abcdefghijklmnopqrstuvwxyz"
        set theSourceCharacters to "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
    else
        return theText
    end if
    set theAlteredText to ""
    repeat with aCharacter in theText
        set theOffset to offset of aCharacter in theComparisonCharacters
        if theOffset is not 0 then
            set theAlteredText to (theAlteredText & character theOffset of theSourceCharacters) as string
        else
            set theAlteredText to (theAlteredText & aCharacter) as string
        end if
    end repeat
    return theAlteredText
end changeCaseOfText