How do I use Messages with Google Hangouts on MacOS Mojave?

You can't. Apple removed the option to log in to Jabber with 10.14, Messages is iMessage-only now.

I have attempted to move all the frameworks over from High Sierra. I successfully got the messages application from High Sierra to open on Mojave and it was able to send and receive messages. But I was unable to add a Jabber account due to an error connecting. I think this might require another framework or something else, but it probably works with the System Preferences account setup plane so changing it may cause other problems. I will look more into.

So as the others answers says I don't think its possible.

For those wondering I copied over the following Frameworks

• MessakesKit.franework
• PhotoLibrary.framework
• SocialUI.framework
• SocialAppsCore.framework
• Ofc the High Sierra Messages.Application

I see this has been already answered, but I'd like to chip in and recommend an alternative messenger: https://adium.im/

Adium is a free and open source multiprotocol messenger for all operating systems(including Mac OS).

Google-Hangouts (aka: GH, aka: classic Hangouts, previously aka: Google-Talk, aka: GT) support was dropped from Mojave & Catalina, so "Messages" app cannot use that web-service (aka: network) anymore normally, and a complete solution is not-found yet. (aka=also-known-as)

• But user "JBis" has shown at-least some partial way to copy specific files, settings, etc from older macOS to Mojave/Catalina. Though its not a fully working solution, hopefully other users can collaborate & improve & find a complete solution.
• at this moment, i would suggest to use an alternative & trustworthy app that can connect with Google-Hangouts/GH/GT network, which is: Thunderbird app, it can connect with Google-Hangouts IM (instant-messaging) network (aka: web-service), or use your own choice of other app. (See detail instruction shown below for more info, if you have time)
• Another option is to use a HyperVisor¹ (aka: Virtualization¹) app (f.e.: VirtualBox) to run another/second instance of a macOS (f.e.: High-Sierra as a guest macOS), inside a guest virtual Mac computer/machine in the host macOS Mojave/Catalina/BigSur, and use the "Messages.app" with GH network support/feature & other apps inside guest macOS, as specific feature necessary for user is missing(aka: not-present) in host macOS. (See detail instruction & goto bottom side for more info, if you have time)
  
  

EXTRA INFO , PLEASE SKIP READING BELOW , IF YOU DON'T HAVE TIME:

THUNDERBIRD:
Apple has removed many other supports from many other Apple apps, and if you were depended on those, then
Thunderbird ^{1, 2} (TB) is a better alternative choice for some of those missing options.
TB (since v?) has began to include GPG ¹ tool, and from TB v78 (for Windows/Linux/Unix, & different version ? on macOS) it transitioned to use builtin OpenPGP ^{1, 2, 3}, so separate GPG/PGP tool not needed anymore), so user/you can send+receive digitally signed and/or encrypted emails with it, for free.

• At this moment in time (Sept-8, 2020):
  simple & trustworthy, and more uptodate, and more devs supported, and more user supported, etc based
  SOLUTION (for Catalina & Mojave users) is, use THUNDERBIRD.
  TB has builtin¹ ability/feat to connect with Google-Talk ¹/Google-Hangouts¹ text messaging web-service (aka: network). But it supports only basic features: text chat. TB also supports instant messaging and chat using IRC, XMPP, Twitter, (Yahoo-Messenger, Skype, Facebook, etc¹), it can also function as News Client, Feed Reader, etc . Uses XULRunner framework . TB is created with C, C++, JavaScript, CSS, Rust, XUL, XBL. License: MPL2.0.
  • If you want to use more (advanced) features from Google-Hangouts web-service (aka: GH network, previously aka: GT network) then do this : TB includes basic web-browser (aka: lightweight web-browser), so a web-browser tab can be opened-up inside TB, and used to connect with Google-Hangouts web-service (aka: GH network) website https://hangouts.google.com/ directly, to send/receive IM messages with your any contacts-list/address-book.
    Inside TB web-browser tab, you may need to stop annoying & privacy-rights violating ADs/deceptions, So you may need to get AdBlock type of addon's XPI file from Mozilla Firefox addons website, and then load that XPI file inside TB if the addon is not available in TB addons website: get uBlock Origin addon, (imo, its better than AdBlock addon in many aspects) . Detail info on installation is here: HowToUse AdBlock TypeOfAddon To Block ADs/etc In Basic WebBrowser and HowToUse Basic WebServices in Basic WebBrowser. ( PLEASE UPVOTE THESE RELATED LINKS TOO IF YOU FIND THEM HELPFUL . THANKS IN ADVANCE IF YOU'RE UPVOTING ).
  • you may need to Allow "Less Secure" Access in your Google Account settings https://support.google.com/mail/answer/78754 . By the way, to Google using other company's software is "less secure" even if other company/software uses stronger & better encryption+security (to protect people/user's private data), and even if other company/software does-not SPY on user's data the way (hypocritical) Google does.
    
    

GH = Google-Hangouts (previously aka: Google-Talk, aka: GTalk, aka: GT, etc) . IM = Instant-Messaging (aka: Chat) . aka = also known as . Ggl = Google . f.e.=for-example . i.e.=in-example . e.g.=example-goes . Lic = License (aka: Permissions & Restrictions).

GH/GT/XMPP/etc SERVICE/NETWORK:

• GH network (aka: GH web-service) uses Jabber aka XMPP protocol for (text) IM, contact-list, etc, and uses Jingle (libjingle, a proprietary protocol) for VoIP(Voice-overIP)/Video-over-IP, etc.
  • User/you can use/enter these config info ^{1, 2} (if you're asked-for or prompted):
    • connect server: talk.google.com
    • protocol: XMPP
    • username: [email protected] or [email protected] (or [email protected])
    • password: your-Gmail-Or-Google-Accnt-Password
    • connection port: 5222 (also see "firewall(outbound)" info below)
    • connection security: select SSL/TLS (or Require Encryption type of option).
    • local alias: usually, leave this field blank.
    • domain: specify just "gmail.com" (w/o double-quote symbols) if you're using Google account for-example "[email protected]" which has Google domain "gmail.com" , but if you've registered for your own domain "example.com" based email-address "[email protected]" with Google, then specify your domain: "example.com".
    • screen Name: your Google account username portion (without any @ symbol or the domain-name after it).
    • firewall(inbound): no input/inbound port opening is required.
    • firewall(outbound):
      • for XMPP clients allow TCP 5222, TCP 5223, TCP 5224.
      • for Android phones: TCP 5228 & 5229.
      • for XMPP federation: TCP 5269.
      • for original GH client allow UDP from 19302 through 19309, TCP from 19305 thru 19309, TCP 443 (and UDP 443).
      • you may whitelist these IP-address ranges for GT/GH/GM, etc: GT/GH services use 172.253.117.*/24, 74.125.136.*/24, 172.217.0.*/24, 108.177.*.*/16, 173.194.*.*/16, 172.22.0.1/?, etc. Google-Meet media components additionally use these: IPv4: 74.125.250.0/24, IPv6: 2001:4860:4864:5::0/64. Note: these IP ranges should be same in most areas but in some areas it will be different based on your client-software's (and Ggl server's) location.
      
      

OTHER APPS THAT CAN CONNECT WITH GH/GT/XMPP SERVICE/NETWORK:

I'm now using macOS Catalina (and also use Snow Leopard), so after upgrade from much below version (macOS Lion),
i was also amazed to find out that, support for GH/GT was dropped !!

• Apple has changed many many other factors & apps in macOS 10.15.x Catalina, (this type of major changes began with macOS 10.14.x Mojave).
• in Catalina, Apps MUST need to be 64-bit only, if any initial starting components are 32-bit, then it won't start/work/run.
  So Mojave is the last version where both 32bit & 64bit mac app can work/run normally.
• by the way, in Catalina there are non-normal (aka manual) ways to load support for 32bit app selectively, but its not easy (or safe) obviously.
• but WINE-project based solutions are easier, to run 32bit (and some 64bit) WINDOWS apps in macOS : Big Sur (64bit-only), Catalina (64bit-only), Mojave(32bit-&-64bit) & before . Also see this answer for loading Wine on 64bit-only macOS, by using MacPorts (MacPorts is written in TCL & C. Lic:BSD). WINE is written in C, has Lic: LGPLv2.1+.
• few months back, i've seen a website-page, a user has obtained Hangouts files, either after installing (old) Hangouts in Windows 1st, Or used (old) Hangouts from old computer. Copied those GH files in macOS & used WINE app to run it in (Mojave) macOS.
  • as far as i'm aware, Wine-project has began to support running 32bit & 64bit Windows apps in 64bit-only macOS (Catalina, Big Sur, etc).

And as far as i remember, there was a native (32bit/64bit) Hangouts app for macOS. And there was also a native app for Windows (which i'm 100% sure).

• My web-search did show website that is sharing native Hangouts app for macOS (and Windows) , but Google removed + discontinued linked files, so NOT downloadable anymore from a trustworthy or the source website.
• if that native Hangouts (32bit version) app for macOS can be obtained, then that should've worked in Mojave 32bit+64bit macOS, but that will not work on (64bit-only) Catalina.
  • Google-Talk v1.0.0.104 (Windows) here https://dl.google.com/googletalk/googletalk-setup.exe, v 1,606,064 bytes, MD5:8260031b32d9101c9f222161a87ff2f1, SHA1:83b5614bcbe19d992a617e059943c297ddffc848, file-date: Mar-24, 2008.
    To use Windows app in macOS use WINE-project's macOS app.
    Note: by the end of 2017 or sometime closer, GT services were transitioned gradually into GH services.
  • Google-Hangouts (GH) chrome app is here, and GH chrome extension is here, also see below "Chromium" section.
    Note: by the end of 2019 or around that time, GH services began to be transitioned gradually into GM(Google-Meet) services, (as of Sept-2020 it has not completed yet).
• Android & iOS has Google-Hangouts (mobile) app in AppStore/PlayStore. It allows text, voice, video chat, group chat/conference, etc.
  • for video/audio chat, you may use Google-Duo¹ (or Google-Meet) from any desktop/mobile web-browser. Google-Duo (mobile) app is free & in iTuneStore/PlayStore. It allows video, audio, text chat, group chat/conference (upto 12 users), etc.
• Google-Hangouts (standalone/desktop app) version ? hash-code ? .
  Google-Hangouts (desktop) app may be still available in macOS AppStore for older than macOS Mojave (that is, for macOS High-Sierra), or may be available from app caching sites.
  • Warning: AppStore/PlayStore app caching/sharing 3rd-party websites may be sharing altered/modified apps, Or may be sharing apps that has been removed by Apple/Google for violating Privacy-Rights or other Human-Rights Or removed for containing malware (aka: virus), that is, they have SPYING codes/components or malware codes, etc, And it is also possible that some app was exposing Apple/Google/Microsoft or Govt SPYING or Human-Rights violation, etc, so the app was removed/censored/banned completely or in some locality . If you can atleast obtain hash/checksum (integrity) code of original file from source/authentic site, then getting files from caching site (or torrent) is ok, provided you understand the risks mentioned few sentences earlier.
    
    

Other app options mentioned in this page by other users have been considered+used by me, but i have chosen TB,
and, you are or any user is always free to choose whatever you/user prefer.

• By the way, Adium last version 1.5.10.4 is old (3yrs), not uptodate, various security patches are not applied. Can allow text based chat. It supports many IM/chat network/service connections . Adium is written in objC, Cocoa. Uses libpurple. Licensed under GPL.
  
  

Regular/free google account holders can use Google-Hangouts (previously aka: Google-Talk, aka: GTalk) web-service (aka: GH IM network, aka: GH chat network) directly from Google's Hangouts website:
https://hangouts.google.com/
you have to use a web-browser software to access that, so use: Safari, Firefox, Chromium, etc, Or, use a basic web-browser tab inside Thunderbird (TB) to access above GH web-service directly, and the process to use TB based web-browser tab is described in detail in top side section. Another choice is, use Chromium web-browser (see below Chromium section) to turn the URL https://hangouts.google.com/ into a web-app, so you can launch it instantly & access directly.

• By the way, i do NOT RECOMMEND using Google-CHROME ( aka: Ch / GC / Chrome / GCh / gCh / gCH etc) , because it acts like a SPYWARE (aka: anti-Privacy-Rights software), it loads various auto startups in various deep levels of OS without fully aware consent of computer's owner, etc, etc various issues. And it has many non-open-source components.
  If you must have to use something similar, then use "Chromium" (aka: CM / CH / Google-Chromium ) , it is open-source web-browser , it was initially developed by Google LLC, based on WebKit . Chrome uses "Blink" rendering engine and V8 (JavaScript engine). "Blink" is based on modified "WebKit" engine . WebKit was made open-source(LGPL) by Apple Inc, and then Adobe Systems, KDE, Igalia, & others joined . Later Microsoft, Igalia, Yandex, & others joined to develop CHROMIUM / CM web-browser . Later Facebook, Opera Software, Adobe Systems, Intel, IBM, Samsung, & others joined to develop Blink . Many many web-browsers are actually based on modified/customized version of this CM web-browser or Blink engine from CM. CM is written in C, C++, JavaScript, and different portion uses different licenses: 3-clause BSD, MIT, LGPL, MS-PL, etc, so CM is distributed with tri-licensed MPL+GPL+LGPL.
  • and be aware that many devs/companies/corps have tendency to apply new/experimental feats (aka: "features", etc) that are NOT free of Privacy-Rights violating activities, (in another word, those feats can be abused to SPY to collect private data (unconstitutionally) from user's device, & user's private data in server or private data in transit, etc . And also be aware that many devs also have tendency to either HIDE, or NOT report & not apply "FIX"/solution that will remove/stop SPYING or stop private data collection).
• So "Chrome" & "Chromium" are not same software product.
  • Google uses codes from "Chromium" (CM) web-browser, and also adds their own proprietary (and non-OpenSource) codes, to build "Google-Chrome" (gCH).
• Most of the extensions/plugins for "Chrome" will also work with "Chromium".
• Various web-sites/web-services (for example: https://hangouts.google.com/ site) can be used as an app via Chromium.
• So you will be able to use "Google-Hangouts", etc Chrome extension/plugin (or Chrome app) with "Chromium" web-browser.
• Chromium (web-browser), can be obtained from here:
  • https://SourceForge.net/projects/osxportableapps/files/Chromium/
  • https://download-chromium.appspot.com/
• Then obtain "Google-Hangouts":
  • either, chrome extension: https://chrome.google.com/webstore/detail/google-hangouts/nckgahadagoaajjgafhacjanaoiihapd
  • or, chrome app: https://chrome.google.com/webstore/detail/hangouts/knipolnnllmklapflnccelgolnpehhpl
  • follow instruction here.
    
    

Parrot uses Hangover, a swift based library (which is based on hangups-library), to connect with GH network, etc. Lic: MPL2.0.

Monal ¹ can use XMPP networks (so Google-Talk/GT/GH is supported), and Jingle VoIP, etc, and it is available from Github, iTunes-AppStore, macOS-AppStore. Its written with objC, C++, and licensed under BSD-Lic.

YakYak is open-source & can connect with Google-Hangouts web-service (aka: GH network). Allows text based chat in the app, and allows video/voice chat via Electron which uses Chromium core, etc.

• YakYak is written in CoffeeScript (nodejs) based on hangupsjs (aka: hangups) using trifl on top of Electron(aka: Atom-Shell). Licensed under MIT-Lic. Please also see below sub-section for warning about Electron.
  
  

Google-Messages for Desktop ¹ (GMfD) (aka: Android-Messages for Desktop, AMfD) is an unofficial (aka 3rdparty) open-source project based solution which uses a wrapper around the official https://messages.google.com/ web client. The GMfD is not-affiliated & not-endorsed by Google or Google-Messages¹. Its written with JavaScript, objC, & appears to be licensed under BSD or MIT Lic . GMfD uses risky Electron.

• Be aware of these: Electron ^{1, 2} framework (EJS) previously known as Atom-Shell, developed by GitHub . EJS uses+integrates various components, & each components have many many known bugs/vulnerabilities ¹ so EJS may not be very secure . EJS uses Chromium web-browser core . EJS based apps may also have very significant overhead (aka: excessive memory, CPU, etc usage) ^{1, 2} due to their Chromium core (aka: Blink) dependency, when compared with similar native apps . EJS is written in C++, JavaScript, Objective-C++, Python and Objective-C, and Licensed under MIT-Lic.
  

You can use Google-Meet (GM) https://meet.google.com/ web-service URL from any modern web-browser, it will allow to use your free Google accounts (since Mar/Apr 2020, Google has allowed free usage, until July 2020, after that free-users will have reduced features & reduced minutes, restrictions, etc). GM allows video, voice, text chat, group chat/conference, share desktop, etc. It was previously known as "Hangouts Meet". You can contact with your any Google-Hangouts contacts-list/address-book. Android & iOS has Google-Meet app in AppStore/PlayStore. There is no native GM app for macOS or Windows. You can create a web-client app for the web-service URL https://meet.google.com/ by using Chromium (or Chrome) web-browser.

If your Google account is a type of G-Suite account (business/organizaion type, etc type of google account),
then get 64-bit Google-Chat app from here: Google-Chat (64bit) for G-Suite
Google-Chat is NOT for regular/free Google account users/holders.
Google-Hangouts-Chat (GHC) available here, also iTunes-AppStore(iOS).
GHC is for organization, etc type of users, (not for free Google account users).

If you are a software developer, then look into below option(s):
these can/may allow to use GT/GH network when compiled in macOS, or when compiled after necessary code modifications:

• ChatSecure ¹ can connect with XMPP networks, etc from iOS(iPhone/iPad),etc, it supports connection via proxy, etc, and its available in iTunes-AppStore(iOS). Its written in objC, Swift, etc, and licensed under GPLv3+. To use in macOS, source code modifications & compilation is needed.
  
  

HYPERVISOR / VIRTUALIZATION BASED SOLUTION:

These HyperVisor¹ app(s) can be used to run a guest macOS (in our case, here, it will be an older High-Sierra macOS) under your current host macOS (Mojave/Catalina/BigSur), and you can use both of your (host/current and guest/virtual) macOS together at the same time:

• VirtualBox (VB) is a virtualization (aka: hypervisor) software that can create (guest) virtual machine/computer. Obtain¹ & install it. Create a virtual machine inside VB with 2GB RAM, 30GB hard-disk allotted for it, and choose machine type macOS based. Get High-Sierra macOS installer app or dmg file from Apple ^{1, 2, 3}, convert that into a DVD disc image "ISO" type of file¹, & attach the ISO file with a virtual dvd-drive inside a virtual Mac machine inside the VirtualBox app. Start virtual guest Mac machine (inside the VB app), install the older macOS High-Sierra, and configure it, now you can use "Messages.app" inside Virtual Mac computer (with High-Sierra macOS) which can connect with Google-Talk/Google-Hangouts.
  • VB is open-source and written in C, C++, x86 Assembly, Python, and its licensed under GNU GPLv2, CDDL¹, PUEL¹.
  • VM = Virtual Machine . VB = VirtualBox . GVM = Guest VM . HM = Host Machine/Computer . HS = High-Sierra macOS.
• xhyve is an open-source lightweight OS X virtualization software based on bhyve. Its written in C, C++, and licensed under BSD-2-Clause Lic.
• you may also use other virtualization software: Parallels Desktop for Mac (its proprietary and not-open-sourced) , Bochs (its open-source) , etc.

If you do not have DMG or app installer for HS macOS, then first goto notpeter's github page:

• get hash/checksum code-numbers of High-Sierra (HS) version 10.13.6 or another 10.13.x
  • HS 10.13.6 (17G65) "InstallESD.dmg" file has hash/checksum code-number SHA1:69159caf25666ea1c5d466e158e075d947f6a9ee, (& Size: ?), and the file "BaseSystem.dmg" inside that, has this hash/checksum code-number: SHA1:6b7522d7f78b93441ecd7dee73430af9b07b3744, (& Size: ?).
• look/search for a torrent that has HS dmg file, investigate 1st before download if the included dmg's hash-code matches any one of those hash/checksum code-numbers shown above, or else if you find-out after downloading that downloaded dmg file's hash code did not-match then your network/data bandwidth will be wasted . Do Not Run Downloaded "App"/"DMG"-File Without Calculating/Checking It's Hash Code First . After download you MUST have to calculate+findOut dmg file's hash/checksum code. If calculated hash code does not match with SHA1 hash-code(s) shown above, then downloaded file is not original file, they were modified & So such file might contain fishy/malware apps/code inside it, So you MUST move such into a folder name "DeleteThese" or "TrashThese", & search again.
  • instead of finding torrent with specific HS dmg from an unknown person , you can also request a friend that has HS to share the dmg file, but as the dmg file's size is huge, sharing such huge file is not easy. And request your friend to first "zip" the dmg file with a password . Your friend must create a torrent file with that (password-protected) "zip" file, and your friend must also mention inside that torrent that it is a password-protected "zip" file (for private use, only with a specific friend), during creation of that torrent file . Then unknown people will not obtain it (as they do not have the password) so your friend did not share it unfairly & did not share with mass users, and as you are both owner of real Mac (hardware) computer, so its not an unfair sharing . It will take long time to transfer such huge file, if connection is not-speedy or has less bandwidth.
  • If you do not want to obtain file in above mentioned way, then you have to read a lot, search & go thru various web-pages in MacRumors forum website and in this apple.StackExchange.com website, where various users have collaborated & shared info & shown how a user can obtain files from Apple servers or Apple computers, & then how a user can extract/convert that to an ISO file. Few of such linked-pages are now shown in above VB paragraph.
• extract "BaseSystem.dmg" & convert it into ISO file, instruction is here.
• use the process mentioned in VB section above, to use the ISO file to load the HS macOS in a virtual Mac machine, then you can finally use the GH/GT network via "Messages.app" from High-Sierra macOS.
• spend time to fine-tune configurations inside the HS(High-Sierra) guest macOS : use configurations mentioned here ^{1, 2} to speed-up macOS, etc.
  • Basically do this: disable all apps & services inside the (High-Sierra) guest macOS that are already present in host macOS (Mojave/Catalina/BigSur), when guest macOS will use lesser (background) services & apps, etc, then it will use lesser RAM & lesser CPU, so it will run faster . Reboot VM, & while VB is running, take Snapshot of Mac-VM, so that you can go back to Snapshot point of time+state where it was running faster & was using lesser computing resources . Close VB, & make backup copy of all files related to Mac-VM . Later, if you find that snapshot is not-working, then you can use the backup files, to overwrite in Mac-VM folder, to go back to fine-tuned stage.
    
    

WARNING:
And also be aware of this : Google-Hangouts network (web-service) does not use end-to-end (E2E) encrypted packets , only part of the communication data is actually encrypted, i.e: data transit from end-user to google central/federated server(s).