How can I compile a library into WebAssembly that can be called from JavaScript code?

You will have a hard time trying to compile a whole library by invoking the Emscripten compiler on individual translation units; the encoder is not entirely contained in just one file. You need to engage the build system to build the full library.

Fortunately for you, much of the work in porting the library to WebAssembly has already been done; there is even a README file with instructions on building a demo program, which uses SDL to decode a file and render it to a <canvas>. You can look how the demo is set up in CMakeLists.txt and modify it to your liking so that the encoder is exported from the library instead of the decoder.

Preferably though, you would create your own CMake project that imports the encoder library and exports the functions you would like to be callable from JavaScript code. Assuming you put the library in the libwebp/ subdirectory, create a CMakeLists.txt with something like this:

project(mywebpenc)
cmake_minimum_required(VERSION 3.22)

add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/libwebp)

add_executable(mywebpenc ${CMAKE_CURRENT_SOURCE_DIR}/mywebpenc.c)
target_include_directories(mywebpenc PRIVATE ${CMAKE_CURRENT_BINARY_DIR})
target_include_directories(mywebpenc PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/libwebp/src)
target_compile_definitions(mywebpenc PUBLIC EMSCRIPTEN)
target_link_libraries(mywebpenc webpencode webpdsp webputils)

set_target_properties(mywebpenc
    PROPERTIES LINK_FLAGS "-s WASM=1 \
        -s EXPORTED_FUNCTIONS='[\"_mywebpenc_encode\"]' -s INVOKE_RUN=0 \
        -s EXPORTED_RUNTIME_METHODS='[\"cwrap\"]'")

Then put a function named above as mywebpenc_encode with the interface you want to export to JS into mywebpenc.c. As a proof-of-concept, I tried a simple stub:

#include <webp/encode.h>

extern size_t mywebpenc_encode(
    const uint8_t* rgb, int width, int height, int stride,
    float quality_factor, uint8_t** output)
{
    return WebPEncodeRGB(rgb, width, height, stride, quality_factor, output);
}

Build the project with emcmake cmake . && make mywebpenc. You will obtain two files, mywebpenc.js and mywebpenc.wasm, which you should be able to interface with like with any other Emscripten library.