Solution 1:

The syntax for the renderHook call is not quite right in your test.

Note the curly brackets, you should return useInitialMount() from renderHook callback, not just call it inside it (hence why you get undefined).

test('should return true in the very first render and false in the next renders', () => {
  const { result } = renderHook(() => useInitialMount());
  expect(result.current).toBe(true);
});

Edit: To clarify, the difference here is that:

Calling () => { useInitialMount(); }); returns undefined, there are no return statements so the function will return undefined by default.

But calling () => useInitialMount() (which is short syntax for () => { return useInitialMount(); }) will return the value of calling the hook.

Reference: Arrow Functions > Functions body.