ECHO behaviour with and without double quotes with hex

Can someone help me to understand behaviour of echo? I am trying the following commands in Ubuntu:

$ echo -e \xaa
xaa
$ echo -e "\xaa"

▒
$

As you can see, with double quotes, while printing hexadecimals, output is some garbage. I know -e can be useful to interpret \n to a newline and other sequences. I just want to understand how echo with -e option handles hexadecimals.

Without the quotes, \x is parsed by the shell to become just x:

$ printf "%s\n" echo -e \xaa
echo
-e
xaa    
$ printf "%s\n" echo -e "\xaa"
echo
-e
\xaa

See man bash, section QUOTING:

   A non-quoted backslash (\) is the escape character.  It  preserves  the
   literal value of the next character that follows, with the exception of
   <newline>.  If a \<newline> pair appears,  and  the  backslash  is  not
   itself  quoted,  the \<newline> is treated as a line continuation (that
   is, it is removed from the input stream and effectively ignored).

Your grep is misleading:

$ man echo | grep -o \xHH
xHH

grep -o prints exactly those characters which matched, indicated grep never received the \.

Unless you run /bin/echo or env echo, the shell's builtin echo will be run. So, if you want to check the documentation, run help echo, or look in man bash. man echo is for /bin/echo:

$ echo --help
--help
$ env echo --help
Usage: echo [SHORT-OPTION]... [STRING]...
  or:  echo LONG-OPTION
Echo the STRING(s) to standard output.

  -n             do not output the trailing newline
  -e             enable interpretation of backslash escapes
  -E             disable interpretation of backslash escapes (default)
      --help     display this help and exit
      --version  output version information and exit

If -e is in effect, the following sequences are recognised:

  \\      backslash
...

See man bash, section SHELL BUITLIN COMMANDS:

  echo interprets the following escape sequences:
  \a     alert (bell)
  \b     backspace
  \c     suppress further output
  \e
  \E     an escape character
  \f     form feed
  \n     new line
  \r     carriage return
  \t     horizontal tab
  \v     vertical tab
  \\     backslash
  \0nnn  the eight-bit character whose value is  the  octal  value
         nnn (zero to three octal digits)
  \xHH   the  eight-bit  character  whose value is the hexadecimal
         value HH (one or two hex digits)