How can I list files under a URL to a directory on a web/HTTP server?

This is an answer to your question #2.

URLs are simply pointers to resources, which sometimes can be files or directories. It all depends on the actual web server configuration what response is returned to URLs.

If we are restrict discussion to web servers which only serve files from filesystem, then it depends on the webserver configuration if they allow listing directory contents or not.

In nginx webserver, it is the ngx_http_autoindex_module that creates directory listing for visitors. Apache2 has a similar setting.

In general, directory indexes should be disabled, because they provide no value to the application and are a security risk.