How to set up an OAuth2 Authentication Provider with AWS API Gateway?
Solution 1:
(1) Do you want to implement OAuth 2.0 endpoints (authorization endpoint and token endpoint) on API Gateway? In other words, do you really want to implement an OAuth 2.0 server (RFC 6749) on API Gateway?
(2) Or, do you want to protect your Web APIs implemented on API Gateway by OAuth 2.0 access token?
These two are completely different things.
Because it seems you wanted to select OAuth 2.0 instead of AWS-IAM, I guess what you wanted to do is (2). If so, you can find an example here: Amazon API Gateway + AWS Lambda + OAuth
Updated on 2016-Apr-6
On Feb 11, 2016, a blog entry of AWS Compute Blog, "Introducing custom authorizers in Amazon API Gateway", announced that Custom Authorizer had been introduced into Amazon API Gateway. Thanks to this mechanism, an API built on Amazon API Gateway can delegate validation of a Bearer token (such as an OAuth or SAML token) presented by a client application to an external authorizer.
How to protect APIs built on Amazon API Gateway by OAuth access tokens utilizing the new mechanism, Custom Authorier, is described in "Amazon APi Gateway Custom Authorizer + OAuth".
FYI:
OAuth 2.0 is a mechanism for authorization, not authentication. But people often use OAuth 2.0 for authentication and there are many software libraries and services using OAuth 2.0 for authentication.
Cognito (Identity) is a solution related to authentication, not authorization.
Custom in Cognito is a place to specify OpenID Connect Providers. OpenID Connect is a solution for authentication. What makes things complicated is "OpenID Connect is built on top of OAuth 2.0". See OpenID Connect site for details.
Solution 2:
AWS API Gateway supports Amazon Cognito OAuth2 Scopes now. You can create Amazon Cognito user pool authoriser and configure it as your Authorisation method in API Gateway. In order to make use of OAuth scopes, you need to configure a resource server and custom scopes with your Cognito userpool. You can configure multiple app clients in Cognito userpool with different scopes or request different scopes from your application code while authenticating users from your Userpool. Based on the scope received in the access token from Cognito, API Gateway will allow/deny the caller of your APIs. A step-by-step guide to implement this feature can be found here