Spring multiple authentication methods for different api endpoints

You have two filter chains. Neither of them have an entry point pattern properly configured http.antMatcher. That means they are configured to use /** as their entry point pattern.

For example

    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http
            .authorizeRequests()
                .anyRequest().fullyAuthenticated()

is the same thing as saying:

    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http
            .antMatcher("/**")
            .authorizeRequests()
                .anyRequest().fullyAuthenticated()

What we are saying here is

1. http - the security filter chain
2. http.antMatcher - the entry point to the security filter chain
3. http.authorizeRequests - start of my endpoint access restrictions
4. http.authorizeRequests.antMatchers - list of URLs with specific access

So what you need to do is change your @Order(1) filter chain to narrow down the pattern. For example: http.antMatcher("/api/transaction/**")

Your configuration will now look like

    @Configuration
    @Order(1)
    public static class ApiWebSecurityConfig extends WebSecurityConfigurerAdapter{

        @Override
        protected void configure(HttpSecurity http) throws Exception {
            http
                .antMatcher("/api/transaction/**") //customized entry point
                .authorizeRequests()
                .antMatchers("/api/transaction/testf").authenticated().and()
                .x509()
                .subjectPrincipalRegex("CN=(.*?)(?:,|$)")
                .userDetailsService(new X509UserDetailsService())
                ;
        }
    }

    @Configuration
    @Order(2)
    public static class ApiTokenSecurityConfig extends WebSecurityConfigurerAdapter{

        @Override
        protected void configure(HttpSecurity http) throws Exception {
            http
                .antMatcher("/**") //this is default
                .authorizeRequests()
                .antMatchers("/oauth/token", "/api/dealer/login").permitAll()
                .and()
                .authorizeRequests()
                .anyRequest()
                .authenticated()
                ;
        }

With your existing configuration the filter chain named ApiWebSecurityConfig will trap all calls. The other filter chain, ApiTokenSecurityConfig, is never used.

You can see another description in this answer

SpringSecurity: Make RESTful API basic-auth authentication possible via only a single endpoint