custom filter for Fail2Ban

Yes, you can stop there and fail2ban will do it's thing. You can test this with the fail2ban-regex(1) utility. The summary is

fail2ban-regex [OPTIONS] <LOG> <REGEX> [IGNOREREGEX]

so using the work you have done (Note that I had to add a datetime to the start of your provided log entry otherwise it complains.)

fail2ban-regex "Mar  3 07:36:19 racoon: ERROR: Invalid exchange type 243 from 103.14.62.181[11950]" "racoon: ERROR: Invalid exchange type 243 from <HOST>"

Running tests
=============

Use regex line : racoon: ERROR: Invalid exchange type 243 from <HOST>
Use single line: Mar  3 07:36:19 racoon: ERROR: Invalid exchange ty...

Matched time template MONTH Day Hour:Minute:Second
Got time using template MONTH Day Hour:Minute:Second

Results
=======

Failregex: 1 total
|- #) [# of hits] regular expression
|  1) [1] racoon: ERROR: Invalid exchange type 243 from <HOST>
`-

Ignoreregex: 0 total

Summary
=======

Addresses found:
[1]
    103.14.62.181 (Mon Mar 03 07:36:19 2014)

Date template hits:
2 hit(s): MONTH Day Hour:Minute:Second

Success, the total number of match is 1