"-Based" vs. " Based"

'Machine learning' is modified by a two-word adjective. An adjective consisting of two or more words is called a compound adjective, and the individual components have to be joined with a hyphen.

'Transform-based' i.e. option #1 is the correct choice.

First of all, just so we are on the same page, you are doing machine learning that is based on a chirplet transformation, which is adaptive, right? I am asking this because for spelling (and understanding your actual writing) it is crucial to see which parts of your construct actually specify which other parts. Another way to see this is to bracket everything as much as possible. I here understand the correct bracketing to be:

{[Adaptive (Chirplet Transform)] Based} Machine Learning

So, the chirplet transform is adaptive, the entire thing describes based which in turn describes machine learning. Now, a simple rule of thumb is this: If any bracket closes before the end of the subject or object, everything within that bracket has to be hyphenated – unless it’s an adverb ending on ly or a proper name. Here, the curly braces close before Machine Learning, and thus everything inside has to be hyphenated:

Adaptive-Chirplet-Transform-Based Machine Learning

That is of course very difficult to read (and not just because of the hyphens), so I suggest to change it to:

P300 Brainwave Classification with Machine Learning based on the Adaptive Chirplet Transform