Starting a sentence with "Starting with"

When you want to summarize a list of items, can you use "Starting with" at the beginning of a sentence in order to explain the first item?

For example, is the following grammatically correct?

I love all kind of animals. Starting with kittens and puppies, the cutest animals in the world. I'm rather fond of little ducklings as well.

To further clarify, I would think this is grammatically incorrect and a comma is required to connect "starting with" with the first sentence.

Many people would object to your second sentence as not having a verb. They might accept

I love all kinds of animals. Starting with kittens and puppies, I particularly like baby mammals. I'm rather fond of little ducklings as well.

Your example is not grammatical as your second "sentence" does not form a complete thought: you have the participial phrase "starting with kittens and puppies" but do not indicate who or what is starting, then "the cutest animals in the world," a noun phrase again with no action suggested.

I could rephrase as

I love all kind of animals, starting with kittens and puppies— the cutest animals in the world. I'm rather fond of little ducklings as well.

I love all kind of animals. To start [with], I love kittens and puppies, the cutest animals in the world. I'm rather fond of little ducklings as well.

I love all kind of animals. First of all, I love kittens and puppies, the cutest animals in the world. I'm rather fond of little ducklings as well.

Yes, you can start a sentence with 'starting with' - it is a perfectly grammtical construction (it is a sentence modifier).

As has been noted, whereever you put the phrase that begins with 'starting with' or any phrase for that matter, it is only grammatical as part of a complete sentence. There is nothing special about 'starting with' that makes it grammatical or not in your example sentence.