It sounds very much as though the Original Poster has a vocalised /l/. What this means is that when /l/ occurs at the end of a syllable, it is realised as as a vowel, usually [ö] or [ʊ].

This is becoming more common in Southern Standard British English (aka RP) and in other regional varieties too. It is especially common in London regional English.

Although this is now very common, it used to be sneered at by RP speakers. However, it has long been the case that many RP speakers regularly use this allophone of /l/ after bilabial consonants, for example in the final syllable of the word people.

In short, this is a common allophone of /l/ often found in syllable codas. The Original Poster has nothing to worry about!