What is the meaning of the phrase 'Here be dragons'?

Solution 1:

"Here be Dragons" was a phrase frequently used in the 1700s and earlier by cartographers (map makers) on faraway, uncharted corners of the map. It was meant to warn people away from dangerous areas where sea monsters were believed to exist. It's now used metaphorically to warn people away from unexplored areas or untried actions. There are no actual dragons, but it is still dangerous.

Solution 2:

It's a jocular archaism. I wouldn't like to say it was ever really spoken in earnest, by a speaker who genuinely feared he was entering land so little known it might contain dragons.

It may have got a bit of extra currency from the tendency for old maps to be "embellished" with drawings. Unexplored areas (land, sea, or unknown) would often be decorated with serpents, and perhaps the legend Here be dragons.

Today it's just a jokey way of saying "There are/may be some really scary things here!". In OP's context, really difficult technical problems.

LATER: Having consulted Wikipedia on this one, I should say the only relevant ancient usage known is the Hunt-Lenox Globe (early 1500s), with the Latin HC SVNT DRACONES (hic sunt dracones, here are dragons) on the eastern coast of Asia.

The earliest reference I can find is this from 1892 claiming cartographers from the 1300s used the phrase, but this NGram strongly suggests the myth didn't become widespread until quite recently.

But it's a nice legend, so I'll keep my "perhaps" at the end of the second paragraph.