Do the words with non-palatalized pronunciation of g/c ("get", "give") always have a Germanic origin?

"Always" is a word that is often difficult to maintain in questions of this sort. Certainly most instances of [ge]/[gi] written ge/gi/gy are from Germanic, but not all, as you can easily see by looking through the list you linked to. (By the way, Old English had its own palatalisation, which gives us yet and if, which once began with 'g').

Similarly, there probably are examples of ce, ci pronounced [ke], [ki], but I can't think of any. They're certainly very rare, because k is normally used instead. Words with this sequence are mostly from Norse, because in Old English the [k] palatalised to [tʃ], now written ch. So church, Old English cirice (cf Northern and Scottish dialect kirk, from Norse).

No. Words with non-palatalized <c> (representing /k/) before <i>, <e> or <y> are rare, and there is no regular source of them. As Colin Fine mentions, the letter <k> is generally used in these contexts, even for Germanic words.

There are more words with non-palatalized <g> (representing /g/) before <i>, <e> or <y>. However, only some of these are of Germanic origin; there are others from other sources, including, surprisingly, Latin and French.

Non-palatalized <c>, and <k>, in words from Latin and Greek

(While you set aside this class of words in your question, I wanted to write a section about them just for the sake of completeness.) I can think of two words where this occurs, and it is optional in both. The first is the well-known example of Celt(ic). The second is in certain compound words containing "-cephal-" (such as encephalitis), where for some weird reason some people use /k/.

A few more words from Greek have /k/ spelled "k" in this context, such as kinematic and keratin.

For British English speakers, another word with <c> as /k/ before <e> (and obligatorily, not optionally in modern pronunciation) is sceptic.

Non-palatalized <g> in words from Latin and Greek

Gibbous comes from Latin, but mysteriously appears to have always been pronounced with hard /g/ (according to the Oxford English Dictionary).

Another word from Latin, renege, has non-palatalized <ge> because it wasn't actually followed by a front vowel historically; it comes from Latin renegare. Formerly, it was spelled with <gue> to indicate the lack of palatalization. This is described by tchrist's answer to the linked question.

The word gizzard, according to the OED, comes from an Old French word something like giser (modern French gésier); the /g/ of the modern English pronunciation is thought to have come from a variant pronunciation that existed in French (as reflected by an alternate spelling guiser), but which has not been explained.

The word gynecologist and the prefix giga- used to have pronunciation variants with /dʒ/, but they are now generally pronounced with hard /g/. These words do come from Greek, but we'd expect velar softening to apply anyway.

Non-palatalized <g> in Germanic words

Words inherited from Old English

As Janus Bahs Jacquet mentions in the comments, Old English actually had a similar historical sound change whereby velar consonants (/g~ɣ/ and /k/) were palatalized before front vowels or the front semivowel, although the resulting palatal consonant sounds were still written in Old English with the letters <g> and <c>. In general, Old English palatal <g> corresponds to the Modern English consonant written <y>, and Old English palatal <c> corresponds to the Modern English consonant written <ch>. So words that were spelled with <ge, ce, ci> in Old English generally correspond to Modern English words spelled with <ye, che, chi>. Examples: yell (Old English gellan), chest (Old English cest), child (Old English cild).

Despite this, there are some modern English words with non-palatalized <gi> and <ge> that derive regularly from Old English. In Old English, unpalatalized velar consonants could be found before front vowels in the following environments:

• before the front rounded vowel <y> (which generally corresponds to modern English <i>)
• before <e> when it represented the front vowel that developed from umlaut of "o"

So words like gild (Old English gyldan) and geese (Old English gēs; umlauted plural of gōs) are examples of <gi> and <ge> with non-palatalized /g/ from Old English.

I don't know of any parallel examples of words from Old English spelled with <ci> and <ce> and pronounced with non-palatalized /k/, however. The sound /k/ is regularly spelled in these contexts as <k>: consider the words king (Old English cyning), kiss (Old English cyssan), and keen (Old English cēne).

Words borrowed from other Germanic languages

The most important group in this category is loans from Nordic languages, which were not subject to the process of palatalization that affected Old English velars. Norse influence is believed to be responsible for the /g/ in get and give.

In some cases, however, it is hard to determine if an unexpected /g/ instead of /j/ is due to Norse influence, or developed as part of an Old English dialect for other reasons that we don't know about. For example, I asked a question about the /g/ in "begin" on Linguistics SE, and it seems that the explanation for this development is still somewhat unclear (it could be Scandinavian/Norse influence, but there is no direct evidence of a Norse word with the correct form).

Palatalized <g> (/dʒ/) in Germanic words

As far as I know, there is no regular Germanic source of word-initial /dʒ/ in English. However, /dʒ/ did arise natively in certain contexts, from the long palatalized voiced velar obstruent or the palatalized voiced velar obstruent preceded by a nasal vowel. Word-finally, this usually shows up in Modern English as /dʒ/ and /ndʒ/ spelled <dge> and <nge> respectively. Examples: bridge (< OE brycg), twinge (< OE twengan), singe (< OE (sen(c)gan).