Why is cat spelt with a 'c' while kitten is spelt with a 'k'?
If the word "kitten" were spelled "citten", that would suggest the pronunciation /ˈsɪtən/. Although English spelling and pronunciation are not perfectly correlated, the association between the pronunciation /kɪ/ and the spelling "ki" (instead of "ci") is actually quite strong; this seems a plausible explanation for why the word pronounced /ˈkɪtən/ is spelled with "ki". The same spelling is used in Modern English in words like king, kiss, kin, which are all pronounced with /kɪ/ and come from words that had spellings with "c" in Old English—the Oxford English Dictionary (OED) entries for these words list "cinyng", "cyssan", "cyn" as spellings that can be found in Old English texts.
Etymology
I hadn't thought before now about the etymological relationship between cat and kitten. The etymology of kitten seems somewhat uncertain to me, but I'm fairly sure that it's irrelevant in any case.
It seems likely that kitten comes from (a dialect of) French
According to an OED entry that was first published in 1901, the word kitten is thought to come from "Anglo-Norman *kitoun, *ketun = Old French chitoun, cheton, obsolete variant of French chaton kitten." The following two quotations from the OED entry show that the spellings kitoun and ketoun used to be used in English:
1377 Langland Piers Plowman B. Prol. 190 Þere þe catte is a kitoun þe courte is ful elyng.
a1425 Edward, Duke of York Master of Game (Digby) ix Þei beer hir kitouns..as oþer cattes, saue þei haue not but two ketouns at ones.
The asterisk before the Anglo-Norman forms *kitoun and *ketun indicates that these are reconstructions: we don't actually have any direct evidence for the existence of these form in Anglo-Norman. But based on other words, we know that French "ch" often corresponded to Anglo-Norman /k/.
The OED mentions
The French form chitoun occurs in Gower Mirour de l'omme 8221: Teut ensement comme du chitoun, Qi naist sanz vieue et sanz resoun.
German? Germanic? Probably not
I don't know of any evidence that suggests that the word kitten comes from or was influenced by German. Relatively few English words come from German.
A large number of English words are Germanic in etymology, but what this means is that they come from the common ancestor of languages like English, German, Icelandic and Dutch—this common ancestor is older than the spelling systems of any of the descendant languages spoken today. The modern German form "Katze" did not exist in that common ancestor. However, that common ancestor may have had a word pronounced something like *kattuz (this is a reconstructed form, so the use of "k" here is a purely arbitrary convention of modern linguists, not anything that could affect the spelling of the cognate words in descendant languages). This word is obviously related to Latin cattus (the source of French chat), but according to Wiktionary, the nature of the relationship is unclear.
It's true that the modern English form kitten looks like it could have come in some way from this Germanic base. There are other English words ending in -en that look somewhat like diminutives, such as chicken and maiden. In fact, there is an OED entry for a Germanic ending -en that is supposed to be a diminutive suffix (-en, suffix1):
Etymology: < Germanic -îno(m, formally the neuter of -îno- , -en suffix4.
Used to form diminutives from ns. (esp. names of animals), as in chicken n.1, kitten n., maiden n. and adj., Middle English ticchen kid; also in Middle English stucchen small piece.
It's obviously tempting to interpret kitten as containing this Germanic suffix—so tempting, in fact, that whoever wrote this OED entry did in fact include the word kitten as an example, even though this seems to contradict the etymology given in the OED entry for the word kitten!
But I would assume that the OED entry for kitten is correct, and the OED entry for -en, suffix1 is wrong, because I don't think there is an easy way to explain the early spellings with -oun if we assume that the word originally comes from Germanic. If you look at the historical spellings recorded in the OED for the words chicken and maiden, there is a lot of variation over time in the spelling of the vowel before the "n", but none of the variant spellings uses the digraph "ou".
It seems possible though that the words in English that end in a Germanic suffix -en contributed to the development of the modern spelling of the end of the word kitten (the use of the spelling -en could instead be explained as just a respelling of a reduced vowel in an unstressed syllable, but there are words like like cotton, button, and iron that have a reduced vowel in the last syllable and are spelled with -on, so it's clear that vowel reduction hasn't caused all words that end in /ən/ to come to be spelled with "en" in present-day English).
This is one of those times one needs to probably go back to before (old) German and Latin.
Linguists have partially reconstructed the mother of all European languages, called Proto Indo-European (PIE). Very often the main descendant branches - which for English speakers are really 2, the Germanic and Latin branches - have enough divergence that new words came in, different words formed, and there is scarce resemblance between the branches for some words.
When the French ruled England durin the Norman Conquest, they introduced Latin-based words in parallel with Germanic-rooted ones, the French ones being seen as 'classier' or more 'delicate'. This is often true with food: Germans talk about eating "cow flesh", not "beef" (which comes from French "bouef"); "swine flesh", not pork (from Frech "porc"); etc. And it's not just food: English had "house", pronounced the same as German "haus", but "mansion" - being a very classy house - is from French "maison".
Cat is one of those unusual words that appears to have altered very little from PIE in both the Germanic and Latin branches. So, in Dutch, cat is "kat", and in German "katze", but the word in Latin was "cattus".
You can weave through old, middle, and modern French, German, and other languages, but the animal is called something that sounds like what we spell "cat" in all of them (actually in French it lost the hard 'c', and is now "chat", where the 'ch' is pronounced like 'sh' and the 't' is silent - so it sounds like 'sha' - still, it starts with 'c' owing to its Latin heritage).
But the reality is that the English word is a re-convergence of a similar word going back to PIE, so we have lots of reasons to put either as 'k' (like German katze) or a 'c' (like Latin cattus) on front; Germanic languages tend to not use 'c' as a hard sound in front, but 'k', though Latin had a hard 'c' (at least if not followed by 'e' or 'i').
So,in the end, its really rather random in this case, because there isn't a single precedent.
catkin
an inflorescence that hangs from some branches of trees, so called because it resembles a cat's tail. Note that it too begins with the letter C
late 16th century: from obsolete Dutch katteken ‘kitten’.
Collins Dictionary
kat
Aktually, there is already the word kat, a spelling variant of khat, a white-flowered evergreen shrub, Catha edulis, from Afrika and Arabia, whose leaves have narkotik properties. And in kase you're interested, the pronunciation is eksaktly the same as that of cat /kæt/.
Khat is derived from the Arabic qāt and dates back to the 19th century.
cit
One reason why the first letter of cat and kitten are spelled differently is that the syllable cit is nearly always pronounced /sɪt/, if kitten were spelt (spelled) citten, it would probably be pronounced /ˈsɪt(ə)n/.
There are twelve 6-letter words that start with cit, no one is pronounced with the "hard" letter C (k), they are pronounced consistently with a soft ‘s’ sound
citrus, citron, citric, citied, cither, citral, citify,
citole (ˈsɪtəʊl), citrin, cities, citing, citer(s)
There are two 7-letter words starting with cit-; one of which is citizen, which coincidentally has the -en suffix, while cittern, a 16th-century stringed instrument similar to a lute, is spelled with double ‘t’ just like kittern–oops–kitten.