Why is the J in "hallelujah" not pronounced as /dʒ/, but as /j/?

This is only a partial answer.

The pronunciation of these names was not directly loaned from Hebrew pronunciation to English pronunciation, and the /j/ to /dʒ/ change is not an English sound change

The sound change that created /dʒ/ from /j/ did not occur in English, but rather in certain Romance languages as they developed from Latin.

Latin /j/ in word-initial position regularly developed to /ʒ/ in French. An example of this change is French jeune /ʒœn/ "young" from Latin juvenis (which started with /juwen/). Because of this, /ʒ/ is the regular pronunciation of the letter J in French—not only in inherited words like this, but also in learned words that were "borrowed" from Latin, such as judiciaire, where the retention of the consonant d shows that the word was not inherited directly from Latin to French.

Even though the modern French pronunciation of the letter "J" is /ʒ/ rather than /dʒ/, it is thought that an earlier intermediate stage of the /j/ to /ʒ/ change was /dʒ/ (as in modern Italian giovane /ˈd͡ʒovane/). Since many English words of Latin origin were taken from French or from French speakers, the French feature of pronouncing /dʒ/ for "j" became an established part of how English speakers pronounced Latin words.

The pronunciation of "j" as /dʒ/ can apply to words that didn't come from French or from French speakers, but though analogy, not as a regular "sound change" in the sense that linguists use the term. The use of "j" = /dʒ/ in the established pronunciations of some Hebrew terms and names could either be because of analogy, or in some cases the English pronunciation might be based on a French or Latin form of the name; I don't know enough about the history of how these names were transmitted to be sure.

Hallelujah/alleluia is used with /j/ and not /dʒ/ in French alléluia

"Hallelujah" seems to be a special case, although I'm not sure of the reason. It coexists with the spelling "alleluia", which uses "i" instead of "j". We can't reliably distinguish these two letters in historical sources, because the I/J distinction is relatively recent, but another spelling variant that the Oxford English Dictionary says used to be used is "alleluya". In this context, I think the spelling "y" unambiguously indicates a pronunciation something like /j/ rather than /dʒ/. In fact, the Oxford English Dictionary's four earliest citations of alleluia (from the fifteenth and sixteenth centuries) use the spelling "alleluya", and the entry says that this form was also used earlier in French (it refers to "Anglo-Norman aleluia, Anglo-Norman and Middle French alleluya (13th cent.)").

The modern French form alléluia also has /j/, not /ʒ/. I don't know why. I'd guess it might be related either to the timing of when the form was loaned into French, or to the sound's position in the middle of the word between vowels.

All this was made possible by the history of the letter "J". From OED:

J, n.

Etymology: the tenth letter of the alphabet in English and other modern languages, is, in its origin, a comparatively late modification of the letter I. In the ancient Roman alphabet, I, besides its vowel value in ibīdem , mīlitis , had the kindred consonantal value of modern English Y, [...]

From the 11th to the 17th cent., then, the letter I i represented at once the vowel sound of i , and a consonant sound /dʒ/, far removed from the vowel. [...] it was not till the 17th cent. that the device of utilizing the two forms of the letter, so that i, i, should remain as the vowel, and j, j, be used for the consonant, was established, and the capital forms of the latter, J, J, were introduced.

No word beginning with J is of Old English derivation. Many are from Latin, chiefly through French; some from Greek, and a few from Hebrew and Arabic.

However, many printing fonts lacked a capital "J" and so "I" was used for quite a while during the transition. So we see that all your words would have been spelled with an "I" although the reader would adjust the pronuniation a foreign word as close to the original as was possible for them.

An example of varied "y", "i", "j" spelling from the OED. First Jehovah:

Jehovah, n. Pronunciation: /dʒɪˈhəʊvə/

Etymology: The English and common European representation, since the 16th cent., of the Hebrew divine name Yhwh . This word (the ‘sacred tetragrammaton’) having come to be considered by the Jews too sacred for utterance, was pointed in the Old Testament by the Masoretes with the vowels ' (= ă), ō, ā, of ădōnāi , as a direction to the reader to substitute Adonai n. for the ‘ineffable name’; which is actually done by Jerome in the Vulgate translation of Exodus vi. 3, and hence by Wyclif. Students of Hebrew at the Revival of Letters took these vowels as those of the word Yhwh (IHUH, JHVH) itself, which was accordingly transliterated in Latin spelling as IeHoVa(H), i.e. Iehoua(h. It is now held that the original name was IaHUe(H), i.e. Jahve(h, or with the English values of the letters, Yahwe(h, and one or other of these forms is now generally used by writers upon the religion of the Hebrews. The word has generally been understood to be a derivative of the verb hāwāh to be, to exist, as if ‘he that is’, ‘the self-existent’, or ‘the one ever coming into manifestation’; this origin is now disputed, but no conjectured derivation which has been substituted has found general acceptance.

The following is cited as the first use of the form Iehoua (Jehova)

1518 P. Galatinus De Arcanis Catholicae Veritatis ii. f. xlviij Non enim hę quatuor literę [yhwh] si, ut punctatę sunt, legantur: Ioua reddunt: sed (ut ipse optime nosti) Iehoua efficiunt

β. Examples of recent forms of the word.

1869 R. Martineau tr. H. Ewald Hist. Israel II. 130 Jahveh alone was the true defence.

1892 Montefiore Hibbert Lect. 45 Yahveh, to the Israelite, was emphatically the God of Right.

1899 R. H. Charles Eschatol. 8 As the natural God, Yahwe was the invisible Head of the nation.

This thus explains why Yhwh, IHUH, and JHVH all represented the same sound.

(It may be worthwhile considering that God is the only Being who knows how to pronounce His own name - this is in line with the general advice that only the owner of a name knows how to pronounce it.)

Jah is simply a shortened form, but, followed the same transition:

= Jehovah n.

1539 Bible (Great) Psalms lxviii. 4 Oh synge vnto God,..prayse ye him in his name Ia [1611 Iah] and reioyse before hym.

1613 S. Purchas Pilgrimage 154 In the name of Iah the God of Israel. There is none like to Iah our God.

1758 C. Wesley & J. Wesley Hymns of Intercession 32 Jah, Jehovah, Everlasting God, come down.

And now to "hallelujah"

Etymology: < Hebrew hallĕlū-yāh ‘praise (ye) Jah (= Jehovah)’; the verb is the imperative plural of hallēl : see Hallel n.

1.a. The exclamation ‘Praise (ye) the Lord (Jah, or Jehovah)’, which occurs in many psalms and anthems; hence, a song of praise to God; = alleluia int.

1535 Bible (Coverdale) Psalms cv[i]. (heading) Halleluya.

1557 Bible (Whittingham) Rev. xix. 1 I heard the voyce of muche people in heauen saying, Halleluiah.

1625 R. Sanderson Serm. I. 115 The abridgement is short, which some have made of the whole book of Psalms but into two words, hosannah, and hallelujah.

1818 Scott Heart of Mid-Lothian ii, in Tales of my Landlord 2nd Ser. I. 64 That the psalms they now heard must be exchanged, in the space of two brief days, for eternal hallelujahs or eternal lamentations.