Incomplete hypergeometric function
Solution 1:
Let’s see if there is a closed form in terms of some more general functions which appear in literature. Your problem is:
$$\frac{1}{\mathrm{B}(\beta, \gamma-\beta)} \int_0^1 t^{\beta-1} (1-t)^{\gamma-\beta-1} (1-zt)^{-\alpha} \; \mathrm{d}t$$
First let’s use the Binomial series to get a sum representation for $|zt|<1$ with easier variables:
$$ \int t^{b-1} (1-t)^{c-b-1} (1-zt)^{-a} dt= \int t^{b-1} (1-t)^{c-b-1} \sum_{m=0}^\infty\binom{-a}{m} (-1)^m t^m dt $$
Remember the Incomplete Beta function:
$$\text B_p(a,b)=\int_0^p t^{a-1} (1-t)^{b-1} dt$$
Therefore we can integrate termwise:
$$\int t^{b-1} (1-t)^{c-b-1} \sum_{m=0}^\infty\binom{-a}{m} (-1)^m z^mt^m dt = \sum_{m=0}^\infty\binom{-a}{m} (-1)^m z^m \int_0^p t^{m+b-1} (1-t)^{c-b-1}dt = С+\sum_{m=0}^\infty\binom{-a}{m} (-1)^m z^m \text B_t(m+b,c-b)= \sum_{m=0}^\infty\frac{(-a)!}{m!(-m-a)!} (-1)^m z^m \text B_t(m+b,c-b)$$
From here, the goal would be to get a double hypergeometric series and use the First Appell function which is almost the same as this question where I derived a very similar problem’s closed form in terms of the First Appell function, but I am not sure what to do about the $-m$ factorial in the denominator as it cannot be converted into a factorial with a natural number coefficient, but maybe I will come up with something. Here is the answer anyways with the Pochhammer Symbol function $(a)_n$:
$$\int t^{b-1} (1-t)^{c-b-1} (1-zt)^{-a}dt=\frac{t^b}{b}\text F_1(b;b-c,a;b+1;t,tz)+\frac{t^{b+1}}{b+1}\text F_1(b+1;b-c+1,a;b+2,t,tz)+C,\text F_1(a;b_1,b_2,c;z_1,z_2)=\sum_{m=0}^\infty\sum_{n=0}^\infty \frac{(a)_{m+n}(b_1)_m(b_2)_n z_1^m z_2^n}{(c)_mm! n!}$$
Let me try to derive it another way. Please correct me and give me feedback!