Prove that determinant of a matrix (with polynomial entries) is non-zero
Solution 1:
I think you are asking if the matrix has full rank for all ${\bf x}\in (0,1)^n$. I can show that the matrix has full rank for some ${\bf x}\in (0,1)^n$.
The matrix has full rank when all variables are assigned the same value $x\in (0,1)$. Under this assignment the matrix becomes
$$ \left[\begin{array}{cccc} 0&r&\cdots&r\\ r&0&\cdots&r\\ \vdots&\vdots&\ddots&\vdots\\ r&r&\cdots&0 \end{array} \right] = r(J-I) $$ where $r = \binom{n-2}{k-1}x^{n-k-1}\left(1-x\right)^{k-1}$, $I$ is the $n\times n$ identity matrix, and $J$ is the $n\times n$ matrix of all $1$'s.
The matrix $J$ has rank 1 and trace $n$, so its e-values are $0$ (multiplicity $n-1$) and $n$ (multiplicity $1$). Therefore the e-values of $J-I$ are $-1$ (multiplicity $n-1$) and $n-1$ (multiplicity $1$). Hence the determinant of $J-I$ is $(-1)^{n-1}(n-1)$. Hence the determinant of your matrix, after every variable has been assigned the value $x$, is $(-1)^{n-1}(n-1)\left[\binom{n-2}{k-1}x^{n-k-1}\left(1-x\right)^{k-1}\right]^n$, which is not zero when $x\in (0,1)$.
Solution 2:
A possibly useful way to rephrase the question:
For any subset $\mathcal A \subseteq \{1, 2, \dots, n\}$, let
$$p_{\mathcal A} = \prod_{i \in \mathcal A} x_i \prod_{i\notin \mathcal A} (1-x_i),$$
and let $Q_\mathcal A$ denote the quadratic form on $\mathbb R^n$ given by
$$Q_\mathcal A (v) = \left(\sum_{i \in \mathcal A}v_i\right)^2 - \sum_{i \in\mathcal A} v_i^2 = \sum_{i,j \in \mathcal A, i\neq j}v_i v_j.$$
Consider the quadratic form $$Q = \sum_{\mathcal A} p_\mathcal A Q_\mathcal A,$$ where we sum over subsets of size $n-(k-1)$. Up to a diagonal change of coordinates, the matrix of this quadratic form is the matrix in the question. So the goal is to show that $Q$ is a nondegenerate quadratic form.
It appears that $Q$ has signature $(1, n-1)$. Since it's easy to find a one-dimensional subspace of $\mathbb R^n$ on which $Q$ is positive definite, it suffices to find a codimension 1 subspace $V$ of $\mathbb R^n$ on which $Q$ is negative definite.
Note that up to scaling $Q$ is the expected value of $Q_\mathcal A$, where $\mathcal A$ is chosen via independent Bernoulli trials with probability $x_i$ for each event $i \in \mathcal A$, but conditioned on $|\mathcal A| = n-(k-1)$. Sensible candidates for $V$ might be orthogonal complements to vectors of the form $(v_1, \dots, v_n)$ where $v_i$ is somehow related to this probability distribution, e.g. $v_i = x_i$, or $v_i = x_i/(1-x_i)$, or $v_i = \mathbb P(i \in \mathcal A)$.