Let the vectors a = - i + j + 3 k and b = i + 3 j + k . For some λ,...

Given $\vec{a} = -\hat{i} + \hat{j} + 3\hat{k}$ and $\vec{b} = \hat{i} + 3\hat{j} + \hat{k}$.

The vector $\vec{c}$ is given by:
$\vec{c} = \lambda \vec{a} + \mu \vec{b} = (-\lambda + \mu)\hat{i} + (\lambda + 3\mu)\hat{j} + (3\lambda + \mu)\hat{k}$

Using the first condition $\vec{c} \cdot (3\hat{i} - 6\hat{j} + 2\hat{k}) = 10$:
$3(-\lambda + \mu) - 6(\lambda + 3\mu) + 2(3\lambda + \mu) = 10$
$-3\lambda + 3\mu - 6\lambda - 18\mu + 6\lambda + 2\mu = 10$
$-3\lambda - 13\mu = 10 \quad \dots (1)$

Using the second condition $\vec{c} \cdot (\hat{i} + \hat{j} + \hat{k}) = -2$:
$(-\lambda + \mu) + (\lambda + 3\mu) + (3\lambda + \mu) = -2$
$3\lambda + 5\mu = -2 \quad \dots (2)$

Adding equation (1) and equation (2):
$-8\mu = 8 \Rightarrow \mu = -1$

Substituting $\mu = -1$ into equation (2):
$3\lambda + 5(-1) = -2$
$3\lambda = 3 \Rightarrow \lambda = 1$

Substituting the values of $\lambda$ and $\mu$ back into $\vec{c}$:
$\vec{c} = (-(1) + (-1))\hat{i} + (1 + 3(-1))\hat{j} + (3(1) + (-1))\hat{k}$
$\vec{c} = -2\hat{i} - 2\hat{j} + 2\hat{k}$

The magnitude squared of $\vec{c}$ is:
$|\vec{c}|^2 = (-2)^2 + (-2)^2 + 2^2$
$|\vec{c}|^2 = 4 + 4 + 4 = 12$

Answer: $12$