Let λ ∈ ℝ, → a = λ ^ i + 2 ^ j - 3 ^ k , → b = ^ i - λ ^ j + 2 ^ k...

Given,

\overset{\to}{a} = \lambda\overset{^}{i} + 2\overset{^}{j} - 3\overset{^}{k}

\overset{\to}{b} = \overset{^}{i} - \lambda\overset{^}{j} + 2\overset{^}{k}

Also given,

( ( \overset{\to}{a} + \overset{\to}{b} ) \times ( \overset{\to}{a} \times \overset{\to}{b} ) ) \times ( \overset{\to}{a} - \overset{\to}{b} ) = 8\overset{^}{i} - 40\overset{^}{j} - 24\overset{^}{k}

⇒ ( \overset{\to}{b} - \overset{\to}{a} ) \times ( ( \overset{\to}{a} + \overset{\to}{b} ) \times ( \overset{\to}{a} \times \overset{\to}{b} ) ) = 8\overset{^}{i} - 40\overset{^}{j} - 24\overset{^}{k}

Now using vector triple product,

$\overset{\to}{A} \times ( \overset{\to}{B} \times \overset{\to}{C} ) = ( \overset{\to}{A} . \overset{\to}{C} ) \overset{\to}{B} - ( \overset{\to}{A} . \overset{\to}{B} ) \overset{\to}{C}$ we get,

⇒ ( ( \overset{\to}{b} - \overset{\to}{a} ) \cdot ( \overset{\to}{a} \times \overset{\to}{b} ) ) ( \overset{\to}{a} + \overset{\to}{b} ) - ( ( \overset{\to}{b} - \overset{\to}{a} ) \cdot ( \overset{\to}{a} + \overset{\to}{b} ) ) ( \overset{\to}{a} \times \overset{\to}{b} ) = 8\overset{^}{i} - 40\overset{^}{j} - 24\overset{^}{k}

⇒ 0 + ( ( \overset{\to}{a} - \overset{\to}{b} ) \cdot ( \overset{\to}{a} + \overset{\to}{b} ) ) ( \overset{\to}{a} \times \overset{\to}{b} ) = 8\overset{^}{i} - 40\overset{^}{j} - 24\overset{^}{k} . . . . . ( 1 )

( ( \overset{\to}{a} - \overset{\to}{b} ) \cdot ( \overset{\to}{a} + \overset{\to}{b} ) ) = \left[\left(\lambda - 1\right)\overset{^}{i} + \left(2 + \lambda\right)\overset{^}{j} - 5\overset{^}{k}\right] . \left[\left(\lambda + 1\right)\overset{^}{i} + \left(2 - \lambda\right)\overset{^}{j} - \overset{^}{k}\right]

$⇒ ( ( \overset{\to}{a} - \overset{\to}{b} ) \cdot ( \overset{\to}{a} + \overset{\to}{b} ) ) = 8$

From equation (1) we get,

⇒ 8 ( \overset{\to}{a} \times \overset{\to}{b} ) = 8\overset{^}{i} - 40\overset{^}{j} - 24\overset{^}{k}

Now, \overset{\to}{a} \times \overset{\to}{b} = \left|\begin{cases}\overset{^}{i} & \overset{^}{j} & \overset{^}{k} \\ \lambda & 2 & - 3 \\ 1 & - \lambda & 2\end{cases}\right|

= ( 4 - 3\lambda ) \overset{^}{i} - ( 2\lambda + 3 ) \overset{^}{j} + \left(- {\lambda}^{2} - 2\right)\overset{^}{k}

$⇒ \lambda = 1$

∴ \overset{\to}{a} = \overset{^}{i} + 2\overset{^}{j} - 3\overset{^}{k}

\overset{\to}{b} = \overset{^}{i} - \overset{^}{j} + 2\overset{^}{k}

⇒ \overset{\to}{a} + \overset{\to}{b} = 2\overset{^}{i} + \overset{^}{j} - \overset{^}{k}, \overset{\to}{a} - \overset{\to}{b} = 3\overset{^}{j} - 5\overset{^}{k}

⇒ ( \overset{\to}{a} + \overset{\to}{b} ) \times ( \overset{\to}{a} - \overset{\to}{b} ) = \left|\begin{cases}\overset{^}{i} & \overset{^}{j} & \overset{^}{k} \\ 2 & 1 & - 1 \\ 0 & 3 & - 5\end{cases}\right| = 2\overset{^}{i} + 10\overset{^}{j} + 6\overset{^}{k}

$⇒ \left|( \overset{\to}{a} + \overset{\to}{b} ) \times ( \overset{\to}{a} - \overset{\to}{b} )\right| = \sqrt{4 + 100 + 36}$

Now,

${\left|\lambda ( \overset{\to}{a} + \overset{\to}{b} ) \times ( \overset{\to}{a} - \overset{\to}{b} )\right|}^{2} = {\lambda}^{2}{\left|( \overset{\to}{a} + \overset{\to}{b} ) \times ( \overset{\to}{a} - \overset{\to}{b} )\right|}^{2}$

$∴$ Required answer $= 4 + 100 + 36 = 140$