Solved: Triangulation help needed.

[mikerickson]

Here's my approach,
we are given 3 points (P1, P2, P3)
(These are vectors, each Pi has 3 coordinates)

We have a (partialy) unknown V =(given X, given Y, unknown Z)

Calculate
Q1 = P1-V
Q2 = P2-V
Q3 = P3-V

Then take the inner products
Q1 • Q2
Q1 • Q3
Q2 • Q3

And divide by the product of the lengths of Qi, Qj to give us the cosines of the angles between Qi and Qj

(Q1 • Q2) / Len(Q1) / Len(Q2)
(Q1 • Q3) / Len(Q1) / Len(Q3)
(Q2 • Q3) / Len(Q2) / Len(Q3)

Sum the arcsines of these values.

If V is interior to the triangle P1 P2 P3, i.e. lies in the plane defined by the 3 points, then the sum of those arcsines = 360°

So use Solver to find the Z value that will make that sum = 360
(Edit: Corrected this line and attached spreadsheet)

This method has significant round-off error. On one test, Excel returned .998 when 1 was the answer.