cursed unity: raycasts without colliders

(The first part of what may or may not become a series about some of the ridiculous things I find myself doing in this ridiculous engine. I am by no means encouraging you to try any of this at home, but hey, I can’t stop you!)

So, I’ve recently found myself wanting to make small tweaks to scenes in a Unity game I’m working on—but in the Game view rather than the Scene view. After all, the Game view shows exactly what the game will look like—

—while the Scene view, well, not so much.

Wouldn’t it be nice to be able to click that planet on the left, without leaving the Game view, and nudge it over just a bit for a more aesthetically pleasing shot?

Well, to do that, we need to know which object the user clicked on. “Ah!” you say, “of course! We can use a ray cast!”

Hold on. Those stars and that planet? They’re all background objects. They’re just there to look good; none of them have colliders. Why would they? And sure, we could add colliders to them (they’re just spheres, after all), but what about more complex shapes? Do we really want to build a bunch of complex colliders just for editor functionality? No. No, we don’t.

Okay, so we want to raycast against the actual scene geometry. This has to be possible, right?— the editor itself does it when you click on something in the scene view!

So you go looking for that handy functionality in the documentation… and it is nowhere to be found. Yep. Unity can do this, but it doesn’t want to share that useful capability with the user. So we have to go find it ourselves—diving into the realm of undocumented functionality.

Well, there’s a function on HandleUtility called IntersectRayMesh— that seems useful!

internal static bool IntersectRayMesh(Ray ray, Mesh mesh, Matrix4x4 matrix, out RaycastHit hit)

But it’s marked internal, so we can’t reach it in a straightforward way. Reflection to the rescue…

static readonly MethodInfo intersectRayMesh = typeof(HandleUtility).GetMethod("IntersectRayMesh", BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static);

Now we can call Invoke on that MethodInfo— though passing in parameters gets a little tricky. We have to build a list of C# base objects and pass that in.

object[] rayMeshParameters = new object[] { ray, mesh, localToWorldMatrix, null };
if ((bool)intersectRayMesh.Invoke(null, rayMeshParameters))
{
  var rhit = (RaycastHit)rayMeshParameters[3];
}

And now we can pick against… a mesh. Hmm. That gets us a little closer, but we want to be able to pick against the entire scene. Well, we could just run this function on every mesh in the scene, but to make things a little less agonizingly slow, let’s at least check their bounds first. That way at least we’ll minimize the number of ray-mesh tests we need. While we’re at it, it’s handy to to be able to pass a function that decides whether a particular GameObject is pickable, so we can filter our results (I called it validate here, though in retrospect ‘filter’ would probably have been better).

class SceneMeshRaycast
{
    // Raycast against all scene geometry (not just colliders). Editor use only because it uses undocumented editor functionality
    // (and also because it is extremely slow).
    public static bool Raycast(Ray ray, out RaycastHit hit, out GameObject gameObject, System.Func<GameObject, bool> validate = null)
    {
        var renderers = Object.FindObjectsOfType<Renderer>();

        float bestDist = float.MaxValue;
        gameObject = null;
        hit = new RaycastHit();

        foreach(var r in renderers)
        {
            if (validate != null && !validate(r.gameObject))
                continue;

            Mesh mesh = null;
            if (r.TryGetComponent(out MeshFilter meshFilter))
                mesh = meshFilter.sharedMesh;
            if (mesh != null && r is SkinnedMeshRenderer)
                mesh = (r as SkinnedMeshRenderer).sharedMesh;
            if (mesh == null) continue;

            if (r.bounds.IntersectRay(ray, out float distance) && distance < bestDist)
            {
                object[] rayMeshParameters = new object[] { ray, mesh, r.transform.localToWorldMatrix, null };
                if ((bool)intersectRayMesh.Invoke(null, rayMeshParameters))
                {
                    var rhit = (RaycastHit)rayMeshParameters[3];
                    if (rhit.distance < bestDist)
                    {
                        hit = rhit;
                        bestDist = rhit.distance;
                        gameObject = r.gameObject;
                    }
                }
            }
        }

        return gameObject != null;
    }

    static readonly MethodInfo intersectRayMesh = typeof(HandleUtility).GetMethod("IntersectRayMesh", BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static);
}

Is this a perfect solution? Heck no. For one thing, it only works in the editor. And for scenes with a lot of objects (especially if they’re closely spaced and/or have very high polycounts) it’s likely to be very slow. Furthermore, relying on undocumented functionality means it could break with future Unity updates.

We could go a lot further with this— construct some kind of bounding volume hierarchy to further reduce unnecessary tests, write our own ray-mesh collision code to future-proof it…

But you know what? For now, and for what I need it for, it’s more than good enough.