How I failed to make a game: Raycasting on the GBA

Not sure if you care at this point, given that you stopped working on project, but there's a better way to find the ray direction for each column than using sin/cos for every one, which will also get rid of the slightly warped look:

Calculate the two vectors from the camera at the very left and right of the screen (using your fov angles and sin/cos, that's fine). Then, to find the ray direction vectors for each column, interpolate linearly between your left and right direction vectors, and possibly normalize the resulting vectors if your ray walking algorithm requires it.

This will create a perspective that integrates tightly with sprites that you 3D project the usual way, and lines will stay straight lines.

Of course I still care :) Thank you for sharing this technique, I will try it out.

In my experience there are two mistakes people sometimes make that leads to the fisheye lens look. The first, as parent mentioned, when calculating the ray directions you should linearly sample a line rather than interpolating view angles. Second, when doing per-column perspective division, you should divide by dot product of the forward vector and the difference between the ray intersection and the camera location. Often if someone is making the first mistake they are also making the second.

Following this will give you the normal pinhole camera 3d projection that we all expect to see from a 3D game.

Rule of thumb I found as a beginner in 3D graphics: any time polar coordinates seem like the obvious solution, there's usually a better way.