Is this an algorithm in the family of "2D bin packing"?
As for packing sprites for games... I remember the fun on very constrained devices where many of the sprites where actually more round than square. So it was about packing both rectangles (for tiles and "rectangle" sprites) and packing circles too. The size of the spritesheet had to be kept small but in memory things were okay'ish (as in: a "round" sprite could be extracted from the spritesheet in its own rectangle).
Thankfully thing is: this can be done with a much more powerful machine than the device the sprite sheet shall eventually be on.