X originally was created on/ran on a graphics terminal - the DEC VAXstation 100. The VS100 was quite different to the later X thin client terminals: it required an adapter card to be installed in a host system, and the software which ran on the VS100 could directly access a chunk of shared memory on the host.
Ports to workstations with inbuilt graphics hardware came later.
References: