No, I am saying the English themselves spelled
color without the "u" before the alternate spelling emerged. All of these contested "-or" words --
color,
honor,
labor, etc. -- were borrowed directly from the Latin without the "u." It was sheer snobbery and ostentation that lead the British later to Anglicise them with a "u" in order to make the words seem "more British." A notable exception to this rule is
glamour, which came out of the French, if I am not mistaken. It is no coincidence that both the British and the Americans still spell it with the "u."