A common excuse given by traitors at the time (and by Lost Causers now) is the idea of ‘state loyalty’. And at the time, state loyalty was a very real thing. People would sometimes legitimately think of themselves as members of their state first, and of the USA second, and several traitors - like Robert E. Lee - only defected once their state did, under the justification that they served their state first and foremost.

Samuel Phillips Lee is pointing out here that his commission as an officer, that he swore an oath to uphold, was to the USA, not to Virginia. He was sworn to defend the USA, not ‘Virginia’, and so was every other man who held a commission in the US military before secession broke out, regardless of what side their state legislature chose.