The reason you're getting odd voltages is because the xfmr secondary is not grounded, and thus there's no reference point. You're reading meter impedance, VERY high impedance ground faults (not to worry, they're so small there's no hazard), capacitance in the test leads, etc.
Does this xfmr have an X0 or X4 terminal? If so, it's the mid-point of one of the phases, and if you're going to ground it, that's the spot. Then you'll have a 240/120 3PH 4W delta system. There are thousands of these in operation. You'll read 120V to ground on 2 phases, 208 to ground on the 3rd, and 240 between any two phases.
If the xfmr has just X1, X2, and X3, then you'll need to study the system a bit more to determine if it should be grounded. If it is to be grounded, ground X2, and you'll have a grounded B system. Also known as a corner grounded system. There are not a huge number of these still in use, they were more popular years ago. Be careful using the grounded B with single phase loads as at least one of the legs will be 240V to ground. Usually it's ok, but this is one thing that needs to be studied.
I've seen many of these systems ungrounded, and other than a possible code violation, very few problems. I'd actually suggest that 'if it ain't broke, don't fix it'.
Does this xfmr have an X0 or X4 terminal? If so, it's the mid-point of one of the phases, and if you're going to ground it, that's the spot. Then you'll have a 240/120 3PH 4W delta system. There are thousands of these in operation. You'll read 120V to ground on 2 phases, 208 to ground on the 3rd, and 240 between any two phases.
If the xfmr has just X1, X2, and X3, then you'll need to study the system a bit more to determine if it should be grounded. If it is to be grounded, ground X2, and you'll have a grounded B system. Also known as a corner grounded system. There are not a huge number of these still in use, they were more popular years ago. Be careful using the grounded B with single phase loads as at least one of the legs will be 240V to ground. Usually it's ok, but this is one thing that needs to be studied.
I've seen many of these systems ungrounded, and other than a possible code violation, very few problems. I'd actually suggest that 'if it ain't broke, don't fix it'.