Hmm, I had a similar issue the other night. I popped in a standard DVD (The Ramones documentary End of the Century, look into it), and the PS3 played it in a funky manner:
The PS3 upscaled the DVD quite well, but I couldn't understand why it didn't make the picture as big as possible. The DVD wasn't widescreen, that part I got, but there was a lot of empty screen area at the top and bottom.