I'm assuming poll size tells the machine to measure the paper? You still have to remember there is going to be unusable space at the end of the paper that the rollers hold onto and the machine is not going to account for this because in the vinyl world you don't worry about how much vinyl is at the end, you are just going to cut it off and continue to use the roll.
You are trying to use the machine as a plotter, which is not what it was designed for and thereby missing some of the abilities a true plotter would have. You seem to be getting pretty close to your goal, but as you've discovered it's just going to be a matter of trial and error to find the settings and conditions that work for you.