Are your control points check points or ground control points or a combination of both?
When you are taking away the "error" you are basically doing what I mentioned, although not in the correct way. I am unsure about the resolution of the geoid in your area, but in the US, NAVD88, geoid 18...