You may, indeed, need to manually adjust your brakes, even though they are "self adjusting".
Like other automotive "self adjusting" brakes, they only "self adjust" during the backing up cycle. Even in autos, most of us don't backup enough for the system to adjust for all the wear that we put on the brakes going forward. Since you like the brakes to do "more" braking when at a higher speed, my guess is that your wear pattern is higher than the adjustment made when backing into a campsite and at a slow speed (you have to have a fair amount of speed for the drums to the rotate the adjuster and you need to make "firm" stops when applying the brakes). Most backing up with a trailer is done at "dead slow" and the brakes don't actually apply enough torque to the adjuster to get them to turn, particularly if you've turned the gain down...the TV is doing most of the braking then. There's nothing wrong with using your controller in the way that you do, I also run at a higher gain when at highway speeds and then lower it when in town or maneuvering around the campground.
Most likely "adjusting" your "self adjusting" brakes will take care of the problem. You could, alternatively, go to a big empty parking lot and backup while repeatedly making firm brake applications to get the adjusters to take up the slack; might take a while though if they have a lot of slack to take up. That was always an issue with the self adjusting brakes in autos and most people found that the rear brakes would last the life of the car unless they manually adjusted them so that they worked or they did a lot of backing up with sharp brake applications to rotate the adjuster.
My 2 cents, your mileage may vary...
2 Cats; J-Lo and Ragamuffin :R
2014 Thor Tuscany 40RX DP
2011 Ram 2500 Longhorn CTD HO
2011 Keystone Cougar 318SAB (now gone)
2008 FunFinder X 210WBS (Sadly gone)