What you are describing is very common on RV generators, especially gas units, and it usually points to the engine not getting a stable, balanced load signal at idle or near-idle conditions. When the load is light, the generator has very little reason to stay at a perfectly steady speed, so any small problem in fuel delivery, air flow, governor response, or engine tuning shows up as surging. Under a heavier load, the engine is forced to work harder, which can hide the problem because the governor settles into a more consistent position.
The first thing I would look at is the fuel system. Old gasoline, a partially clogged carburetor jet, varnish buildup, or a dirty fuel filter can all cause hunting at light load. RV generators sit for long periods, and even a little stale fuel can leave deposits that affect low-speed operation. If the generator runs better after a load is applied, that often means the main jet is close to working but the idle and transition circuits are restricted. On many small engines, those tiny passages are the first place to clog.
Next, check the air side. A dirty air filter, an intake leak, or even a loose hose can make the engine run too lean. A lean mixture often causes surging because the engine speeds up, then loses power, then speeds up again as the governor keeps chasing the correct RPM. If the generator has been stored a while, inspect the carburetor gasket, intake boots, and any vacuum lines for cracks or looseness.
Governor problems can also cause this behavior, but I would rule out fuel and air issues first. A worn or sticky governor linkage may not react smoothly under light load, especially if the linkage is dry, bent, or rubbing against something. Make sure the springs and rods move freely and that nothing is binding. If the governor is out of adjustment, it may overcorrect at low load and seem much better once the engine is under demand.
Another possibility is that the generator is simply too lightly loaded for the way it is designed to operate. Some RV generators do not like running at extremely low load for long periods. A small steady load, such as turning on a converter, space heater, or another safe resistive load, can help stabilize them. That said, it should still run smoothly, so this is more of a symptom management step than a real fix.
If the unit has spark plugs, check them too. Worn or fouled plugs can make low-load running rough before they become obvious under heavier load. Also look at service history: if the oil is old, the fuel is old, or maintenance has been skipped, those are all strong clues.
The short version is that light-load surging usually comes from a fuel or carburetor issue first, then air leaks, then governor adjustment. If you tell the make and model of the generator, people can often narrow it down to the most likely failure points and the exact parts to inspect.