Without removing the seat belts..this works pretty well for belts that are not damaged, pilled, frayed..
Pull the seatbelt out of the retractor as far as it'll go and cinch the buckle way back so the belt stays out.
Buy some quality carpet cleaner such as Hoover makes for their carpet cleaning machines.
Prepare a strong solution of the cleaner with hot water in a 2 gallon plastic pail..
Put a couple of towels on the seat..Put the bucket on the seat..Immerse the seat belt into the solution.
Scrub both sides of the belt for ten minutes with a scrub brush, dipping it back into the solution frequently.IF the water-solution is noticeably dirty after scrubbing & dunking for 10 minutes, Change the water solution and reimmerse and leave it over night.
Next day, take a look and see..IF they look clean, rinse them in a bucket of warm water and let dry for several days BEFORE you retract them back into the retractor..Use a hair dryer IF you are in a hurry but make sure they are truly dry before retracting them.
If not clean yet, repeat the process..Took a few years to get dirty and worth the time to get them cleaned up .Jim