Welcome to the forum.

To remove the door panel, first you'll need to remove the window crank (manual), lock knob (manual), door handle, and grab assist.  Then remove the screws and w/ a panel removal tool, pop the snap buttons around the periphery.  Then move the panel up to unhook it from the top of the door and disconnect any wiring.

Now this is for my '73 so yours may be different to some degree.

+1 on the door panel advise from 73 shark. For your '81 it's even a bit easier with a small screw holding on the interior handle bezel then three screws holding on the rubber arm rest. There will be one more screw up near the top front corner of the little map pocket, another below and rearward of the armrest and the last at the top rear corner of the panel, all pretty easy to get to. Not sure what your exact door handle issue is but I've had good success loosening things up in the door latch itself by squirting in a bunch of WD 40 into the latch. You should be able to do this from the top without removing the panel. The latches were lubed from the factory with white grease and over time dust, dirt and grime collect in the grease and stiffen things up. Rinsing it out with a generous amount of WD 40 will clean out the mechanism and may solve your latch issue, might be worth a try. There is a drain hole at the bottom corner of the door below the latch. Expect most of the WD 40 to drain down out of the hole. A wad of paper towel stuffed into that hole should collect most of it. If that doesn't solve your issue, then it might be the handle mechanism itself. Worn, broken or otherwise it may have to be replaced, pretty common since the driver handle gets most of the use. Good luck !!