Was very helpful Mike! Finally got around to it today. Here's a quick tutorial for anyone else needing to do this.
First, twist the mirror in/back towards the car to access a small opening. Use this to pry the outer portion of the mirror arm open and give it a good yank. There's two clips that hold this on. I broke one of them pulling it off, but it holds very secure even with the one.
Under here there are two screws to take off the mirror.
Once the mirror is off, there is another screw below that holds the plastic painted portion of the door to the rest of the door structure.
From there, go to the back of the door to it's latch. There's a single screw to take out there.
Then take out the three screws on the interior-side of the door at the top (the ones used to mount windows).
And voilà! a few light pulls and the plastic painted portion comes right off (~4 clips at the bottom).
(I have no idea how this optical illusion happened - the reflection makes it look like my car frame (black) is made of the stones/snow on the ground.
Here are photos of the clips that hold the door together from the inside at the top. One had fallen off when I installed the TWY Rain windows and was rattling around within the door.
Here's also the part number, should anyone ever need to order a right side black door body part