You bring up some good points. Your points are very similar to the follow-up conversation I had with my insurance advisor when I told him I wanted to cancel my policy. You can see the full details of my conversation with him in my follow-up article Terms vs. Whole Life Insurance Part 2.
In brief, my insurance plan for after my 30 year term life policy is up is to self-insure as you said. The main purpose of insurance is to leave money behind for your loved ones after you die. In my case, by that point, I should have the same amount of money as, if not more than, the whole life policy as you said.
I did take out my graphs to the future to include retirement in the follow-up article. I made some generous assumptions about returns and taxes to favor the whole life policy where I could. Even with the odds tilted in favor of the whole life policy, it barely manages to keep up. The investment account again provides just as much, if not more, money in the long run and will give me access to more of my funds if needed.