People often are. People would rather believe that something beyond their control is responsible for their 'fate' (sic), rather than being responsible for it. Don't interpret this to mean that I'm disparaging you or your reluctance in particular; it's simply the case that for the most part, people fear being overwhelmed with the depth of responsibility they would face if they chose to see it.
Yes, this is a common charge against those reluctant to apply the concept of choice, and yes, the reasons you mention certainly apply to many people.
However, my reasons for being reluctant about choice are different.
Namely, we must distinguish between explanation and instruction. Generally, it is my impression that people are very poor at distinguishing the two, and usually mix them up. A lot of mental distress (with the according consequences) derives from this mixing up the two.
Of course the two are closely connected, but not to be confused.
One thing is to say,
You should pay attention to what you do. You should be careful. You should evaluate your options and make a rational decision. - this is an instruction.
Another thing is to say,
You chose to get angry. You chose to make a mistake. You chose to ignore me. There are no accidents. Everything happens for a reason. - that is an explanation, at least a possible one.
But nowadays, it seems rude to use phrasings with should, need, ought, and must, and people frequently veil their instructions in explanations. Yet, their intentions are not clearly expressed, explanations get mixed up with instructions, and chaos flourishes.
(On a side note, it is this fear of responsibility that is the primary thrust behind organized religions, IMO

)
Perhaps. I am more inclined to think that the main thrust behind many religions, esp. theisms, is the double bind resulting from an overwhelming and irrational sense of responsibility that is instilled into people. Double binds are excellent tools for manipulation.
For example, according to some Christian doctrines, we are responsible for either going to hell, or to heaven; according to them, this is a choice. But IMO, this is not a choice, because there are several requirements to be met that are not under my control - for example, the Bible in fact being the Word of God, and Jesus actually having redeemed us. I can't prove that they are what they claim, but if I don't prove it, I'll go to hell.
So I think the hell/heaven situation is not about responsibility, but more about a unilaterally imposed obligation (to believe what someone tells us), under threat of extreme duress. And according to human law at least, unilaterally imposed obligations do not constitute responsibility.
If anything, many religions have severely twisted the concept of responsibility.
The process is gradual, but ultimately it does lead to a decision. Now, whether or not you could say this choice is made consciously or not is a matter for another debate.
Suffice it to say, when it comes to how an individual judges their life, it is only that individual that can make that judgment (in any meaningful way). In effect, the choice is made passively, over time, as one encounters events and their effects. It is only after the fact that one chooses to apply the label 'tragedy' (or comedy, or waste, etc....).
Again, I'm cautios here and refer to distinguishing between explanation and instruction.