But this only holds if you assume that "you" is the physical material rather than the unique "pattern" of that material.
This is what I was going to point out. In my opinion, all this talk of chairs and axes has gotten us off on entirely the wrong foot from the very start. Considering chair-continuity is no more instructive to the question of self-continuity than it is to the questions of continuity of love, justice, deception, or any other abstraction which isn't necessarily defined in physical terms.
Now the question is, ought we to define "the self" as your physical body, or ought we to define it as something else? I contend that one's "self" ought to be defined as their conscious, phenomenal experience; their qualia, if you will. Anything that doesn't share my conscious, phenomenal experience is not part of my self. This is necessarily an arbitrary distinction, but it has both logical and intuitive appeal.
Logically, it doesn't lead to any contradictions that come readily to my mind. The only candidate, I think, would be the "problem" of temporary lapses in consciousness. But I would answer that the events we generally refer to as lapses in consciousness (e.g., sleep) are not really cessations from conscious experience
per se, but rather only losses of memory. Importantly, consciousness is not synonymous with memory -- this is why we can say that the forgetful drunk is indeed conscious. Conversely, there is little justifiable reason to suppose that we somehow "switch off" consciousness while we sleep, only to switch it back on later. Brain imaging technology (and basic medical knowledge) tells us that the brain is doing
some things while we sleep. It's more reasonable to suppose that our phenomenal experience of sleeping simply doesn't get stored into long term memory (excepting dreams), just like the drunk's phenomenal experience. It's not clear that this is an empirically falsifiable statement in any way, but its no less falsifiable or verifiable than the alternative (i.e., no phenomenology while sleeping), and I think I've outlined good reasons for accepting the former rather than the latter.
For an interesting case that highlights the intuitive appeal, see
alien hand syndrome, where something that
is part of one's own physical body is nevertheless regarded as not part of one's self. It isn't connected to the patient's conscious intentionality, and the patient therefore considers it to be foreign.