Welcome to sciforums, SusyT. I read your question but I don't have the answer for you. Laws vary from state to state. If this is a "can I" type question then you need to talk to a lawyer.
hi Abandonment would involve you making absolutely NO attempts to visit, contact or have a relationship with the children. What their mother is doing is called "frustration of parental rights/parenting time" (that's the term in most states). Child support and visitation orders are exclusive of one another. If you never had a court order for visitation, it was up to the two of you to determine a feasible schedule. If there was an order, the custodial parent is in violation by not alerting you to changes in physical location of the children.
Possible? Probably, but probable? Not possibly.... If the adoption agency learns about it, they would be less likely to grant the right to adopt, that is my best guess...