CAN MY EX SUE FOR CUSTODY IF HE SAYS HE DOES NOT SEE MY DAUGHTER BUT HE MAKES NO EFFORT TO COME GET HER?

My ex and I have a 10 year old daughter. We have been separated since she was 2. We became divorced officially in 2007. The divorce decree states that I have custody and that he gets her every summer and every other winter break. I have been letting him get her every winter break except for 2010. With this being said, the decree also states that he is responsible for coming to get her. He lives in Fl and I live in Ga. He has not made effort to get her so my mother who lives in Fl also comes to get her those times. Mind you she pays for all of the gas and expenses to come get my daughter. It has been this way since I have been in Ga which was 2006. He sporadically was paying child support (like when he was in the army and was forced to because for awhile he was not sending us any money at all while he was deployed.)Just last year since I have finally gone after him for child support he conveniently has complained that he does not get to see our daughter enough because my mom "hogs"her. Apparently this was his basis for not paying child support and that is why he had an issue on the rare occasion that he was paying it. I told him that my mom is doing us a favor by coming to get our daughter and that since he has not come and got her himself that this is the way it is. My mom always makes every effort in order to make sure he sees her. However there have been times that he has brought our daughter back to my mother or not picked her up because he was working or had something else going on such as getting drunk. So this year IN May when my daughter got out of school for the summer, I made sure to give him a chance to come and get her. I had no valid number in order to call him or anything. We waited 2 weeks and still no call. Finally I told my mom she could come get my daughter. Last night I decided to be nice and send him a message on facebook to let him know that our daughter was down there in Fl and if he wanted to see her then he could get in touch with my mother. He wrote me back threatening that we would resolve this issue legally. He has a history of drugs and drinking which is partly why he got discharged from the army, his dad was an alcoholic and his mom is a paranoid schizophrenic who has in the past dated childmolesters This is not the type of people that I feel would be the proper guardians for my daughter. I do not drink, I go to school full time online and I am a stay at home mom. I cannot see a basis for him taking her from me based upon the fact that he does not get to see my daughter as much as he would like but I feel that it is his fault because he never makes the necessary steps to come get her per the divorce agreement. Do I have anything to worry about?