Pretty Ballycastle is a great destination on our northern coastline! I hope I can help you with planning your way from Dublin to Ballycastle.
On our website you can find an article about travelling with public transport
around the Emerald Isle - including links to the comprehensive National Journey Planner as well as Bus Éireann, Irish Rail and Translink NI Railways. These are all quite handy for planning your journey as well as checking live arrival and departure times.
More specific about your planned route: there is a 10.00am train from Dublin Connolly to Belfast Lanyon Place and from there a connecting train to Coleraine. From here there are several buses to Ballycastle. The journey takes a little over five hours. It is of course also possible to take a bus to Belfast and onwards - this also depends a little on your preferred mode of travel.
I hope this is helpful and that you can look forward to an enjoyable journey to Ballycastle!