If you don't have a chimney cover or it is rusted out and the water has been leaking every time it rains it is likely the metal chimney stack is rusted out and the chimney insulation is damaged. Dangerous not to replace it. The only way you are going to know is by digging in and exposing it. You will be able to tell right away just by looking at the chimney stack.
If your chimney is brick with a metal stack the water is also going to break down the mortar of the brick and that will also need to be repaired.
https://www.highschimney.com/reasons-chimney-leaks/