Casting a tab’s audio requires a good Wi-Fi connection. You may get a better connection by moving your router closer to your Chromecast Audio.
If you wish to connect your Chromecast Audio without using Wi-Fi and have an available Ethernet connection nearby, you might benefit from using the Ethernet adapter.
If you have the option of using a wired connection, or if you have a dual-band Wi-Fi router and can use 5GHz on your computer, this may yield better performance. Please note that your Chromecast must be already set up and connected to your Wi-Fi network and AP Isolation/Client Isolation mode must be turned disabled on your router.
Other applications (e.g. BitTorrent) or devices on your network may reduce available bandwidth; minimizing these may help.