By design, anyone who has access to your Home Wi-Fi network can cast to your TV. This makes devices with Chromecast built-in great for sharing videos between friends! If you want to prevent people from casting to your TV, make your Home Wi-Fi network password protected and only give out the password to people you trust.
Need more help? Ask other Chromecast users and top contributors in the Chromecast help forum.