When you start a torrent download, you usually send an announce request to a tracker, so it will see your real IP. If you want to stay anonymous, you need to use either a VPN or an anonymity network such as TOR. Unfortunately, not all torrent users are aware of this.
In the beginning, HTTP was used to connect to a tracker, but the use of TCP/IP as its carrier protocol meant the packets incurred considerable overhead. In order to speed up trackers, a variant of the UDP protocol has been used instead. However, because UDP is seldom used, it is not always supported by current VPN solutions such as TOR. This means that UDP packages will not be sent through the VPN and therefore leak your IP address.
Imagine that you spend hours setting up your system to be as anonymous as possible and after all that hard work your torrent client injects your real IP into every packet it sends to a tracker. This is really frustrating, but you can avoid it by specifying an IP address that the tracker should use to communicate with your computer.
The IP/Hostname field allows you to specify an IP address when you report to a tracker. This is used whenever your WAN (Internet) address is not reported correctly to the tracker for a number of reasons (e.g. when you are behind a proxy server). You can also specify a hostname in this field, so if you are using a dynamic DNS service, you can input your domain name. Unfortunately, many trackers ignore this information, in which case setting this option will have no effect.
Some clients always include an IP to prevent failing handshakes in the presence of a proxy, VPN or TOR network. The specified IP, either local or for a VPN endpoint, depends on the client and the network configuration. If your system uses a dual stack (i.e. IPv4 and IPv6) with a preference for IPv6, as is common with many operating systems nowadays, the client will likely use the IPv6 address.
As with anonymous Internet surfing, DNS configuration is generally neglected. If your client uses your ISP's DNS server, a malicious tracker can detect your ISP. However, this requires some effort, since the attackers would need to run their own DNS servers.
Using a public DNS server (e.g. from Google) might also leak some information via edns-client-subnet queries (see here: https://00f.net/2013/08/07/edns-client-subnet/).
Also be aware that some torrent client do have a built in fallback DNS Server configuration.
In any case, you are obviously not affected if you use an IP instead of a hostname to contact a tracker.
The DHT mechanism uses UDP, so if your VPN does not support UDP, you are leaking your IP if DHT is enabled. Using only HTTP-based trackers will not prevent this leak.
A lot of TOR users use a proxy only for communication with a tracker, while communication with peers is direct in order to avoid the performance penalty imposed by the TOR network. This means that your public IP is available to other peers and could be used maliciously. For example, an attacker could use the usually random port numbers to relate your VPN's IP to your real IP or trick you into connecting to his own machine. One possible solution is to route all peer traffic over TOR as well, however this is not a good idea (see https://blog.torproject.org/blog/bittorrent-over-tor-isnt-good-idea). The use of a socks proxy is recommended instead.
The six leaks we described above are only some of the easiest to exploit. Essentially, inspecting torrent requests and/or running a DNS server is all an attacker has to do. But there are many more ways in which the anonymity of torrent users can be hacked. Such attacks require more effort, as attackers need to actively scrape DHT traffic and/or peer information, or run a malicious TOR exit node. For further information on these types of attacks, check out "Compromising Tor Anonymity Exploiting P2P Information Leakage" (https://hal.inria.fr/file/index/docid/471556/filename/TorBT.pdf).