The most likely reason is that your client does not have a public IP address, but a private address behind Network Address Translation (a very common setup). A common private addresses are of the form 10.X.X.X or 192.168.X.X.
NAT device replaces private address in a packet from your client with its IP address, which is public. For the outside world it seems that the traffic originates from the NAT device. The main advantage of NAT is that it allows many clients to use a single public IP address (IP addresses are sparse). But NAT introduces a problem: an incoming packet needs to be routed to a correct client, but it does not have a client IP address. Some other strategy needs to be used to recognize to which client a packet should be routed and such routing is not always possible.
For example, the most well known limitation of NAT is that you can't simply start a listening TCP server behind a NAT and then connect to it from the outside world. The reason is that NAT device has no idea that your server listens on a given port and thus, it has no way to known that TCP SYN packets from the outside world need to be passed to your client. A workaround for this is to explicitly configure the NAT device to route SYN packets directed to a given port to a specific client (port forwarding).
When a client behind a NAT initiates a TCP connection, a NAT device remembers state of the connection (client address, client port, remote address, remote port). Because of this, when a response arrives, the device knows to which client the response should be passed. When the connection is closed, the device discards state information. This is why your client can communicate over TCP.
But UDP is more difficult to handle because it is connectionless and stateless. When the device sees a UDP packet it does now known if a reply is expected and when, so not all NAT devices handle UDP traffic. Although there is a technique that allows to do it.
To summarize: the problem you are seeing is not C# specific. Setting IP address in packets from your server to the client IP address won't help, because it is most likely a private address that is not routable. To solve the problem you need to use a NAT device that can pass UDP traffic in both directions. But if you are developing a generic client to be used in many setups, you may want to reconsider if UDP is the best option. TCP connection forwarding is a basic feature that all NAT devices support, but some devices may not support UDP.