It’s not a stupid question. You absolutely can lie about the source address for a packet. On a local network segment (with no VLANs), it will probably arrive just fine. If it needs routing, it may go through something that says ‘hmm, packets from this subnet aren’t allowed to come from over here’ and drops it on the floor.
The important question is: and then what?
The main use for the source address is to allow the destination to reply. If you send a packet with a faked source address, any reply will go to the faked source address and not to you. Sometimes that’s useful. Some NAT tunnelling things used to rely on this doing terrifying things with intermediaries taking part of a TCP handshake.
In general, most connection-oriented things (TCP, QUIC) require some handshake and so you’ll end up failing to establish the connection if you fake the source. As an attacker, you need to compromise the target network stack with the first packet (which is sometimes possible) to get anything useful from stateful protocols. You may (if you can guess the sequence number) be able to inject a packet into the middle of a TCP stream, but generally that will just show up as a decryption failure in TLS (you are using TOS for everything, right?).
The more fun attacks rely on reflection. DNS, for example, is designed to be low latency and so is (ignoring newer protocol variants) a single UDP packet request and a single UDP packet response. With things like DNSSEC signatures (or entries with a lot of round-robin IPs), the response can be much bigger than the request and so you can send a request to a DNS server with a small request and a spoofed source, and the DNS server will reply with a big packet to your victim. DNS servers and other parts of network infrastructure have mitigations for this, but it’s easy to accidentally make a new protocol that provides this kind of amplification. QUIC has a rule that the first packet must be at least as big as its response (so sometimes requires padding) to establish a connection precisely to avoid this kind of issue.
If you think that’s scary, remember that networks are everywhere. Most busses are now actuallly networks. PCIe is a network and PCIe source addresses are used by the IOMMU to determine which devices can access which bit of the host memory. Where does the PCIe source address come from? The device puts it in the PCIe packet. It’s trivial for a PCIe device (including an external one connected via Thunderbolt) to spoof the ID of a valid one and initiate DMA to and from regions exposed to the other device. IDE and TDISP should mitigate these problems when they’re actually deployed (I don’t know of any shipping hardware that implements them yet. I think IDE is sufficient but it’s been a while and I don’t remember what things are in which spec).