I used kubespray, seems calico settings not set correctly for firewalld.

If you enable debug mod:

firewall-cmd --set-log-denied=all; firewall-cmd --reload

# Then check kernel logs for rejected packages:
dmesg | grep -i reject

you can see

[ 6295.597397] filter_IN_public_REJECT: IN=calib16e4fbb6b0 OUT= MAC=ee:ee:ee:ee:ee:ee:c2:ac:76:d7:9b:dd:08:00 SRC=10.233.65.20 DST=169.254.25.10 LEN=45 TOS=0x00 PREC=0x00 TTL=64 ID=47463 DF PROTO=UDP SPT=42342 DPT=53 LEN=25 MARK=0x50000
[ 6296.095563] filter_FWD_public_REJECT: IN=calib16e4fbb6b0 OUT=tun0 MAC=ee:ee:ee:ee:ee:ee:c2:ac:76:d7:9b:dd:08:00 SRC=10.233.65.20 DST=1.0.0.1 LEN=45 TOS=0x00 PREC=0x00 TTL=63 ID=20762 DF PROTO=UDP SPT=50598 DPT=53 LEN=25 MARK=0x10000

Recreating zone with correct source mitigates the issue:

firewall-cmd --permanent --delete-zone=kubernetes_pods
firewall-cmd --permanent --new-zone=kubernetes_pods
firewall-cmd --permanent --zone=kubernetes_pods --set-target=ACCEPT
firewall-cmd --permanent --zone=kubernetes_pods --add-source=<IP>/<SUBNET>
firewall-cmd --reload

It tooks solid 3 hours of mine, good part I am much comfortable with firewalld :)