Hmm, due to the absolute install_name specified by LC_ID_DYLIB in libswiftCore.dylib in toolchains, Swift executables always load libswiftCore.dylib from absolute /usr/lib/swift/libswiftCore.dylib without checking @rpath. So just adding -rpath linker option did not work unfortunately.