Debugging Swift on Fedora Linux

If you'd like to hack on Apple's Swift Programming Language on Fedora, for, say, troubleshooting purposes or whatever, here are the steps I've been using:

Set up the environment and build a release version

First, I recommend using a container; I would share mine but I haven't been able to figure out how to do that with podman.

Basically you want to start with a fedora image in the usual way, but it's super important you pass --privileged to it like:

podman run -it --privileged fedora:34 /bin/bash

The container has to be set up to build RPMs; you need to install a bunch of stuff like so:

dnf install -y fedora-packager fedora-review vim git

Then using git, clone my repo:

git clone https://github.com/tachoknight/swift-lang-packaging-fedora.git

and check out the previous commit:

git checkout e3f2

that will put you at the right commit; there's a Swift 5.5-DEVELOPMENT version that I started working on as well.

Run the "justbuild.sh" script; it should get all the dependencies needed to build Swift. Depending on what kind of machine you're running, prepare for about three hours to build everything at minimum.

Time passes 😴

Installing Swift

Assuming everything is built okay, you should be able to run:

dnf install -y ~/RPMS/x86_64/swift-lang-5.4-1.fc34.x86_64.rpm

This will install Swift install /usr/libexec/swift. In that directory you'll see Swift's own directory structure (e.g. bin, lib). Great, Swift is installed but in release mode.

NOW we get to the debug part. The whole purpose of doing the stuff above was to get everything in place so we can start fiddling with the source without having to deal with patch files. Plus it also guarantees that everything is working before it gets all screwed up (for me anyway 🙃).

First we need to get rid of the existing binaries:

cd ~/rpmbuild/BUILD
rm -rf usr

Now we're going to set up Swift for debugging. Type:

cd ~/rpmbuild/BUILD/swift-source

Then type:

rm -rf binforpython

This removes some stuff that the spec file creates due to some historical issues around Python 3 and Swift still wanting Python 2. Anyhoo, now type:

vim ./swift/utils/build-presets.ini

On line 818, under the section [preset: buildbot_linux], you should see "release" on its own line. Change that to "debug". Save and exit the file.

Now type
cd ~/rpmbuild/SPECS

And here is where we're going to rebuild Swift in debug mode. It's very important this line be entered exactly as entered:

rpmbuild -bc --short-circuit ./swift-lang.spec

What the command above does is to simply recompile the code without re-downloading anything. It's important that this be the only command to use when rebuilding Swift because rpmbuild wants to scrub the ~/rpmbuild/BUILD directory clean so you'd lose any changes.

W A R N I N G

You thought it took a long time to build Swift in release mode? Double it. Maybe even triple it. Kick this job off before you go to bed; it will likely still be going in the morning (or evening, depending on your schedule).

Using the debug version of Swift

The new rpm has the debug version of everything, but I don't really bother doing that and instead simply copy the contents of ~/rpmnbuild/BUILD/usr into something like /usr/libexec/swift-debug, renaming the existing /usr/libexec/swift to /usr/libexec/swift-release and then create a symlink to the one you want to use (I like to keep the release version around as a santiy check).

Running swift in the debugger

If you want to run a particular swift tool, it's easy to run it in the debugger, which is included with the toolchain as /usr/libexec/swift/bin/lldb. So if you want to run the actual swift program in the debugger, type:

/usr/libexec/swift/bin/lldb /usr/libexec/swift/bin/swift

When building a package that requires swift build, that actually invokes the program /usr/libexec/swift/bin/swift-build. So debugging the "swift build" command would be:

/usr/libexec/swift/bin/lldb /usr/libexec/swift/bin/swift-build

Getting TimescaleDB running on Kubernetes for internal use

Introduction

As a fun use of the data ShopMon generates, I wrote a quick program that adds that data to a PostgreSQL database for looking at some interesting metrics, usage-by-hour, area, etc. While on a run I was listening to a podcast where the topic of time series databases came up and realized I had a perfect use case as the sensor data ShopMon creates is time-specific along with the location of the sensor. Hey, let's learn about time series databases!

Enter TimescaleDB

The first thing I did to get the lay of the land is to read the Wikipedia article on time series databases, which I found surprisingly thin but the discussion page turned out to have a lot more content and, surprisingly, more information as the authors and editors were arguing back and forth about what constituted a valid time series database and in the course of reading some of the more heated comments, learned about TimescaleDB and the first thing I saw was that it was a time series database built on top of PostgreSQL. Well, I love PostgreSQL and thought I'd give this a try; I have no practical experience with any other time series database so why not start with this one?

Hey, I built a Kubernetes Cluster!

I have recently come into posession of three retired Dell PowerEdge servers, two 2950s and a R900 (which, at 4U, is a beast to handle by oneself). I decided to dedicate these machines to running Kubernetes pretty much exclusively, so with ESXi installed as the hypervisor and with a bunch of VMs running Linux with Docker, I installed Kubernetes (specifically the Rancher distribution) and after deploying a couple of test pods figured that I would get my TimescaleDB database running on it. Oh, and there's even a GitHub repository for getting it running on Kubernetes!

Disclaimer

I'm using this Kubernetes cluster for learning and experimentation so I apologize in advance if there's something that screams "not-good-practice"; I'm still trying to figure out what those are (and what the best practices are).

Installation

The first thing I did was clone the repo and have a look around. I opted to use the "timescale-single" version as I figured to keep it as simple as possible initially.

Preparing the Chart

Apparently Helm charts have become the defacto-standard way to install software on Kubernetes and admittedly there's a lot to like in terms of standardizing deployments. TimescaleDB had an interesting "pre-chart" step that creates certificates and users for the database and puts them in the right spot for Helm to pick up.

Kustomize

Within the timescaledb-single directory, I entered:

./generate_kustomization.sh ztdb

where ztdb is the name of the Kubernetes workload (why ztdb? Zesty Time Database)

No LoadBalancer

One thing I'm kinda meh about is how it seems Kubernetes seems really geared towards being used in cloud environments (which, coming from Google, isn't exactly a huge surprise) but in my case I have, essentially, my own private cloud where I don't want to make anything public.
So when a Load Balancer is specified, deployments essentially get hung up forever waiting for the cloud provider to give it a public IP that will never come. So, in values.yaml I went to the "loadBalancer" section and set enabled to false. According to the comment, this is cool to do and the primary node will be exposed as a Headless Service. Gonna admit right here that I get the idea of a headless service, but what that means for accessing the database from external tools was something to figure out later.

Persistent Volumes and Volume Claims

TL;DR: The Kubernetes TimescaleDB deployment requires six persistent volumes to be ready to be bound.

I set up a separate VM with a large disk to act as an NFS server (all the physical machines use RAID-10 and I installed a number of the VMs on an EqualLogic iSCSI-based SAN I found, also in RAID-10).

On my NFS server I created six directories and exported them so I could set them up as Persistent Volumes (this where I use the Rancher interface to do all the heavy lifting for me). The TimescaleDB deployment would create the volume claims all by itself, so I just provided the volumes and that was it.

Installation

Okay, so now to do the actual installation. In the the timescaledb-single directory I ran:

helm install ztdb . --set service.type=NodePort

The ztdb must match the name of the workload used in the Kustomize step above.

The chart will print out a bunch of information about getting the PGPASSWORD_POSTGRES, PGPASSWORD_ADMIN, and MASTERPOD info. You'll definitely want this later on so once the deployment is finished (it took about ten minutes on my cluster), run the commands and get those values.

Now for the sketchy part

This is where it get confusing; I was able to connect to the database via the headless server (all documented by the stuff printed to the screen during the deployment), but I need to connect to the database from my laptop or other machine via a "public" (i.e. non Kubernetes-pod) IP address; everything is LAN-based (in my case, a 10.150 network) and if I can't connect to the server via DBVisualizer or some other tool, then there isn't much point to any of this 🙃.

Get the IP of the node running the master

Remember during the installation when the chart told you how to get the $MASTERPOD? In my case, it prints pod/ztdb-timescaledb-0. Back in the Rancher interface, under Workloads, I have "ztdb-timescaledb" which, when clicked, shows the three pods, one of which is "ztdb-timescaledb-0". Clicking on that pod brings up info about it, and one of the things we learn is what host it's on and more importantly, the host's IP address (in my case, 10.150.9.107).

Fiddling with Services

Rancher, interestingly, has two interfaces, the "Cluster Manager" interface which I guess is meant to be more informational, dashboard-y, and the "Cluster Explorer" interface which is where you can really get your hands dirty (and presumably do a lot of damage). In the Cluster Explorer interface, select Services and find the service with the same name as the deployment, in my case ztdb (there will also be, for me, ztdb-config and ztdb-replica but we don't need to fiddle with those).

Selecting "ztdb", the details of the service are shown, including a link to "Listener IPs". There is the Cluster IP and nothing else. Well, we can't connect to the Cluster IP so we're going to add an External IP.
Clicking on the Three Dots™ button in the upper right and selecting "Edit as Form", the first thing you'll see is a warning that the service is managed by the Helm app and that changes will be overwritten the next time the app is changed. Okay, that's fair, duly noted. Click on "Listener IPs" and click Add under External IPs, this is where we add the IP address of the node running the master (note: it may be possible to try use other Node IPs, I haven't tried that one yet).

Click Save and now we have an external IP that can access the database. Ah, but we're not quite done!

Setting up DBVisualizer et al

So now we should be able to connect to the database using database tools. First thing besides the IP address, what's the username and password? The password for the postgres user is defined via PGPASSWORD_POSTGRES as was mentioned during the installation. This is going to be a random string so might as well copy and paste it into the password field. The port is the standard PostgreSQL port of 5432 and if you attempt to connect, you'll see that it won't work because it is refusing non-SSL connections.

Setting up the SSL connection

SSL is enabled by default in the values.yaml file as part of the database configuration. You can see it's also part of the connection parameters where hostnossl is set to reject. All this can be changed beforehand, but since we have a database that only wants to talk via SSL, we will need to use the SSL certificate.

Where is the certificate? Back in the chart directory, there is a subdirectory called kustomize that has another subdirectory that is the name of the deployment (again, for me, ztdb). In that directory you'll find the tls.crt file that was generated and that is the SSL cert you'll need for all connections to the database.

Setting up the connection to use SSL varies from tool to tool; in my case I want to use DBVisualizer so under the connection properties there is a "Driver Properties" entry that has all the various driver properties that typically don't need to be set in regular ol' configurations. In our case, sslmode needs to be set to verify-ca and sslrootcert is where you set the path and filename of the certificate contents (it doesn't need to be tls.crt).

With these two properties applied, the tool should now be able to properly connect to the database and you're good to go!

Extending LVM size on Linux

Introduction

I built a Kubernetes cluster for running parallel Jenkins builds for Swift and realized that the disk size I had selected for the worker nodes was woefully insufficient; all my jobs would be evicted simply because of disk pressure.

I used Ubuntu 20.04 for the builders and so by default an LVM volume is created which is nice because then I can extend its size instead of dealing with mounting a regular disk and dealing with setting up the directory and all that in the /etc/fstab file.

Here are the steps. I'm using Ubuntu 20.04 but I presume the info is relevant regardless of Linux flavor.

Step 1 - Add another disk to the machine

In my case my machines are running under VMWare ESXi. I thought to include a bunch of screenshots but in the end this step is: install the new disk somehow. Good luck with that.

Step 2 - Log into the machine and go root

Standard sudo su - here.

Step 3 - Scan for the new disk

The new disk won't be visible until you scan for it, which is easily done via:

for host in /sys/class/scsi_host/*; do echo "- - -" | sudo tee $host/scan; ls /dev/sd* ; done

This will roll through the SCSI buses (I had 25? WTF?) and eventually it found /dev/sdb. If there is already a /dev/sdb on your machine it'll be /dev/sdc etc.

Step 4 - Create a new Physical Volume on the disk

Enter:

pvcreate /dev/sdb

and if you run lvmdiskscan -l you'll see the new disk in the list.

Step 5 - Add the Physical Volume to the existing Logical Volume

The name of my Volume Group is called ubuntu-vg. You can find out what the VG is called by running lvdisplay and taking note of the LV Name parameter.

So now you can add the PV to the LV by entering:

vgextend ubuntu-vg /dev/sdb

And you should see a message that the volume group was extended.

Step 6 - Extend the Logical Volume

Now we're gonna tell the logical volume to use all that extra space we just gave it:

lvm lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv

Step 7 - Now actually let the LV use the space we just allocated

Ah, but here's the critical last step, the filesystem (in my case EXT4) needs to be updated to include all the new space:

resize2fs -p /dev/ubuntu-vg/ubuntu-lv

And we're done

You should be able to run df -h and see the LVM now has all that additional space; in my case I added 50G to each node and now I really do have the space for my Swift builders. Yay!

Get and Put Files With NextCloud

To put a file into NextCloud using curl:

curl -u USERNAME:PASSWORD -T THE_FILE_YOU_WANT_TO_UPLOAD https://yournextcloudinstance.com/remote.php/dav/files/USERNAME/

and to get a file:

curl -u USERNAME:PASSWORD https://yournextcloudinstance.com/remote.php/dav/files/USERNAME/THE_FILE_YOU_WANT_TO_DOWNLOAD --output THE_FILE_YOU_WANT_TO_DOWNLOAD

Basically a reference for myself. 🙂

Swift on Fedora, All Good

Re-reading my previous post about the difficulties in keeping Swift playing nice with other software in /usr/bin, I came up with a different idea that worked extremely well when I tested it on my local machine: install the entire Swift toolchain in /usr/libexec/swift. The rational is this: The Swift toolchain is very sensitive to locations of files and libraries; executables assume they can invoke other executables in the same directory, and there are explicit references to libs located in ../lib.

So instead of trying to fight it, I opened an issue with the packaging committee to allow the package to be reconfigured to:

Install the entirety of the Swift toolchain in /usr/libexec/swift
Symlink swift, swiftc, and sourcekit-lsp from /usr/libexec/swift/bin to /usr/bin
Remove the swift-lang-runtime package

The last issue is because there is no guaranteed ABI stability for the Linux version of Swift and I felt it would be better to not pretend that would work as an update to swift-lang would also update swift-lang-runtime as well, as there are no weak dependencies between the two packages. Thus I feel it was more prudent to consolidate everything under swift-lang.

This setup has worked so much better; I was able to happily remove a whole set of patches to fiddle with the source to change where to look for other programs, and maintaining Swift has become a lot easier insofar as I don't worry about whether it's my rather invasive changes that are breaking everything. This has been working through multiple versions of 5.2 and, as of this writing, has been working for 5.3-dev as well.

Swift 5.2 and Fedora Linux – REPL issues

Swift 5.2 (and quickly followed up with Swift 5.2.1) were released in late March and, while it is packaged up and ready-to-go as an RPM for Fedora/CentOS/RHEL users, it's not "ready to go" insofar as not all the functionality works. Specifically, while the package does compile Swift code, the REPL does not work. I've been trying to discern why for a long time and, while I have figured out how to make it work, it would not pass muster as a distro-provided RPM. At the moment I'm not quite sure what to do and would appreciate any feedback on how to proceed.

Prelude - What's going on here?

I have a public repo that contains all the scripts, patches, etc., that is used to build Swift for Fedora (along with CentOS and RHEL, but I'm just going to keep saying Fedora; assume the other two distros as well) and I have been tracking 5.2 development in its own branch while 5.1.x was still considered production. On first blush, 5.2 didn't have many significant changes over the 5.1.x versions, but while it was relatively easy to get the entire toolchain to compile and build, the REPL in 5.2 was broken from the very start.

"Broken" is a loaded word and I should note that, built as-is without any patches to relocate Swift executables, everything works fine. Hunky-dory. Woo. For background, the Swift toolchain, as provided by Apple for Ubuntu, for example, has its own usr directory structure (e.g. usr/bin,usr/include, etc.) and is meant to be extracted, as-is, under a "safe" location like, say, /usr/local/swift and then set the path to include /usr/local/swift/usr/bin and congratulations, you have a working Swift installation. Unfortunately, Fedora forbids packages installing their files into /usr/local and as a consequence, has to live in the regular /usr where it can hobnob with other installed packages like Vim, cat and Clang.

Ah, and there's the issue. Clang, LLVM, and LLDB can be installed independently and in a case of who-got-there-first (spoiler: not Swift) those packages have a pretty clear lock on the executable names. Clang itself isn't the issue, it's LLDB with its accompanying lldb, lldb-server, etc., that can live in /usr/bin and since Swift provides its own custom lldb for the REPL, attempting to have both installed doesn't work. So, as part of the packaging of Swift for Fedora, I discovered that swift looks for lldb in the same directory. This patch, which I've used since the beginning of the Swift package for Fedora, changes the location to /usr/libexec/swift-lldb so as to keep Swift's lldb out of the way of the official LLDB package. More-or-less, that was the toughest nut to crack; the second more difficult aspect of packaging Swift was preventing the .so files Swift places in usr/lib in being installed in /usr/lib, as they are technically 64-bit and should go in /usr/lib64, but attempts to put the files there did not work (I forget the exact reasons, it was awhile ago).

Where We're At In FedoraLand

So that brings us to Swift 5.2.1. As mentioned above, compiling Swift programs works fine, but if you want to use the REPL, you are greeted with error: Could not construct an expression context for the REPL.. After many experiments (a fair number of Docker containers were used as there it's safe to really muck around in /usr/lib etc.) I have determined how to "fix" the install to make it work (if you want to follow along, you can get the container here):

1. 1. 1. 1. Move the lib* files from /usr/lib/swift into /usr/lib
        
        Run ldconfig to rebuild the cache
        
        Run ln -s /usr/lib/swift/clang /usr/lib/clang. This was gleamed by examining previous patches, the Swift source, and trial and error. It is our first "breaking" change insofar as Clang, if installed, would have created /usr/lib/clang already.
        
        Try running swift now. You will get a new error: error: REPL executable does not exist: '/usr/bin/repl_swift'. This is surprising, the package puts lldb in /usr/libexec/swift-lldb and I figured that it would look for companion executables, similar to how swift does it, in the same directory (and that is how it works in Swift 5.1.x).
        
        Okay, well, let's do a symlink: ln -s /usr/libexec/swift-lldb/repl_swift /usr/bin/repl_swift. Yay, now we have a new error: error: failed to launch REPL process: process launch failed: unable to locate lldb-server.
        
        Right, so, let's do the same thing again: ln -s /usr/libexec/swift-lldb/lldb-server /usr/bin/lldb-server. Now we're really in a pickle because lldb-server is an existing LLDB executable, so there's no way packaging the file in this location is going to work.
        
        Try swift again, and behold!
        Welcome to Swift version 5.2.1 (swift-5.2.1-RELEASE). Type :help for assistance. 1>

The REPL now works! I've run through a couple of test suites and everything works as expected, with all the nice changes 5.2 brought to the language.

Well, That's Nice. But What's Wrong?

This is where I'm stuck. The *.so files, as packaged for Fedora, are in /usr/lib/swift-lldb and I have an ld conf file that is installed to make them findable; indeed, if a file is renamed or moved, trying to run swift complains that the .so file can't be found (which is the correct behavior).

It seems like we're back to programs looking for files in hard-coded places, but for library files? Seems...unlikely. But yet after many attempts to make it work and keep the files in /usr/lib/swift-lldb, I've made zero progress. If anything, the library files are found in the right place, if you believe the output from strace (which has been invaluable for troubleshooting) but it's only when they're moved to /usr/lib, per the steps above, that things start to work. Then there's the surprising insistence on looking for repl_swift and lldb-server in /usr/bin; no amount of cajoling can get lldb to notice them, sitting happily next to it, in /usr/libexec/swift-lldb.

So What Now?

Well, I think my go-at-it-alone approach is not working anymore; a lot of time has been spent with little progress. I'm hoping that Linus's Law can apply here and someone sees what I've been missing and we can get a working REPL that doesn't intrude on other packages and ship 5.2.1 to Fedora users.

Strange issue building Swift 5.1 on RHEL 8

Something to look into:

Building Swift 5.1 on a RHEL 8 box with clang-7.0.1 I keep running into a strange issue where it fails about 3/4 of the way through the build with the error:

 FAILED: src/swiftDispatch.dir/Dispatch.swift.o src/swiftDispatch.dir/Dispatch.swift.swiftmodule src/swiftDispatch.dir/Dispatch.swift.swiftdoc
cd /home/rolson/rpmbuild/BUILD/swift-source/build/buildbot_linux/libdispatch-linux-x86_64/src && /home/rolson/rpmbuild/BUILD/swift-source/build/buildbot_linux/swift-linux-x86_64/bin/swiftc -frontend -module-name Dispatch -module-link-name swiftDispatch -I /home/rolson/rpmbuild/BUILD/swift-source/swift-corelibs-libdispatch -O     -Xcc -fblocks -Xcc -fmodule-map-file=/home/rolson/rpmbuild/BUILD/swift-source/swift-corelibs-libdispatch/dispatch/module.modulemap -emit-module-path /home/rolson/rpmbuild/BUILD/swift-source/build/buildbot_linux/libdispatch-linux-x86_64/src/swiftDispatch.dir/Dispatch.swift.swiftmodule -emit-module-doc-path /home/rolson/rpmbuild/BUILD/swift-source/build/buildbot_linux/libdispatch-linux-x86_64/src/swiftDispatch.dir/Dispatch.swift.swiftdoc -o /home/rolson/rpmbuild/BUILD/swift-source/build/buildbot_linux/libdispatch-linux-x86_64/src/swiftDispatch.dir/Dispatch.swift.o -c /home/rolson/rpmbuild/BUILD/swift-source/swift-corelibs-libdispatch/src/swift/Block.swift /home/rolson/rpmbuild/BUILD/swift-source/swift-corelibs-libdispatch/src/swift/Data.swift -primary-file /home/rolson/rpmbuild/BUILD/swift-source/swift-corelibs-libdispatch/src/swift/Dispatch.swift /home/rolson/rpmbuild/BUILD/swift-source/swift-corelibs-libdispatch/src/swift/IO.swift /home/rolson/rpmbuild/BUILD/swift-source/swift-corelibs-libdispatch/src/swift/Private.swift /home/rolson/rpmbuild/BUILD/swift-source/swift-corelibs-libdispatch/src/swift/Queue.swift /home/rolson/rpmbuild/BUILD/swift-source/swift-corelibs-libdispatch/src/swift/Source.swift /home/rolson/rpmbuild/BUILD/swift-source/swift-corelibs-libdispatch/src/swift/Time.swift /home/rolson/rpmbuild/BUILD/swift-source/swift-corelibs-libdispatch/src/swift/Wrapper.swift
<unknown>:0: warning: missing submodule 'CDispatch'
/home/rolson/rpmbuild/BUILD/swift-source/swift-corelibs-libdispatch/src/swift/Block.swift:13:8: error: no such module 'CDispatch'
import CDispatch
       ^
[60/91][ 65%][5.922s] Generating swiftDispatch.dir/Time.swift.o, swiftDispatch.dir/Time.swift.swiftmodule, swiftDispatch.dir/Time.swift.swiftdoc
[61/91][ 67%][6.216s] Generating swiftDispatch.dir/Data.swift.o, swiftDispatch.dir/Data.swift.swiftmodule, swiftDispatch.dir/Data.swift.swiftdoc
[62/91][ 68%][6.360s] Generating swiftDispatch.dir/Wrapper.swift.o, swiftDispatch.dir/Wrapper.swift.swiftmodule, swiftDispatch.dir/Wrapper.swift.swiftdoc
[63/91][ 69%][6.724s] Generating swiftDispatch.dir/Source.swift.o, swiftDispatch.dir/Source.swift.swiftmodule, swiftDispatch.dir/Source.swift.swiftdoc
[64/91][ 70%][6.967s] Generating swiftDispatch.dir/IO.swift.o, swiftDispatch.dir/IO.swift.swiftmodule, swiftDispatch.dir/IO.swift.swiftdoc[65/91][ 71%][7.313s] Generating swiftDispatch.dir/Block.swift.o, swiftDispatch.dir/Block.swift.swiftmodule, swiftDispatch.dir/Block.swift.swiftdoc
[66/91][ 72%][9.094s] Generating swiftDispatch.dir/Queue.swift.o, swiftDispatch.dir/Queue.swift.swiftmodule, swiftDispatch.dir/Queue.swift.swiftdoc
ninja: build stopped: subcommand failed.
Building the standard library for: swift-stdlib-linux-x86_64
Running Swift tests for: check-swift-all-linux-x86_64 check-swift-all-optimize-linux-x86_64
swift/utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting
swift/utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting
error: Bad exit status from /var/tmp/rpm-tmp.Ffec07 (%build)

The funny thing is that if I go to ~/rpmbuild/SPECS/ and type rpm build -bc --short-circuit ./swift-lang.spec then it will pick up from where it left off, and finish completely! The problem is that does not finish building the RPM file, it just validates that the issue seems to be something weird with the version of Clang that ships with RHEL8; I haven't seen the problem on any Fedora machine and mock builds with EPEL8 seem to be fine (crosses-fingers).

Well, it happened…

...The Fedora project moved /usr/bin/python to point to /usr/bin/python3 instead of /usr/bin/python2. The immediate result is that Swift no longer builds on Fedora Rawhide because of, surprise, Python errors. This may be a lot of work, so before running off and getting into a big mess o' Python changes, I asked on the Swift forums whether anybody is already working on it and, if not, well, guess it's time to put the build scripts on the blocks and do some major changes.

Swift 5.1 (Development) working with Fedora

So post-WWDC, Swift 5.1 seems to be in high gear for release, I presume, with Xcode 11. I made a few attempts at getting it to build and package properly and after some modest changes, removing outdated patches and a particularly scary Lua scriptlet in the spec file, it packaged correctly. Whee, go me.

However, while the compiler (swiftc) has always been reliably working, the REPL has always given me fits. In 5.1 the old clangloc.patch was invalid and I traced down the changes to lldb/source/Plugins/ExpressionParser/Clang/ClangHost.cpp and found that I needed to change the suffixes it was looking for in static const llvm::StringRef kResourceDirSuffixes[]. This was something of a pain because in the original version, the first thing in the list is lib, which I figured was okay, so I made the appropriate changes and....it didn't work.

I knew that I didn't have an lldb directory in /usr/lib, and in reality it was looking for the clang-specific files, which I had put in /usr/lib/swift/clang. On a hunch I added some printf()s in the function that read this array and, with lib in the array, the lldb executable, which controls the REPL, was looking for:

/usr/lib/lib/swift/clang

To verify that was, in fact, the problem, I spun up a Fedora Docker image, added the RPMs, and fiddled with the /usr/lib directory to mimic above and, hey, it worked! So the fix was easy, just remove lib from the array, and, sure enough, we have a working 5.1 REPL on Fedora:

CMake Weirdness with Swift 5 on Linux

The Swift toolchain uses CMake and Ninja. In a nutshell, CMake in a (much) fancier Autoconf and Ninja is a multi-core -aware make equivalent. I tried configuring CMake to use make once to see how different the performance was and, well, let's just say I let it run for about a week before I decided to give up because I wanted to use my computer.

I started getting strange errors when building Swift using mock; it was dying with references not being found in libXCTest.so and before that, warnings that the file itself was empty right before the errors began. Since Apple hadn't updated the Swift 5 snapshot for awhile (this was with swift-5.0-DEVELOPMENT-SNAPSHOT-2019-03-10-a) I figured there was some update Fedora had pushed that caused this. Since I hadn't updated my host machine, it still built fine, so knowing that something was going to break it when I did an update, I ran dnf -y update and in addition to some kernel updates was an update to CMake to version 1.14. Running the build again and, yep, sure enough, it failed with the same error.

The problem was that `/home/rolson/rpmbuild/BUILD/swift-source/build/buildbot_linux/xctest-linux-x86_64/libXCTest.so` was in fact 0 bytes. Teasing apart the CMake file I was able to determine that the file itself was being built correctly, but there was an additional step from the CMake file that had the newly created .so file being copied into the same directory it was already in, and this step was causing it to be zeroed out.

So, armed with the CMake source code, I investigated what code was being executed when running cmake -E copy foo.txt . and sure enough, it turns out that the file was being deleted and recreated, resulting in an empty file. I asked in the IRC chat room if this was, for some reason, expected behavior but did not receive any response. I decided to go ahead and make an attempt at a patch (hey, I've been making patches all over the place for Swift, why not CMake too?) and opened a bug report complete with the suggested fix. Well, turns out the issue is a bit more complicated in CMakeland and while the patch is not ultimately useful, at least the CMake folks are on it.

Here's hoping that once a new CMake version is available and ultimately updated in Fedora, there's one more patch that can be retired from Swift 5.x on Fedora.