That Time I Built an MSIX Package from Scratch

A bit of background.

For the past few years I’ve worked with a client to build an app in Electron. When we started the project, I was the only one with Electron experience, so I was put in charge of all things Electron. Since most of the app’s functionality resides in a web app displayed inside the Electron app, most of my work was initially around creating installers. This expanded into taking the lead on automating that process, which morphed into being the dev-ops guy.

Two years later we’ve got CI/CD run through an AWS CodePipeline which reaches out to a Jenkins server on Mac Mini to build a DMG, and another Jenkins server on a Windows VM to build an NSIS (exe) installer for Windows. The pipeline also updates Cloud Formation Stacks that manage static websites to host our Chromebook app and a download site for the installers.

We’ve actually got several pipelines, one per environment, with each environment tied to a specific Git branch. The app also auto-updates by checking version information in a file on our download site.

The problem.

The problem is really the intersection of multiple things about the app that I didn’t really account for, or even think to account for:

1. Our app is often installed on managed devices. System admins don’t want an NSIS installer, because then they have to manually convert it to an MSIX package and deploy it to their system.
   
   
2. The auto-update system works by replacing files in the installation folder, but files installed via an MSIX package are read-only, so if a system admin creates an MSIX package, and we deploy an update, the app blows up.
   
   
3. System admins really don’t like giving users write permissions in the Program Files folder. If the user can’t write to our app folder, inside the Program Files folder, and we deploy an update, the app blows up.
   
   
4. We can’t let you use an old version of the app, because it connects with backend services that might have had API changes, and our system isn’t mature enough yet to have things like versioned APIs.

This means that system admins hate me. I get it, I completely ignored what they needed, but this is only the second desktop app I’ve shipped, and the first one that was ever marketed towards enterprise customers. I know for next time.

The solution.

We ran through a couple of ideas about how we could solve the problem. We couldn’t let you use the app unless you were on the latest version, so swallowing the errors from the auto-update system was out. We did a quick fix where we tell you you need to have your system admin update the software if the auto-update failed because of write access errors, but that was just a temporary solution. What we needed was our own MSIX package, so I went to work learning exactly what an MSIX package was.

What is an MSIX package?

I always thought of Windows apps as subdirectories under the Program Files folder. Sure, you could compile something and run it from anywhere, but I assumed that Program Files was where Microsoft wanted you to put your app. In order to get it there, you needed an installer, which was just a program that copied all the files to a subdirectory of Program Files, tossed a shortcut on your desktop, and maybe edited the registry.

This is really insecure. There isn’t any reason why I can’t write an installer that deletes a file required by another app, or maybe stick something in that app’s directory that causes it to run differently. I don’t even need to write an installer. If I can get you to execute some malicious code, and you have the ability to install apps, then I can get that code to change all kinds of things to any app you have installed.

Microsoft’s solution to this was APPX, a signed zip file with an XML manifest file inside that the OS uses to install the app. The manifest describes what files need to be installed, what shortcuts need to be created, and what registry changes need to be made. The user wouldn’t have write access to the folder the app was installed in, so malicious code couldn’t easily modify it. For more security, Windows checks the app folder before it is launched, preventing said launch if changes are detected. APPX was originally just for Windows Store apps, but got re-tooled to support deployment to managed devices, and re-branded MSIX.

Making an MSIX package.

Armed with the knowledge of MSIX packages, and how they worked, I needed to figure out how to create one. After googling a bit, I found a few solutions:

The first was a tool Microsoft offers that would allow system administrators to create an MSIX package from an existing installer. This sounded easy, so I dove a bit deeper. To use this tool, you need to create a virtual machine with a fresh Windows install on it. Then you point the tool at it, and run your installer. The tool records all of the registry and file changes made, and composes a manifest file from those changes. It then produces an MSIX package with that manifest, and the installed files. This was interesting, and I did go through the process, so I could inspect the package and get a better feel for what one should look like, but I wasn’t going to be able to automate this in our CI/CD pipeline, so I needed to move on.

Next, I found a few tutorials on how to build a package from Visual Studio. Since we don’t use Visual Studio, and this wouldn’t work in our pipeline, I didn’t explore any further.

Finally, I found some documentation on creating MSIX packages inside a CI/CD pipeline. The process involved using a command line program called MakeAPPX, which was one of many utilities included in the Windows SDK. After downloading the SDK, and finding the utility, I went to work figuring out how to use it.

Preparing things for MakeAPPX

When you use MakeAPPX, nothing is done for you. You need to create a manifest file, and you need to figure out a way to tell the utility what files should be included in the package. I had the generated manifest from the previous MSIX package I created, so after looking at the documentation on what the manifest should contain, and comparing that to the generated manifest, I was able to put something together that I thought would work.

I should note that I scripted the manifest creation. I would actually need to script the whole thing, because we build distinct apps for each of our environments. Early on, we discovered the need to install apps built from different branches on one machine. Without distinct app names and identities, they would end up writing data to the same AppData folders, causing bizarre issues. The solution was to give each environment a distinct app identity, and name, so you could install them side-by-side without conflict. This meant that the manifest file would need to be generated, so the environment specific bits could be populated.

Next, I needed to sort out how to tell MakeAPPX what files need to be included. There was an option to simply give it a folder path, but that seemed a bit ham-fisted. We’ve been using Electron Builder for our installers, which creates a folder with all the required files during the build process, but I wasn’t sure at this point if all of those files were required, or if I would need to add more to support MSIX. The second option I came across was a mapping file, which was just a text file that mapped file paths on your harddrive to paths inside the package. This seemed like a clean solution that was easy enough to implement, so I wrote a script that iterated over the files generated by Electron Builder and added an entry in the mapping file. This option actually worked out well, because I soon realized I needed to add icons to the package at paths matching those configured in the manifest.

I then put everything together in a script that generated the mapping file and the manifest, and passed them to MakeAPPX. After a few adjustments (many adjustments), to the format and content of the manifest and mapping files, I finally got a zero exit code. I had created an MSIX package, time to celebrate! Almost anyway, I needed to sign the package for Windows to install it, so I had a bit more work to do.

Using SignTool.

Signing things in Windows is done with a utility called SignTool, which is also available in the Windows SDK. Even MSBuild uses it when it’s signing things for you. There isn’t a dll somewhere, all bits of software that sign something just wrap calls to this utility. I need to do the same thing.

First, I needed a certificate. When we sign our current Windows installer in our pipeline (a process taken care of by Electron Builder), we use a cert that is tied to a USB dongle. You don’t have to do it this way, but if you don’t, windows often displays a warning when installing your software that it shouldn’t be trusted. This warning was unacceptable to the powers that be, so it was decided to use this USB dongle. As such we don’t build our Windows installer in AWS. Instead, AWS calls out to a Jenkins instance running on a physical machine in a datacenter, with the dongle plugged in. I couldn’t use that certificate, so I created a self-signed cert in Powershell, and made it trusted on my machine.

Certificate in hand (so to speak), I added a bit to my ever-growing script that would call out to SignTool after it created the MSIX package. Without the need to fool with mapping or manifest files, this process was pretty quick. Finally, I had a signed MSIX package, and I was able to install it.

Bypassing Auto-Update.

The first thing our app does on launch, is to reach out to our deployment server and make sure it’s up to date. If not, it downloads the latest version, installs it, and restarts itself. I knew this was going to be a problem, MSIX package files are installed read-only. I tested my theory. I rolled back the version number locally, built another package, and installed it. Sure enough, after downloading the new version of the app, it blew up, unable to overwrite the files.

Microsoft has a solution for this. You can specify a URL in your manifest that points to another XML file that it will check before launching the app. The remote file should contain information pertaining to the latest app version, and you can have Windows download and upgrade the app prior to launch. I’ve got a ticket to implement this, but first I needed our QA folks to be able to make it into the app, and test it.

We already had a code path that would allow you to bypass auto-update, but it requires a build flag. When I originally designed it, I didn’t want end-users to be able to bypass the update code, either intentionally or accidentally, and I still didn’t, so I couldn’t change it to use a command line flag. I dove into the documentation on the manifest, to see if I could sort out a way for the app to “know” it was installed via MSIX, but I was coming up short. Originally, I thought I could check the install path, but MSIX packages are run in a sandbox. When your app asks the OS where it’s run from, it’s told that it’s inside Program Files. This is probably for backwards compatibility reasons, but it didn’t help my cause. I couldn’t use the build flag, because then I would need to build the app twice. Currently, I was just using the build output from Electron Builder, and I would need to make sure that it didn’t package the second build into the existing NSIS installer. This seemed overly complicated, and inefficient.

I finally imagined a simple solution. I would create an empty file at build time and add it to the package. I could then check for the file at runtime and, if it existed, I could bypass the update. This only needed to work until I got the MSIX update implementation done, so it seemed fine for now. After a small modification to the update code, I was able to bypass the auto-update functionality for MSIX packages, and was nearly ready for a PR.

Thinking About the Toolchain.

Now that I had a working solution, I would need to clean it up. I try to review the things I write before I submit a PR, just to make sure the i’s are dotted and the t’s are crossed. It’s also when I update the readme so that other developers can figure out how the solution works. There was something that bothered me. Anyone who wanted to build an MSIX package was going to need MakeAppx and SignTool. So I’d need to write up something on how to download the Windows SDK, find those files, and add them to the PATH environment variable so that my script could find them. I would also need to do this on the build server.

It dawned on me that everyone already had access to these utilities. Electron Builder supports APPX packages, and signing installers, so it must access these utilities somewhere. After a small dive into Electron Builder’s source on Github, I discovered that it downloads the utilities and sticks them in your AppData folder after it’s installed.

I updated my script to use those versions of the utilities, and ran it. Luckily, the version of MakeAppx that Electron Builder uses supports generating MSIX packages, so this worked. It also saved me time, and lowered the barrier for any other developers on the team that might need to work on this process in the future.

It Works on My Machine.

After a long, arduous, journey I was ready to submit my PR. It was merged into our development branch, and I waited a bit for the pipeline to do it’s thing, just in time for lunch. After my midday respite, I check the S3 bucket that should contain my newly minted MSIX package. Nothing. I logged in to AWS and looked at the pipeline. The Windows build task failed. I have a cardinal rule, don’t break the build. I built my own dev-ops environment where I test changes to the pipeline and the AWS stacks specifically to avoid situations like this, but I didn’t think I needed it. I checked the logs in Jenkins to see if I could sort out the issue.

After scanning the logs, I found out what was failing. An internal SignTool error occurred. I wasn’t given any explanation, only a hex error code. Great. I merged another PR that skipped my MSIX script, so the pipeline would start running again, and moved everything over to my dev-ops environment.

I pasted the error code into google, and hoped for the best. I was getting a bit irritated. I had found several forum posts referencing the error code, that provided a link to an explanation and potential fixes in a Microsoft Knowledge Base article. Unfortunately, that article no longer existed.

Eventually, I was able to find current documentation on the error and an explanation. The certificate subject name in my manifest file must match the actual certificate subject name exactly. When I created my self-signed cert, I only used the Common Name (CN) attribute in the Subject Name, and that’s all that was referenced in the manifest. Usually this is enough to identify the certificate in the store. In fact, I use the same string when I tell Electron Builder what cert to use when signing the NSIS installer. It seems SignTool wasn’t looking for a way to find the cert, but instead wanted to validate it.

I remoted into the Jenkins machine, opened the signing cert, and copied the full Subject Name value into my manifest. After pushing the changes through the pipeline, I got a new error. I didn’t even make it to SignTool, now I couldn’t even create an MSIX package.

Thankfully this new error was less cryptic, if still problematic. MakeAppx validates the manifest file, and runs several attributes, including the one referencing the certificate subject name, through a regular expression. The type of long, complicated regular expression that makes you tear up a bit when you realize you need to figure out what it matches.

It turns out that it doesn’t want an exact match for the certificate subject name, it wants all of the attributes defined in the certificate subject name, formatted in a specific way. To be honest, I don’t know a lot about certificates. I know that they have a subject name field with attributes that describe the organization that owns the certificate, including what state or country it’s in, and it’s name. Mostly I deal with self-signed certificates with only a Common Name attribute.

Our official code signing certificate had all of that and more. It had attributes with names that were a series of numbers separated by periods. This was especially worrisome, because the regular expression wouldn’t accept them. I tried omitting those attributes. This only led to the previous SignTool failure I had seen before.

Eventually, I searched for one of the numbered attribute names, and discovered that it corresponds with something called an OID, or Object Identifier, which is just an internationally standardized convention to name “things”. Each code corresponds to a particular type of information. I went back to the regular expression. It wouldn’t accept a series of numbers concatenated with periods, unless those numbers were prefixed with “OID.”. I made the required changes to my manifest attribute, and pushed out another commit.

Finally, I had the MSIX package building in the pipeline. I PR’d my changes to the development branch, and with relief, noticed that it worked in that environment as well. Next, I’ll figure out how to implement an auto-update process for the MSIX package.