In most Linux distributions, there's a package manager ("PM") application we use to update our OS and any installed applications. The Arch PM is pacman, and full updates require an internet connection for its use. It's a relatively simple process:
# pacman -Syu
The problems begin with applications that have built-in package managers specific to that application. There's only a few, but they are often called upon and u. For example:
- LaTeX package manager =tlmgr (TeXLive)
- Git package manager = complicated unless at user level. If at user level, install git with pacman and run it as a user, it will just make a directory of source. Otherwise, git apparently downloads "clones" (versions) into /opt, but then generally you want to chmod the clone to foo:foo and move it to a ~/ directory for building, then pacman to install it.
- Python package manager = pip
- Wine package manager = complicated. Each dll installing in Wine attempts to update itself and will prompt for it. None should be authorized. Also, building a version of Wine that is custom configured for the MSoft app is critical, but the underlying Arch version of wine tends to wipe them out any time it is called.
Using any of these application-specific updaters within Arch leads to failures during the next "pacman". Don't use them at all except when knowing the workarounds (see below). Essentially, there are zero problems for users who install and update the Arch Linux OS and applications using only pacman. Again, the most reliable complete update command:
# pacman -Syu
...or for application removal...
# pacman -Rs
There's also one workaround in Arch that doesn't f*ck up the index: if I download source into some directory and run...
$ makepkg -si [somepkg]
... it will make the package, and then prompt me for the password for install ("-i" flag) at the end and perform a pacman -U [package] at the end of the make. This # pacman -U properly updates the pacman index. As far as I know, # pacman -U is the only way to install outside packages without farking an install. This also means we can reliably use # pacman -Rs to uninstall packages put in through this method.
1. LaTeX - tlmgr 5Gb
2022 edit: I now use pacman, until work on thesis or something, then can do a -Rsn uninstall and do it here.
Don't install any version of LaTex with pacman. It doesn't have enough templates.
Install Tex-Live (LaTeX) directly into a user subdirectory (eg. "/home/foo/latex") and update it through the TexLive PM (tlmgr), but only at the user level, so that no admin level changes, or files used by pacman are affected.
$ cd /home/foo/latex/bin/x86_64-linux
$ tlmgr update --self
$ tlmgr update --all
UNLESS, it's the next year and haven't updated. This is called a "cross-release" update. I had to download that year's update-tlmgr.sh and run it. Described further down in post and here.
2. Git - pacman + manually configure
This took me the longest to learn, as this site agrees. Configuring for upstreaming -- if using as VCS -- is time consuming, but it's different if downloading some source. Update the git app with pacman, update the sources using git commands. Here's the Arch page.
- install the foundational Git application using # pacman -S git.
3.Python - pip 2Gb
Nearly every program uses Python, so we want to be sure not to break our Python installation.
The solution required three independent solutions, all three of which I install. I select one of the solutions depending on what the application requires/expects.This solution takes up several gigabytes.
- initialize a Colab account in Google. See my post here, because there's a few steps to it. Colab handles Jupyter-type operations online. I do this development online (and save files .PY to Drive). I used to do Jupyter environment on the desktop, and this was the heaviest pip user, disrupting pacman almost entirely. IMO, Colab is practically a godsend from Google.
- when an application requires it, a pacman install of Python. Other applications installed using pacman, may also call for additional Python modules which they add to the basic Python, and/or they may also require an older version of Python to be installed.
- using pacman install a virtual environment for the
For
Python, we can use
pip, and for
LaTex we can use
tlmgr, to upgrade or add features to each app. The overall OS also has a package manager. For
Arch, this is
pacman. We can use
pacman to upgrade or add packages to the
Arch OS, including adding
Arch versions of
Python and
LaTeX. The problem is, any actions accomplished with
pip or
tlmgr are not recorded in the
pacman application index. Discrepancies between
pacman's index and
pip and
tlmgr operations lead to significant application and OS problems. Prevention requires decisions during installation.
ContentsThe
Python case is more complicated to *solve* than the
LaTeX situation, because a variety of applications depend on
Python, unlike
LaTeX. However, both situations equally *cause*
pacman update fails for the same, indexing, reason, so that both the
Python and the
LaTeX cases must be solved.
Python
Many applications rely upon Python. More technically, Python is a dependency for for many applications.
Imagine some application with a
Python dependency, say
youtube-dl. Imagine that both
Python and
youtube-dl were installed by
pacman. They both work fine. Now suppose the user decides to add a
Python module using
pip instead of
pacman. Everything might be fine with some applications that depend on
Python, but
youtube-dl is an example of an application sensitive to
Python changes. The
youtube-dl app looks for versions of
Python installed by Arch, but will instead detect versions updated by
pip. This discrepancy in turn leads
youtube-dl to spawn errors, and errors in turn lead to the application exiting. How can this be solved?
$ youtube-dl https://www.youtube.com/watch?v=QLpz7PtiP2k
Traceback (most recent call last):
File "/usr/bin/youtube-dl", line 6, in
from pkg_resources import load_entry_point
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3259, in
def _initialize_master_working_set():
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3242, in _call_aside
f(*args, **kwargs)
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3271, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 584, in _build_master
ws.require(__requires__)
File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 901, in require
needed = self.resolve(parse_requirements(requirements))
More drastically, this can happen with an entire OS. Say the user keeps his OS up to date with pacman, and decides to install Python using pacman. Later, he upgrades Python using Python's pip manager. Still later, the user attempts to update the entire OS, which is a pacman action. As part of its upgrade procedure, pacman verifies the integrity of all installed applications against its index. Since the changes made to Python via pip were not registered into pacman's version index, pacman cannot resolve the pip installed modules against the pacman index. These discrepancies cause the pacman update to exit with errors. No update will be accomplished. How can this be solved?
LaTeX
If having any problems with LaTeX updates, even the tug.org website recommends just replacing with latest edition. However, there is one workaround if things aren't more than a year or two old.
$ tlmgr update --self
tlmgr: Local TeX Live (2020) is older than remote repository (2021).
Cross release updates are only supported with
update-tlmgr-latest(.sh/.exe) -- --upgrade
See https://tug.org/texlive/upgrade.html for details.
For the problem above, download update-tlmgr-latest.sh, make it executable (chmod +x or file manager), and put it in /home/foo/latex or wherever your latex user-level top directory is at. Go into that directory and:
$ sh update-tlmgr-latest.sh -- --upgrade
$ tlmgr update --self
$ tlmgr update --all
.
Wine
Wine also can pull-in specific packages, when we try to make wine bottles. However, any time there's an OS update,
pacman will write a new wine directory in
~/.wine. After it's overwritten, it will use that vanilla version instead of the wine bottle we built for the app. How can this be solved?
In the case of an app like
youtube-dl, it's easy to uninstall the entire app (
# pacman -Rns) and reinstall. Or to play with uninstalling its pieces until one has a hit
The levels of involvement vary, but the strategy is the same for both Python and for LaTeX: install these independently of Arch from the start, in /home/user directory, and then update any path statements necessary for the Arch installed apps to find the program. Once these are in at the user level, there's no problem updating them with pip and tlmgr, because the updates are strictly within one's home directory, not the larger Arch installation. Accordingly, one can add features at will, which is often desirable when running the latest Spyder or what have you. For LaTeX, it's even less of a hassle, because there are no dependencies: one just needs to update $PATH statements following install. One can create an install in either case using, eg. $ mkdir /home/foo/latex and similarly with Python, though perhaps using a new folder for any version changes.
In the case of the OS update failure, one has no pleasant choices :uninstall then reinstall all of Python (immense time drain), or update the OS using an override (
# pacman -SyuI), which then leaves a potentially unstable Python install for its dependencies. This is true with LaTeX also, but nearly zero apps depend on LaTeX.
install notes
I've decided to use the 3rd party updaters to both install (Python -
pip, LaTeX -
tlmgr for TexLive), There are a couple of considerations.
- Install in user-level directory
- update path statements afterward, so the installation can find them. Path statements can be complex, because there are various Path statements to consider.
Sometimes one doesn't have a choice but to install packages via pip, but then what happens downstream with a pacman upgrade is ugly. It will look
like this, and there's a couple of workarounds.
- uninstall the pip upgrades, upgrade the system via pacman, then go back to pip and install whatever necessary pip packages.
- add the --overwrite=* flag to the standard "Syu". This causes pacman to simply overwrite any python files that seem inaccurate. This places pacman in the boss position over pip, but it also farks-up the pip version of the installation which, downstream, is a b*tch.
