Script to compile list of packages installed after initial Distro install

Looking for feedback and contributions, to what would be a shared tool, to generate the list of packages, installed by the User, after the basic Distro installation.

Intent is to be able to either

  • duplicate the same package set on another host (or a host rebuilt from scratch),       or

  • present that set of packages as a checklist table for choosing to install only a subset of the packages currently installed (a future YAD-based project), again on another host or a host rebuilt from scratch.

Current state of the script is to

  • identify the manually installed files,

  • sort the packages by age as determined by the corresponding *.list file

  • identify packages not having an *.list file and determine if those have dependent packages, with a view of determining whether they are worthwhile retaining as part of the replication/rebuild set



Are there any recommendations regarding a better approach to tackling the above problem?



Session log (so far):

# ./PKGS__Report_PostInstall.sh

 List of Manually Installed Packages [659]:

	-rw-r--r-- 1 root root 8801 May  2 22:18 PKGS__Report_PostInstall.manual.txt

 The following contains the list of packages [593] sorted by oldest installation:

	-rw-r--r-- 1 root root 29202 May  2 22:18 PKGS__Report_PostInstall.stat.txt

 The following packages did not have a corresponding '*.list' file to determine installation date:

	-rw-r--r-- 1 root root 4600 May  2 22:18 PKGS__Report_PostInstall.nolist.txt

	 Evaluating  zlib1g-dev ...                     DEPENDANTS
	 Evaluating  tk-table ...                       DEPENDANTS
	 Evaluating  tk-html1 ...                       DEPENDANTS
	 Evaluating  tk-dev ...                         DEPENDANTS
	 Evaluating  tkblt-dev ...                      DEPENDANTS
	 Evaluating  tkblt ...                          DEPENDANTS
	 Evaluating  tk8.6-dev ...                      DEPENDANTS
	 Evaluating  tcl-memchan-dev ...                DEPENDANTS
	 Evaluating  tcl-memchan ...                    DEPENDANTS
	 Evaluating  tcllib-critcl ...                  DEPENDANTS
	 Evaluating  tcl-dev ...                        DEPENDANTS
	 Evaluating  tcl8.6-dev ...                     DEPENDANTS
	 Evaluating  qtbase5-dev ...                    DEPENDANTS
	 Evaluating  qml-module-qtwebkit ...            DEPENDANTS
	 Evaluating  openjdk-17-jdk ...                 DEPENDANTS
	 Evaluating  openjdk-11-jdk ...                 DEPENDANTS
	 Evaluating  mesa-common-dev ...                DEPENDANTS
	 Evaluating  libxrdapputils2 ...                DEPENDANTS
	 Evaluating  libxrandr-dev ...                  DEPENDANTS
	 Evaluating  libxinerama-dev ...                DEPENDANTS
	 Evaluating  libxine2-ffmpeg ...                DEPENDANTS
	 Evaluating  libxi-dev ...                      DEPENDANTS
	 Evaluating  libxcursor-dev ...                 DEPENDANTS
	 Evaluating  libwebp-dev ...                    DEPENDANTS
	 Evaluating  libwebkitgtk-6.0-4 ...             DEPENDANTS
	 Evaluating  libupower-glib-dev ...             DEPENDANTS
	 Evaluating  libssl-dev ...                     DEPENDANTS
	 Evaluating  libssl3 ...                        DEPENDANTS
	 Evaluating  libsqlite3-dev ...                 DEPENDANTS
	 Evaluating  libqt5webkit5-dev ...              DEPENDANTS
	 Evaluating  libqt5opengl5-dev ...              DEPENDANTS
	 Evaluating  libpoppler-dev ...                 DEPENDANTS
	 Evaluating  libpng-dev ...                     DEPENDANTS
	 Evaluating  libosmesa6 ...                     DEPENDANTS
	 Evaluating  libopencsg1-dbg ...                DEPENDANTS
	 Evaluating  libmpg123-dev ...                  DEPENDANTS
	 Evaluating  libjpeg62 ...                      DEPENDANTS
	 Evaluating  libhwloc-contrib-plugins ...       DEPENDANTS
	 Evaluating  libgtkmm-3.0-dev ...               DEPENDANTS
	 Evaluating  libgtk-4-dev ...                   DEPENDANTS
	 Evaluating  libgtk2.0-dev ...                  DEPENDANTS
	 Evaluating  libglu1-mesa-dev ...               DEPENDANTS
	 Evaluating  libglibmm-2.4-dev ...              DEPENDANTS
	 Evaluating  libglfw3-dev ...                   DEPENDANTS
	 Evaluating  libglew-dev ...                    DEPENDANTS
	 Evaluating  libglade2-0 ...                    DEPENDANTS
	 Evaluating  libgl1-mesa-dev ...                DEPENDANTS
	 Evaluating  libfontconfig1-dev ...             DEPENDANTS
	 Evaluating  libfftw3-dev ...                   DEPENDANTS
	 Evaluating  libexpat1-dev ...                  DEPENDANTS
	 Evaluating  libegl1-mesa ...                   DEPENDANTS
	 Evaluating  libdvdcss-dev ...                  DEPENDANTS
	 Evaluating  libdvdcss2 ...                     DEPENDANTS
	 Evaluating  libcairo-script-interpreter2 ...   DEPENDANTS
	 Evaluating  libcairo2-dev ...                  DEPENDANTS
	 Evaluating  libboost-dev ...                   DEPENDANTS
	 Evaluating  libbluray-dev ...                  DEPENDANTS
	 Evaluating  libbdplus-dev ...                  DEPENDANTS
	 Evaluating  libavcodec-extra ...               DEPENDANTS
	 Evaluating  libavcodec-dev ...                 DEPENDANTS
	 Evaluating  libao-dev ...                      DEPENDANTS
	 Evaluating  gtk3-engines-unico ...             DEPENDANTS
	 Evaluating  gstreamer1.0-plugins-bad ...       DEPENDANTS
	 Evaluating  gir1.2-evince-3.0 ...              DEPENDANTS
	 Evaluating  freeglut3-dev ...                  DEPENDANTS
	 Evaluating  caja-extension-fma ...             DEPENDANTS

#

Coud some of these that have ‘DEPENDANTS’ be superfluous, given that the is no corresponding *.list file?



Script: PKGS__Report_PostInstall.sh

#!/bin/bash

dbg=0

###
###	Script Intended To Identify All Post-Distro User-Installed Packages
###
###	NOTICE - Work-in-progress - Looking for feedback, additions, enhancements
###


progress=0

now=$( date '+%Y%m%d_%H%M%S' )

work=$( basename "${0}" ".sh" ).work.tmp	; rm -f "${work}"

marked=$( basename "${0}" ".sh" ).manual.txt	; rm -f "${marked}"
fTemps=$( basename "${0}" ".sh" ).temp.txt	; rm -f "${fTemps}"
fDates=$( basename "${0}" ".sh" ).stat.txt	; rm -f "${fDates}"
fPackg=$( basename "${0}" ".sh" ).packgs.txt	; rm -f "${fPackg}"
fMssng=$( basename "${0}" ".sh" ).nolist.txt	; rm -f "${fMssng}"
report=$( basename "${0}" ".sh" ).${now}.txt	; rm -f "${report}"


###
###	Get Manually Installed Packages
###

comm -23 <(apt-mark showmanual | sort) <(gzip -dc /var/log/installer/initial-status.gz | sed -n 's/^Package: //p' | sort) >"${marked}"

numMrk=$( wc -l "${marked}" | awk '{ print $1 }' )

echo -e "\n List of Manually Installed Packages [${numMrk}]:\n"
ls -l "${marked}" | awk '{ printf("\t%s\n", $0 ) ; }'



###
###	Determine Installation Date of Packages
###

#                               filename perms user group size mods status filetype
#result=`/usr/bin/stat --format "%n|%A|%U|%G|%s|%Y|%Z|%F" "${fNam}" `
tac "${marked}" |
while read line
do
	testor=$( grep "Package: ${line}"'$' /var/lib/dpkg/status )
	if [ -n "${testor}" ]
	then
		list="/var/lib/dpkg/info/${line}.list"
		if [ -s "${list}" ]
		then
			/usr/bin/stat --format "%Y|%n|" "${list}"
			echo "${testor}" >>"${fPackg}"
		else
			echo "${testor}|MISSING|${list}" >>"${fPackg}"
		fi
	fi
done | {
	if [ ${progress} -eq 1 ]
	then
		awk 'BEGIN{
			cnt=49 ;
			printf("%49s", "" ) | "cat 1>&2" ;
		}{
			print $0 ;
			cnt++ ;
			if( cnt == 50 ){
				printf(" | %s\n", $0 ) | "cat 1>&2" ;
				cnt=0 ;
			}else{
				printf(".") | "cat 1>&2" ;
			} ;
		}END{
			if( cnt != 0 ){
				fmt=sprintf("%s%s%s%s", "%", 49-cnt, "s", " | %s\n" ) ;
				printf( fmt, "", $0 ) | "cat 1>&2" ;
			} ;
		}'
	else
		cat
	fi
}>"${fTemps}"



###
###	Sort Packages by Oldest First
###

sort -t "|" --key=1.1,2.0 "${fTemps}" > "${fDates}"

numAge=$( wc -l "${fDates}" | awk '{ print $1 }' )

echo -e "\n The following contains the list of packages [${numAge}] sorted by oldest installation:\n"
ls -l "${fDates}" | awk '{ printf("\t%s\n", $0 ) ; }'
#echo ""
#head "${fDates}"

if [ ${numAge} -ne ${numMrk} ]
then
	echo -e "\n The following packages did not have a corresponding '*.list' file to determine installation date:\n"
	grep '|MISSING|' "${fPackg}" >"${fMssng}"		 #| awk '{ printf("\t | %s\n", $0 ) ; }'
	ls -l "${fMssng}" | awk '{ printf("\t%s\n", $0 ) ; }'
fi



###
###	Identify Dependants for Packages Having No List
###
###	No Dependants Suggests Packages Are Optional and Retention Should Be Confirmed
###	Before Including in List For Host Installation Replication
###

boldON="\033[1m"
boldOFF="\033[0m"

greenON="\033[92;1m"
greenOFF="\033[0m"

yellowON="\033[93;1m"
yellowOFF="\033[0m"

redON="\033[91;1m"
redOFF="\033[0m"

orangeON="\033[33;1m"
orangeOFF="\033[0m"

pNameWidth=$( cut -f1 -d\| "${fMssng}" | cut -f2 -d\: | sed 's+^\ ++' | while read line ; do echo "${line}" | wc -c ; done | sort -nr | head -1 | awk '{ print $1 }' )
test ${dbg} -eq 1 && echo $pNameWidth

pWidth=$( expr ${pNameWidth} + 18 )
test ${dbg} -eq 1 && echo $pWidth

fmtStr="\t %-${pWidth}s"
test ${dbg} -eq 1 && echo "'${fmtStr}'"

echo " Evaluating 'no list file' packages ..."
for package in $( cut -f1 -d\| "${fMssng}" | cut -f2 -d\: | sed 's+^\ ++' )
do
	printf "${fmtStr}" "Evaluating  ${package} ... "

	rm -f "${work}"
	apt-rdepends -r --state-follow=Installed --state-show=Installed "${package}" >"${work}" 2>>/dev/null
	if [ -s "${work}" ]
	then
		echo -e "${boldON}${greenON}DEPENDANTS${greenOFF}"
	else
		echo -e "${boldON}${redON}NO_DEPENDANTS${redOFF}"
	fi
done




echo "\n\n

 FUTURES:
	* interractive logic to selectively purge packages identified as 'no list' if also 'NO_DEPENDANTS'
	* consolidate list of package names for host 'installation replication'
\n"

exit 0
4 Likes

I like that :slight_smile:

I can only add what I usually do to create a post-install list
maybe you find one or two things in it that you consider usable enough to use in one way or another:


Directly after installing my APT based distro, I do this:

apt-mark showmanual > initially_installed

This gives me a list of initially installed pkgs without their dependencies (dependencies will be installed anyway).


Then, when I have my install exectly as I want it to be, i do this:

apt-mark showmanual | grep -vf initially_installed >my_list

This gives me the list of everything I installed afterwards.
I only have to do this procedure just once.


Now every time I install an APT based distro, I only have to run

sudo apt-get -y -m install $(cat my_list)

to get my personal package choices installed

5 Likes