Anthony Takes Note

Often a Linux distribution or software package will add extensive and informative comments to a configuration file. One example of this is the Postfix main.cf file

While the comments can be invaluable, sometimes you want just the basic file without any extras.

To turn a heavily-annotated file into a bare file, you will need to remove 2 sets of lines:

• Any line that begins with a comment symbol (That symbol varies between different applications or programming languages, but it's usually the Hash symbol)
• Any line that is only whitespace.

This is straightforward with the sed command:

cat /etc/postfix/main.cf | sed -e '/^#/d' -e '/^$/d'

• -e simply means “this is a sed command” and is a good way of including multiple commands/searches/etc. That means you don't have to try to merge multiple searches into a single command.
• /search/d means “if the search defined in search matches, delete (d) that line from the output.
• The ^# search means “match lines that have a hashtag at the beginning of the line.”
• The ^$ search means “match lines that have nothing between the start of the line and the end of the line.

Questions? hit me up on Mastodon at @anthonyclarka2@bsd.network.

Docker is an amazing tool and has revolutionized the software and IT industries extremely rapidly. The standard workflow for most users involves building on top of an image created by someone else, often the maintainers of the project or application you are installing.

However, there may be times when you want to build your own #Docker image, such as a strict compliance environment, or when you want to optimize the image layout or disk usage.

Below is one way of creating a docker image. There are many more, but this was one that I found straightforward and easy to understand. It doesn't create the smallest image possible, but it is a good place to start from.

Start with a Fedora computer, be it a VM or a physical development workstation

Install some virtualization packages: dnf install libvirt virt-install virt-manager virt-viewer libguestfs-tools-c lorax

Download a CentOS 7 Kickstart file to /tmp. I've shared one as a github gist or you can grab the original from the CentOS cloud SIG.

fedora :: /tmp » curl https://gist.githubusercontent.com/anthonyclarka2/052f3a24ce722dfadf33c2f2f4807ee7/raw/01c0d7ab7ff1cfaffe3d73c489b2b7f59db27758/centos7.ks -o /tmp/centos7.ks
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3210  100  3210    0     0  21543      0 --:--:-- --:--:-- --:--:-- 21543

Download the CentOS 7 boot ISO image:

fedora :: /tmp » curl http://mirror.centos.org/centos/7/os/x86_64/images/boot.iso -o /tmp/boot.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  507M  100  507M    0     0  15.8M      0  0:00:32  0:00:32 --:--:-- 15.3M

Now use those 2 files to create the base for the image:

fedora :: /tmp » sudo livemedia-creator --make-tar --iso=/tmp/boot.iso --ks=/tmp/centos7.ks --image-name=centos-7-docker.tar.xz
2019-02-03 12:04:14,644: livemedia-creator v29.18-1
2019-02-03 12:04:14,808: disk_img = /var/tmp/centos-7-docker.tar.xz
2019-02-03 12:04:14,809: Using disk size of 3002MiB
2019-02-03 12:04:14,809: install_log = /tmp/virt-install.log
2019-02-03 12:04:14,966: qemu vnc=127.0.0.1:0
2019-02-03 12:04:14,966: Running qemu
2019-02-03 12:04:15,091: Processing logs from ('127.0.0.1', 52022)
2019-02-03 12:34:08,774: Installation finished without errors.
2019-02-03 12:34:08,775: Shutting down log processing
2019-02-03 12:34:08,775: unmounting the iso
2019-02-03 12:34:08,847: Partition mounted on /var/tmp/tmpmgcijjqi size=3145728000
2019-02-03 12:35:44,889: Disk Image install successful
2019-02-03 12:35:44,890: SUMMARY
2019-02-03 12:35:44,890: -------
2019-02-03 12:35:44,890: Logs are in /tmp
2019-02-03 12:35:44,890: Disk image is at /var/tmp/centos-7-docker.tar.xz
2019-02-03 12:35:44,890: Results are in /var/tmp

Note that you need to run the command “livemedia-creator” as root. You're using it to create a tar file of an entire Linux system's disk. It's compressed by the “xz” utility by default, which is the same compression scheme used by Docker and other container libraries.

That will take 30+ minutes. Open a new terminal on that workstation or VM and follow the log. Please be patient, as it often takes a few minutes for the log to start generating output because the VM takes time to boot. I have placed the logs from one run up on my github gists: virt-install.log, livemedia.log, and program.log.

fedora :: /tmp » tail -f virt-install.log

The resulting file, placed in /var/tmp/centos-7-docker.tar.xz can now be used in a very basic Dockerfile to create an image:

FROM scratch

ADD centos-7-docker.tar.xz /

LABEL name="CentOS 7 basic image" build-date="2019-02-01" maintainer="somebody@example.com"

The resulting output from ```docker build -t centos7basic:0.1 .” is about 200MB:

fedora :: projects/docker/centos7 » docker build -t centos7basic:0.1 .
Sending build context to Docker daemon  43.52MB
Step 1/3 : FROM scratch
 --->
Step 2/3 : ADD centos-7-docker.tar.xz /
 ---> 38b7b9cd8813
Step 3/3 : LABEL name="CentOS 7 basic image" build-date="2019-02-01" maintainer="somebody@example.com"
 ---> Running in 31be1218ab5c
Removing intermediate container 31be1218ab5c
 ---> 043e2edf19e0
Successfully built 043e2edf19e0
Successfully tagged centos7basic:0.1

fedora :: projects/docker/centos7 » docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
centos7basic       0.1                 043e2edf19e0        9 seconds ago       204MB

To look at the contents of the tar.xz file use the syntax “tar tJvf centos-7-docker.tar.xz”

More to come once I write it, probably in a new post!

In a recent incident, I was asked to provide a list of destination addresses being delivered to a particular mail server. Here's an example line from a #Postfix log:

Dec 29 05:00:51 mail01.test.example.com postfix/smtp[28704]: AF02145249: to=<emailtest.address@gmail.com>, relay=gmail-smtp-in.l.google.com[209.85.232.27]:25, delay=0.69, delays=0.01/0/0.16/0.52, dsn=2.0.0, status=sent (250 2.0.0 OK 1544545651 n189bb201234abc.123 - gsmtp)

From that log line, you can see that the #email address is surrounded by the angled brackets, “<” and “>”. Those brackets are preceded by the text “ to=”.

From this fantastic stackoverflow page I found some very useful grep commands for use in compiling part of this answer.

I decided to go with the fastest #grep answer, since I was dealing with multiple gigabytes of mail logs from the Postfix Mail Transfer Agent (MTA):

grep -Po ' to=<\K[^>]*'

Now I need to explain what each bit of that means. (Using the GNU Grep 3.3 manual)

• -P Interpret the #regex as a Perl-compatible regular expression (a “PCRE”). This can also be written as --perl-regexp. PCREs are very powerful regexes that are used in a very wide variety of open source applications. Their syntax is covered in their documentation.
• -o aka --only-matching. Only print the matching part of the content when you find a match, as opposed to the default behaviour of printing the entire line. This is useful because it means you don't have to then pipe your output to “cut”, “awk” or other tools to get what you want, but it does mean you need to be more precise with your matches.
• ' to=<\K[^>]*'
  • to=< Matches a space, followed by the characters “t”, “o”, “=”, “<“.
  • \K Report that the match starts here. In other words, you match with the characters “ to=<” but then discard those characters when printing the match. Again, a neat way to only show what you want, rather than having to pipe your output to another application to chop up the output.
  • [^>]* This matches anything that isn't a “>” symbol, or in other words the match is ended when a “>” symbol is encountered. The square brackets are a character class and the caret (“^”) symbol negates that character class. The asterisk at the end means “match zero or more of the preceding thing”. (It's OK to end the match before the “>” symbol because in an email address, the domain name portion (that comes after the “@” symbol) may not contain that character.)

When ran against the log line shown at the beginning, you get the following output:

$ cat tmp2.txt | grep -Po ' to=<\K[^>]*'
emailtest.address@gmail.com

If you are interested in using regexes more, the RegExr site is a visual regex learning tool. You can use it to build up regexes slowly while understanding exactly what they do. You can also paste in a pre-existing regex to see if the site can describe what it does for you.