Welcome to Day 8 of 21 days of Docker, let’s try to dig deeper into images and layers.
A Docker image is built up from a series of layers. Each layer represents an instruction in the image’s Dockerfile. Each layer except the very last one is read-only.
Consider the following Dockerfile
FROM ubuntu:15.04
RUN apt-get -y install nginx
COPY index.html /var/www/html
CMD ["nginx","-g","daemon off;"]- This Dockerfile contains four commands each of which creates a layer
FROM: statements starts out by creating a layer from the ubuntu 15.04 image
RUN: Command install nginx
COPY: add some files to nginx www directory
CMD: command specify which command to run within the container 
Some points to keep in mind
- Each layer is only a layer of differences from the layer before it.
- The layers are stacked on top of each other
- When you create a new container, you add a new writable layer on top of the underlying layers. This layer is often called the “container layer”
- All changes made to the running container, such as writing new files, modifying existing files, and deleting files, are written to this thin writable container layer
- If you want to check the layer of the image
$ docker image history nginx
IMAGE CREATED CREATED BY SIZE COMMENT
f949e7d76d63 2 weeks ago /bin/sh -c #(nop) CMD ["nginx" "-g" "daemon… 0B
<missing> 2 weeks ago /bin/sh -c #(nop) STOPSIGNAL SIGTERM 0B
<missing> 2 weeks ago /bin/sh -c #(nop) EXPOSE 80 0B
<missing> 2 weeks ago /bin/sh -c ln -sf /dev/stdout /var/log/nginx… 0B
<missing> 2 weeks ago /bin/sh -c set -x && addgroup --system -… 56.8MB
<missing> 2 weeks ago /bin/sh -c #(nop) ENV PKG_RELEASE=1~buster 0B
<missing> 2 weeks ago /bin/sh -c #(nop) ENV NJS_VERSION=0.3.5 0B
<missing> 2 weeks ago /bin/sh -c #(nop) ENV NGINX_VERSION=1.17.4 0B
<missing> 4 weeks ago /bin/sh -c #(nop) LABEL maintainer=NGINX Do… 0B
<missing> 4 weeks ago /bin/sh -c #(nop) CMD ["bash"] 0B
<missing> 4 weeks ago /bin/sh -c #(nop) ADD file:1901172d265456090… 69.2MB
A storage driver handles the details about the way these layers interact with each other.
Different storage drivers are available(eg: overlay2, devicemappers, aufs), which have advantages and disadvantages in different situations.
Difference between Image and Container
- To sum up, what we have discussed so far, the major difference between a container and an image is the top writable layer.
- All writes to the container that adds new or modifies existing data are stored in this writable layer.
- When the container is deleted, the writable layer is also deleted.
- The underlying image remains unchanged.
- Because each container has its own writable container layer, and all changes are stored in this container layer, multiple containers can share access to the same underlying image and yet have their own data state(as shown in the above image)
Container size on Disk
- To view the container size on disk
$ docker container ps -s
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES SIZE
424c8c51e49a centos:centos7 "bash" 4 hours ago Exited (137) 3 hours ago objective_feynman 0B (virtual 202MB)
e2452dcdc6e1 centos:centos7 "bash" 4 hours ago Exited (137) 3 hours ago jovial_murdock 0B (virtual 202MB)
b4031e7df866 centos:centos7 "bash" 4 hours ago Exited (0) 4 hours ago charming_herschel 0B (virtual 202MB)
ce32f774053a centos "bash" 5 hours ago Exited (0) 4 hours ago quirky_cori 37B (virtual 220MB)
24756038f07a busybox "sh" 5 hours ago Exited (137) 3 hours ago mynonenetwork1 10B (virtual 1.22MB)
418657ff1fe4 ubuntu "bash" 5 hours ago Exited (0) 3 hours ago mynonenetwork 60B (virtual 64.2MB)
* Containers, Images, Local Volume and Cache Size
$ docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 16 8 1.78GB 1.421GB (79%)
Containers 22 0 651MB 651MB (100%)
Local Volumes 0 0 0B 0B
Build Cache 0 0 0B 0B
OR if you are looking for more verbose output
$ docker system df -v
Images space usage:
REPOSITORY TAG IMAGE ID CREATED SIZE SHARED SIZE UNIQUE SIZE CONTAINERS
multistage latest b536984e0b37 19 hours ago 7.586MB 5.577MB 2.009MB 1
single-stage latest 3b5798ed94c5 19 hours ago 813.8MB 802.6MB 11.24MB 1
mycentosnewapache latest 6d082f2b1c5e 22 hours ago 449.6MB 449.6MB 0B 0
mycentosapache latest 03f61d93b53e 22 hours ago 449.6MB 449.6MB 0B 2
mycustomnginx1 latest fe85db4f4704 44 hours ago 162.4MB 162.4MB 22B 0
mycustomnginx latest 35d8f0969f7c 44 hours ago 162.4MB 162.4MB 22B 0
cmdimage latest 9888f3568e1f 2 days ago 1.22MB 1.22MB 0B 0
healthcheck_image latest ca8599a83d24 2 days ago 1.22MB 1.22MB 0B 0
ping_docker latest 6cb5dc1118a0 5 days ago 1.22MB 1.22MB 0B 0
centos latest 0f3e07c0138f 9 days ago 219.5MB 0B 219.5MB 5
golang 1.13.1 52b59e9ead8e 2 weeks ago 802.6MB 802.6MB 0B 0
nginx latest f949e7d76d63 2 weeks ago 126MB 0B 126MB 2
ubuntu bionic 2ca708c1c9cc 3 weeks ago 64.18MB 64.18MB 0B 3
busybox latest 19485c79a9bb 5 weeks ago 1.22MB 1.22MB 0B 3
centos centos7 67fa590cfc1c 7 weeks ago 201.9MB 201.9MB 0B 4
alpine 3.10.2 961769676411 7 weeks ago 5.577MB 5.577MB 0B 0
Containers space usage:
CONTAINER ID IMAGE COMMAND LOCAL VOLUMES SIZE CREATED STATUS NAMES
424c8c51e49a centos:centos7 "bash" 0 0B 5 hours ago Exited (137) 3 hours ago objective_feynman
e2452dcdc6e1 centos:centos7 "bash" 0 0B 5 hours ago Exited (137) 3 hours ago jovial_murdock
b4031e7df866 centos:centos7 "bash" 0 0B 5 hours ago Exited (0) 5 hours ago charming_herschel
ce32f774053a centos "bash" 0 37B 5 hours ago Exited (0) 5 hours ago quirky_cori
24756038f07a busybox "sh" 0 10B 5 hours ago Exited (137) 3 hours ago mynonenetwork1
418657ff1fe4 ubuntu "bash" 0 60B 5 hours ago Exited (0) 3 hours ago mynonenetwork
efb9e8f5615a ubuntu "bash" 0 88.2MB 5 hours ago Exited (0) 3 hours ago mynewhost1
88d9f7bfa898 ubuntu "bash" 0 88.1MB 6 hours ago Exited (0) 3 hours ago mynewhost
94059b00c326 centos "bash" 0 44.7MB 6 hours ago Exited (0) 3 hours ago myhost
e0fa509a4a38 centos "/bin/bash" 0 53B 6 hours ago Exited (0) 3 hours ago mycustombridge02
757f1a00d053 centos "/bin/bash" 0 0B 6 hours ago Exited (0) 3 hours ago mycustombridge01
394b8e5255ed busybox "sh" 0 0B 7 hours ago Exited (137) 3 hours ago bridge02
46ea63e383c8 busybox "sh" 0 77B 7 hours ago Exited (137) 3 hours ago bridge01
dac9b0fca295 multistage "./helloworld" 0 0B 19 hours ago Exited (0) 19 hours ago laughing_edison
3fd2abfc0e0f single-stage "./helloworld" 0 0B 19 hours ago Exited (0) 19 hours ago beautiful_torvalds
993f69137ed7 mycentosapache "/usr/sbin/apachectl…" 0 1.18kB 22 hours ago Exited (137) 13 hours ago mycentoscustomapache1
c04a61a61e4f mycentosapache "/usr/sbin/apachectl…" 0 5.1kB 22 hours ago Exited (137) 13 hours ago mycentoscustomapache
c9c79d7b72cd 8f2ae044a233 "/bin/sh -c 'yum -y …" 0 215MB 22 hours ago Exited (1) 22 hours ago mystifying_darwin
35355550ab59 centos:centos7 "/bin/bash" 0 215MB 22 hours ago Exited (137) 13 hours ago mycentostest
733c8448e2c8 centos "bash" 0 43B 25 hours ago Exited (0) 13 hours ago mycentos
789f45a23f9f nginx "nginx -g 'daemon of…" 0 15B 29 hours ago Exited (0) 13 hours ago mynginxnew
b4db78c85559 nginx "nginx -g 'daemon of…" 0 0B 29 hours ago Exited (0) 29 hours ago mynginx
Local Volumes space usage:
VOLUME NAME LINKS SIZE
Build cache usage: 0B
CACHE ID CACHE TYPE SIZE CREATED LAST USED USAGE SHAREDThe Copy-on-Write Mechanism
When we launch an image, the Docker engine does not make a full copy of the already stored image. Instead, it uses something called the copy-on-write mechanism. This is a standard UNIX pattern that provides a single shared copy of some data until the data is modified.
To do this, changes between the image and the running container are tracked. Just before any write operation is performed in the running container, a copy of the file that would be modified is placed on the writeable layer of the container, and that is where the write operation takes place. Hence the name, “copy-on-write”.
If this wasn’t happening, each time you launched an image, a full copy of the filesystem would have to be made. This would add time to the startup process and would end up using a lot of disk space.
Let’s take a simple example, to understand how Image Layer works,below is my dockerfile
FROM busybox
RUN dd if=/dev/zero of=/tmp/test1 bs=1M count=50
RUN dd if=/dev/zero of=/tmp/test2 bs=1M count=50- Try to build an image out of this Dockerfile
$ docker build -t mylayer1 .
Sending build context to Docker daemon 2.048kB
Step 1/3 : FROM busybox
---> 19485c79a9bb
Step 2/3 : RUN dd if=/dev/zero of=/tmp/test1 bs=1M count=50
---> Running in 2495248f03df
50+0 records in
50+0 records out
52428800 bytes (50.0MB) copied, 0.030272 seconds, 1.6GB/s
Removing intermediate container 2495248f03df
---> 4c557085e440
Step 3/3 : RUN dd if=/dev/zero of=/tmp/test2 bs=1M count=50
---> Running in aca8401f91cc
50+0 records in
50+0 records out
52428800 bytes (50.0MB) copied, 0.029855 seconds, 1.6GB/s
Removing intermediate container aca8401f91cc
---> bfbfff240e48
Successfully built bfbfff240e48
Successfully tagged mylayer1:latest
- Check the image size
$ docker build -t mylayer1 .
REPOSITORY TAG IMAGE ID CREATED SIZE
mylayer1 latest bfbfff240e48 4 seconds ago 106MB- Let’s make little modification to the Dockefile by trying to remove these two files
FROM busybox
RUN dd if=/dev/zero of=/tmp/test1 bs=1M count=50
RUN dd if=/dev/zero of=/tmp/test2 bs=1M count=50
RUN rm -rf /tmp/test1
RUN rm -rf /tmp/test2- Try to build the image again
$ docker build -t mylayer2 .
Sending build context to Docker daemon 2.048kB
Step 1/5 : FROM busybox
---> 19485c79a9bb
Step 2/5 : RUN dd if=/dev/zero of=/tmp/test1 bs=1M count=50
---> Using cache
---> 4c557085e440
Step 3/5 : RUN dd if=/dev/zero of=/tmp/test2 bs=1M count=50
---> Using cache
---> bfbfff240e48
Step 4/5 : RUN rm -rf /tmp/test1
---> Running in 9059f97edae0
Removing intermediate container 9059f97edae0
---> 3987aea24cc7
Step 5/5 : RUN rm -rf /tmp/test2
---> Running in 9dde88d940ee
Removing intermediate container 9dde88d940ee
---> fd617a5810fa
Successfully built fd617a5810fa
Successfully tagged mylayer2:latest
- Check the image size
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
mylayer2 latest fd617a5810fa 4 seconds ago 106MB- It’s the same, this doesn’t look, right? As we introduce two more new layer
RUN rm -rf /tmp/test1
RUN rm -rf /tmp/test2- As I mentioned above, image is just the collection of layer, and we are deleting test1 and test2 at different layer and that is the reason the size of the image is the same
- Let’s move those remove instruction in the same layer
FROM busybox
RUN dd if=/dev/zero of=/tmp/test1 bs=1M count=50 && rm -rf /tmp/test1
RUN dd if=/dev/zero of=/tmp/test2 bs=1M count=50 && rm -rf /tmp/test2- Re-build the image
$ docker build -t mylayer3 .
Sending build context to Docker daemon 2.048kB
Step 1/3 : FROM busybox
---> 19485c79a9bb
Step 2/3 : RUN dd if=/dev/zero of=/tmp/test1 bs=1M count=50 && rm -rf /tmp/test1
---> Running in 58ecae5b4282
50+0 records in
50+0 records out
52428800 bytes (50.0MB) copied, 0.034398 seconds, 1.4GB/s
Removing intermediate container 58ecae5b4282
---> 062fa630bd3d
Step 3/3 : RUN dd if=/dev/zero of=/tmp/test2 bs=1M count=50 && rm -rf /tmp/test2
---> Running in 493d7e59ea5b
50+0 records in
50+0 records out
52428800 bytes (50.0MB) copied, 0.029036 seconds, 1.7GB/s
Removing intermediate container 493d7e59ea5b
---> 13f26e00997f
Successfully built 13f26e00997f
Successfully tagged mylayer3:latest
- Check the size
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
mylayer3 latest 13f26e00997f 7 seconds ago 1.22MBLooks promising 🙂
What is Dangling Images?
Dangling images are layers that have no relationship to any tagged images. They no longer serve a purpose and consume disk space
Let take a scenario in which these none layers will be created?
- Let pull apache image from the docker registry
$ docker image pull httpd
Using default tag: latest
latest: Pulling from library/httpd
b8f262c62ec6: Already exists
2c31b9311798: Pull complete
7422a3cdf4e3: Pull complete
1919d4fbf9e1: Pull complete
60812fa1ab4c: Pull complete
Digest: sha256:39d7d9a3ab93c0ad68ee7ea237722ed1b0016ff6974d80581022a53ec1e58797
Status: Downloaded newer image for httpd:latest
docker.io/library/httpd:latest- Start the container
$ docker container run -d httpd
1b97fb9667c0cc373c426e1a8ac3df6703c9dc91d63b46e1a52f8cf3a499b935- Check if it’s running
$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1b97fb9667c0 httpd "httpd-foreground" 6 seconds ago Up 4 seconds 80/tcp tender_kirch- Now let’s try to remove the image, associated with this container
$ docker image rm httpd
Error response from daemon: conflict: unable to remove repository reference "httpd" (must force) - container 1b97fb9667c0 is using its referenced image 19459a872194As the container is using this image, we can’t delete it, so the only way we can delete this image by using -f flag
$ docker image rm -f httpd
Untagged: httpd:latest
Untagged: httpd@sha256:39d7d9a3ab93c0ad68ee7ea237722ed1b0016ff6974d80581022a53ec1e58797- Pay special attention to the output of the above command, it says untagged the image
- Now if you try to list the image, used by the container, you will see the output like this and it creates the dangling image.
$ docker image ls |grep 19459a872194
<none> <none> 19459a872194 3 weeks ago 154MB- Now the question is how to clean these kinds of images
- First, stop and remove the container
$ docker container rm -f 1b97fb9667c0
1b97fb9667c0- Run docker image prune to clean dangling image
$ docker image prune
WARNING! This will remove all dangling images.
Are you sure you want to continue? [y/N] y
Deleted Images:
deleted: sha256:19459a87219415cc5751fb48e9aa37ea580b98eed1fe784e76c4bc3c9b3b0741
deleted: sha256:ebefc32f49862ee9f4f672b338f29aefe056971bcc88912ed87968f5f75b0ae7
deleted: sha256:fe3c18ca62ea3f45b69ad643869c6fd3ab567a5ad8207adf8d071d54c9152597
deleted: sha256:62d97f753807d11f7ab72f8ae1911ff22617c9134e3edec13bf909493edb025c
deleted: sha256:c4e27a02760c0a8711ac5e9f7e0fc7cffeb38b44b4041d0c6ded5087586727a8
Total reclaimed space: 84.62MB- Verify it
$ docker image ls |grep 19459a872194NOTE: If you build an image without specifying -t (i.e docker build . ), the image will appear in the list of dangling images as it has no association with the tagged images.
How to create an image out of a container
- Let’s take a scenario, where I created a new container out of busybox image
$ docker container run -dt --name mytestcont busybox
8945e3832ac3abee79250b8bb5288f62296f8430cce954f9a6db331e31e232c8- Logged into that container
$ docker container exec -it 8945e3832ac3 sh
/ #
# Create a new file
/ # touch abc.txt
/ #- To create a docker image out of this container, I need to use docker container commit command
- The command will look like this
docker container commit [OPTIONS] CONTAINER [REPOSITORY[:TAG]]
$ docker container commit -a "plakhera" -m "my docker busybox image from container" 8945e3832ac3 mytestbusyboximage
sha256:aca40c0caa3b8159d2350db6240f368f6e75b06bdc1c98af268858a8089a202c
-a, --author string Author (e.g., "John Hannibal Smith <[email protected]>")
-m, --message string Commit message - To verify it, run the new container using this image
$ docker container run -it mytestbusyboximage sh
/ # ls -l
total 20
-rw-r--r-- 1 root root 0 Oct 11 05:46 abc.txtNOTE: By default, the container being committed and its process will be paused while the image is committed.
This is a good place, to stop for Day 8
Please follow me with my Journey
- Website:https://100daysofdevops.com/
- Twitter:@100daysofdevops OR @lakhera2015
- Facebook:https://www.facebook.com/groups/795382630808645/
- Medium:https://medium.com/@devopslearning
- GitHub:https://github.com/100daysofdevops/21_Days_of_Docker
This time to make learning more interactive, I am adding
- Slack
- Meetup
Please feel free to join this group.
Slack:
Meetup Group
If you are in the bay area, please join this meetup group https://www.meetup.com/100daysofdevops/
Great content!
Thanks