[v0.2x.0] in base image deleted files are present again in derived image

[khassel]

Actual behavior
Files removed in base image are present again in derived image.

Expected behavior
Files removed in base image should not be present again in derived image.

To Reproduce

1. build base image:
   Dockerfile:

   FROM centos:7
   RUN rm -vf /etc/yum.repos.d/*.repo
   

   build command: docker run -v ~/.docker/config.json:/kaniko/.docker/config.json -v $(pwd):/workspace gcr.io/kaniko-project/executor:v0.21.0 --destination karsten13/kaniko-bug-base:latest
2. build derived image:
   Dockerfile:

   FROM karsten13/kaniko-bug-base:latest
   RUN  ls -la /etc/yum.repos.d/;
   

   build command: docker run -v ~/.docker/config.json:/kaniko/.docker/config.json -v $(pwd):/workspace gcr.io/kaniko-project/executor:v0.21.0 --destination karsten13/kaniko-bug-base:ontop

Logs (build derived image):

INFO[0016] Retrieving image manifest karsten13/kaniko-bug-base:latest 
INFO[0033] Retrieving image manifest karsten13/kaniko-bug-base:latest 
INFO[0045] Built cross stage deps: map[]                
INFO[0045] Retrieving image manifest karsten13/kaniko-bug-base:latest 
INFO[0050] Retrieving image manifest karsten13/kaniko-bug-base:latest 
INFO[0057] Executing 0 build triggers                   
INFO[0057] Unpacking rootfs as cmd RUN  ls -la /etc/yum.repos.d/; requires it. 
INFO[0066] RUN  ls -la /etc/yum.repos.d/;               
INFO[0066] Taking snapshot of full filesystem...        
INFO[0072] Resolving 10745 paths                        
INFO[0073] cmd: /bin/sh                                 
INFO[0073] args: [-c ls -la /etc/yum.repos.d/;]         
INFO[0073] Running: [/bin/sh -c ls -la /etc/yum.repos.d/;] 
total 44
drwxr-xr-x 2 root root 4096 May  6 09:24 .
drwxr-xr-x 1 root root 4096 May  6 09:24 ..
-rw-r--r-- 1 root root 1664 Apr  7 22:01 CentOS-Base.repo
-rw-r--r-- 1 root root 1309 Apr  7 22:01 CentOS-CR.repo
-rw-r--r-- 1 root root  649 Apr  7 22:01 CentOS-Debuginfo.repo
-rw-r--r-- 1 root root  630 Apr  7 22:01 CentOS-Media.repo
-rw-r--r-- 1 root root 1331 Apr  7 22:01 CentOS-Sources.repo
-rw-r--r-- 1 root root 7577 Apr  7 22:01 CentOS-Vault.repo
-rw-r--r-- 1 root root  314 Apr  7 22:01 CentOS-fasttrack.repo
-rw-r--r-- 1 root root  616 Apr  7 22:01 CentOS-x86_64-kernel.repo
INFO[0073] Taking snapshot of full filesystem...        
INFO[0073] Resolving 10745 paths                        
INFO[0074] No files were changed, appending empty layer to config. No layer added to image. 

The problem appears since v0.20.0.

It works with v0.19.0. Here the logs of the derived build with v0.19.0:

INFO[0021] Resolved base name karsten13/kaniko-bug-base:latest to karsten13/kaniko-bug-base:latest 
INFO[0021] Resolved base name karsten13/kaniko-bug-base:latest to karsten13/kaniko-bug-base:latest 
INFO[0021] Retrieving image manifest karsten13/kaniko-bug-base:latest 
INFO[0038] Retrieving image manifest karsten13/kaniko-bug-base:latest 
INFO[0049] Built cross stage deps: map[]                
INFO[0049] Retrieving image manifest karsten13/kaniko-bug-base:latest 
INFO[0055] Retrieving image manifest karsten13/kaniko-bug-base:latest 
INFO[0061] Unpacking rootfs as cmd RUN  ls -la /etc/yum.repos.d/; requires it. 
INFO[0071] Taking snapshot of full filesystem...        
INFO[0075] Resolving paths                              
INFO[0076] RUN  ls -la /etc/yum.repos.d/;               
INFO[0076] cmd: /bin/sh                                 
INFO[0076] args: [-c ls -la /etc/yum.repos.d/;]         
total 8
drwxr-xr-x 2 root root 4096 May  6 09:16 .
drwxr-xr-x 1 root root 4096 May  6 09:16 ..
INFO[0076] Taking snapshot of full filesystem...        
INFO[0076] Resolving paths                              
INFO[0077] No files were changed, appending empty layer to config. No layer added to image. 

Additional Information

• Dockerfile --> see above
• Build Context --> see above
• Kaniko Image --> see above

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [no]
Please check if the build works in docker but not in kaniko	- [yes]
Please check if this error is seen when you use --cache flag	- [not tested]
Please check if your dockerfile is a multistage dockerfile	- [no]

[diegosucaria]

Yessss! I faced the exact same issue, thank you for writing this issue!!!
I could not find a proper manner to explain it since it is quite a weird behavior.

I spent several hours tracking what was going on, but when I realized it was the latest kaniko release, I fixed it by replacing the latest tag to v19.0.0 version.

I'll not move kaniko versions ever again!

[tejal29]

@diegosucaria Sorry for the regression, We are making our test coverage better by adding these Dockefiles to reduce regressions in upcoming releases. We will have a fix ASAP

[tejal29]

I used the dive tool to open the images built by kaniko and docker

1. Image built by docker build

2. Image built by kaniko

In both cases, the rm layer has the corresponding files removed ( they are red).

I ran docker build . <You docker file>
then pushed the docker image built at gcr.io/tejal-test/dockerwhiteout . Did a Run ls -al on this image and it also shows the removed files.

/ # /kaniko/executor -f Dockerfile --context=dir://workspace --no-push
INFO[0000] Retrieving image manifest gcr.io/tejal-test/dockerwhiteout 
INFO[0000] Retrieving image manifest gcr.io/tejal-test/dockerwhiteout 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest gcr.io/tejal-test/dockerwhiteout 
INFO[0001] Retrieving image manifest gcr.io/tejal-test/dockerwhiteout 
INFO[0002] Executing 0 build triggers                   
INFO[0002] Unpacking rootfs as cmd RUN  ls -la /etc/yum.repos.d/; requires it. 
INFO[0005] RUN  ls -la /etc/yum.repos.d/;               
INFO[0005] Taking initial snapshot                      
INFO[0005] Taking snapshot of full filesystem...        
INFO[0005] Resolving 10747 paths                        
INFO[0006] cmd: /bin/sh                                 
INFO[0006] args: [-c ls -la /etc/yum.repos.d/;]         
INFO[0006] Running: [/bin/sh -c ls -la /etc/yum.repos.d/;] 
total 44
drwxr-xr-x 2 root root 4096 May  6 17:58 .
drwxr-xr-x 1 root root 4096 May  6 17:58 ..
-rw-r--r-- 1 root root 1664 Apr  7 22:01 CentOS-Base.repo
-rw-r--r-- 1 root root 1309 Apr  7 22:01 CentOS-CR.repo
-rw-r--r-- 1 root root  649 Apr  7 22:01 CentOS-Debuginfo.repo
-rw-r--r-- 1 root root  630 Apr  7 22:01 CentOS-Media.repo
-rw-r--r-- 1 root root 1331 Apr  7 22:01 CentOS-Sources.repo
-rw-r--r-- 1 root root 7577 Apr  7 22:01 CentOS-Vault.repo
-rw-r--r-- 1 root root  314 Apr  7 22:01 CentOS-fasttrack.repo
-rw-r--r-- 1 root root  616 Apr  7 22:01 CentOS-x86_64-kernel.repo
INFO[0006] Taking snapshot of full filesystem...        
INFO[0006] Resolving 10747 paths                        
INFO[0007] file to whiteout%!(EXTRA []string=[])        
INFO[0007] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0007] Skipping push to container registry due to --no-push flag 

[tejal29]

I found the culprit which was an optimization commit i added 67db518

[khassel]

are you talking about

• Step1=build base image or
• Step2=build derived image ?

Step1 is always fine, you can build this with docker, kaniko-v19 or kaniko-v21.

But Step2 fails with kaniko-v21, in this build the deleted files are back.

[khassel]

Thank you for the fix and the quick release!