Don't remove empty directories after build.

Deletion of these empty directories is flawed on docker with overlayfs,
so we better avoid it.
The build artifacts are still removed after gem install.

This fixes nokogiri issue sparklemotion#1370 .
For further details to the root issue see: moby/moby#19082

ext/nokogiri/extconf.rb

@@ -57,7 +57,6 @@ def do_clean
     # clean the ports build directory
     Pathname.glob(pwd.join('tmp', '*', 'ports')) { |dir|
       FileUtils.rm_rf(dir, verbose: true)
-      FileUtils.rmdir(dir.parent, parents: true, verbose: true)
     }
 
     if enable_config('static')