nodes fail to delete workspace if previous job was aborted

[sloretz]

http://ci.ros2.org/job/ci_linux/3083/console failed after I aborted a job on the same node http://ci.ros2.org/job/ci_linux/3082/

17:29:02 + rm -rf ws workspace work space
17:29:03 rm: cannot remove ‘ws/src/ros2’: Directory not empty
17:29:03 Build step 'Execute shell' marked build as failure

SSH'ing to the directory showed it had many files. I did not check the permissions before deleting the workspace using sudo.

ubuntu@ip-10-234-134-39:/home/jenkins/workspace/ci_linux$ ls -R ws
ws:
src

ws/src:
ros2

ws/src/ros2:
common_interfaces  example_interfaces  poco_vendor  rcl_interfaces  rmw_fastrtps  system_tests

ws/src/ros2/common_interfaces:
actionlib_msgs      CONTRIBUTING.md  geometry_msgs  nav_msgs   sensor_msgs  std_msgs  stereo_msgs      visualization_msgs
builtin_interfaces  diagnostic_msgs  LICENSE        README.md  shape_msgs   std_srvs  trajectory_msgs

ws/src/ros2/common_interfaces/actionlib_msgs:
CMakeLists.txt  msg  package.xml

ws/src/ros2/common_interfaces/actionlib_msgs/msg:
GoalID.msg  GoalStatusArray.msg  GoalStatus.msg

ws/src/ros2/common_interfaces/builtin_interfaces:
CMakeLists.txt  msg  package.xml

ws/src/ros2/common_interfaces/builtin_interfaces/msg:
Duration.msg  Time.msg

ws/src/ros2/common_interfaces/diagnostic_msgs:
CMakeLists.txt  msg  package.xml  srv

ws/src/ros2/common_interfaces/diagnostic_msgs/msg:
DiagnosticArray.msg  DiagnosticStatus.msg  KeyValue.msg

ws/src/ros2/common_interfaces/diagnostic_msgs/srv:
AddDiagnostics.srv  SelfTest.srv

ws/src/ros2/common_interfaces/geometry_msgs:
CMakeLists.txt  msg  package.xml

ws/src/ros2/common_interfaces/geometry_msgs/msg:
Accel.msg                       PointStamped.msg        PoseWithCovarianceStamped.msg  TwistWithCovarianceStamped.msg
AccelStamped.msg                Polygon.msg             Quaternion.msg                 Vector3.msg
AccelWithCovariance.msg         PolygonStamped.msg      QuaternionStamped.msg          Vector3Stamped.msg
AccelWithCovarianceStamped.msg  Pose2D.msg              Transform.msg                  Wrench.msg
Inertia.msg                     PoseArray.msg           TransformStamped.msg           WrenchStamped.msg
InertiaStamped.msg              Pose.msg                Twist.msg
Point32.msg                     PoseStamped.msg         TwistStamped.msg
Point.msg                       PoseWithCovariance.msg  TwistWithCovariance.msg

ws/src/ros2/common_interfaces/nav_msgs:
CMakeLists.txt  msg  package.xml  srv

ws/src/ros2/common_interfaces/nav_msgs/msg:
GridCells.msg  MapMetaData.msg  OccupancyGrid.msg  Odometry.msg  Path.msg

ws/src/ros2/common_interfaces/nav_msgs/srv:
GetMap.srv  GetPlan.srv  SetMap.srv

ws/src/ros2/common_interfaces/sensor_msgs:
CMakeLists.txt  include  msg  package.xml  srv

ws/src/ros2/common_interfaces/sensor_msgs/include:
sensor_msgs

ws/src/ros2/common_interfaces/sensor_msgs/include/sensor_msgs:
distortion_models.hpp  image_encodings.hpp  impl  point_cloud2_iterator.hpp

ws/src/ros2/common_interfaces/sensor_msgs/include/sensor_msgs/impl:
point_cloud2_iterator.hpp

ws/src/ros2/common_interfaces/sensor_msgs/msg:
BatteryState.msg     Illuminance.msg       JoyFeedback.msg    MultiDOFJointState.msg  PointCloud.msg        Temperature.msg
CameraInfo.msg       Image.msg             Joy.msg            MultiEchoLaserScan.msg  PointField.msg        TimeReference.msg
ChannelFloat32.msg   Imu.msg               LaserEcho.msg      NavSatFix.msg           Range.msg
CompressedImage.msg  JointState.msg        LaserScan.msg      NavSatStatus.msg        RegionOfInterest.msg
FluidPressure.msg    JoyFeedbackArray.msg  MagneticField.msg  PointCloud2.msg         RelativeHumidity.msg

ws/src/ros2/common_interfaces/sensor_msgs/srv:
SetCameraInfo.srv

ws/src/ros2/common_interfaces/shape_msgs:
CMakeLists.txt  msg  package.xml

ws/src/ros2/common_interfaces/shape_msgs/msg:
Mesh.msg  MeshTriangle.msg  Plane.msg  SolidPrimitive.msg

ws/src/ros2/common_interfaces/std_msgs:
CMakeLists.txt  msg  package.xml

ws/src/ros2/common_interfaces/std_msgs/msg:
Bool.msg            Empty.msg              Header.msg           Int64.msg                MultiArrayLayout.msg  UInt32MultiArray.msg
Byte.msg            Float32.msg            Int16.msg            Int64MultiArray.msg      String.msg            UInt64.msg
ByteMultiArray.msg  Float32MultiArray.msg  Int16MultiArray.msg  Int8.msg                 UInt16.msg            UInt64MultiArray.msg
Char.msg            Float64.msg            Int32.msg            Int8MultiArray.msg       UInt16MultiArray.msg  UInt8.msg
ColorRGBA.msg       Float64MultiArray.msg  Int32MultiArray.msg  MultiArrayDimension.msg  UInt32.msg            UInt8MultiArray.msg

ws/src/ros2/common_interfaces/std_srvs:
CMakeLists.txt  package.xml  srv

ws/src/ros2/common_interfaces/std_srvs/srv:
Empty.srv  SetBool.srv  Trigger.srv

ws/src/ros2/common_interfaces/stereo_msgs:
CMakeLists.txt  msg  package.xml

ws/src/ros2/common_interfaces/stereo_msgs/msg:
DisparityImage.msg

ws/src/ros2/common_interfaces/trajectory_msgs:
CMakeLists.txt  msg  package.xml

ws/src/ros2/common_interfaces/trajectory_msgs/msg:
JointTrajectory.msg  JointTrajectoryPoint.msg  MultiDOFJointTrajectory.msg  MultiDOFJointTrajectoryPoint.msg

ws/src/ros2/common_interfaces/visualization_msgs:
CMakeLists.txt  msg  package.xml

ws/src/ros2/common_interfaces/visualization_msgs/msg:
ImageMarker.msg               InteractiveMarkerFeedback.msg  InteractiveMarker.msg      InteractiveMarkerUpdate.msg  Marker.msg
InteractiveMarkerControl.msg  InteractiveMarkerInit.msg      InteractiveMarkerPose.msg  MarkerArray.msg              MenuEntry.msg

ws/src/ros2/example_interfaces:
CMakeLists.txt  CONTRIBUTING.md  LICENSE  mapping_rules.yaml  package.xml  README.md  srv

ws/src/ros2/example_interfaces/srv:
AddTwoInts.srv

ws/src/ros2/poco_vendor:
cmake  CMakeLists.txt  LICENSE  poco_vendorConfig.cmake.in  poco_vendorConfig-version.cmake.in  README.md

ws/src/ros2/poco_vendor/cmake:
Modules

ws/src/ros2/poco_vendor/cmake/Modules:
FindPoco.cmake

ws/src/ros2/rcl_interfaces:
CONTRIBUTING.md  LICENSE  lifecycle_msgs  rcl_interfaces  README.md

ws/src/ros2/rcl_interfaces/lifecycle_msgs:
CMakeLists.txt  msg  package.xml  srv

ws/src/ros2/rcl_interfaces/lifecycle_msgs/msg:
State.msg  TransitionDescription.msg  TransitionEvent.msg  Transition.msg

ws/src/ros2/rcl_interfaces/lifecycle_msgs/srv:
ChangeState.srv  GetAvailableStates.srv  GetAvailableTransitions.srv  GetState.srv

ws/src/ros2/rcl_interfaces/rcl_interfaces:
CMakeLists.txt  msg  package.xml  srv

ws/src/ros2/rcl_interfaces/rcl_interfaces/msg:
IntraProcessMessage.msg   ParameterDescriptor.msg        ParameterEvent.msg  ParameterType.msg   SetParametersResult.msg
ListParametersResult.msg  ParameterEventDescriptors.msg  Parameter.msg       ParameterValue.msg

ws/src/ros2/rcl_interfaces/rcl_interfaces/srv:
DescribeParameters.srv  GetParameters.srv  GetParameterTypes.srv  ListParameters.srv  SetParametersAtomically.srv  SetParameters.srv

ws/src/ros2/rmw_fastrtps:
CONTRIBUTING.md  fastrtps_cmake_module  LICENSE  rmw_fastrtps_cpp

ws/src/ros2/rmw_fastrtps/fastrtps_cmake_module:
cmake  CMakeLists.txt  fastrtps_cmake_module-extras.cmake  package.xml

ws/src/ros2/rmw_fastrtps/fastrtps_cmake_module/cmake:
Modules

ws/src/ros2/rmw_fastrtps/fastrtps_cmake_module/cmake/Modules:
FindFastRTPS.cmake

ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp:
CMakeLists.txt  include  package.xml  rmw_fastrtps_cpp-extras.cmake  src

ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/include:
rmw_fastrtps_cpp

ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/include/rmw_fastrtps_cpp:
custom_client_info.hpp       get_client.hpp       identifier.hpp               ServiceTypeSupport.hpp       writer_info.hpp
custom_participant_info.hpp  get_participant.hpp  macros.hpp                   ServiceTypeSupport_impl.hpp
custom_publisher_info.hpp    get_publisher.hpp    MessageTypeSupport.hpp       TypeSupport.hpp
custom_service_info.hpp      get_service.hpp      MessageTypeSupport_impl.hpp  TypeSupport_impl.hpp
custom_subscriber_info.hpp   get_subscriber.hpp   reader_info.hpp              visibility_control.h

ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/src:
assign_partitions.hpp      namespace_prefix.cpp                   rmw_node.cpp                         rmw_topic_names_and_types.cpp
client_service_common.cpp  namespace_prefix.hpp                   rmw_node_names.cpp                   rmw_trigger_guard_condition.cpp
client_service_common.hpp  qos.cpp                                rmw_publish.cpp                      rmw_wait.cpp
demangle.cpp               qos.hpp                                rmw_publisher.cpp                    rmw_waitset.cpp
demangle.hpp               rmw_client.cpp                         rmw_request.cpp                      ros_message_serialization.cpp
get_client.cpp             rmw_compare_gids_equal.cpp             rmw_response.cpp                     ros_message_serialization.hpp
get_participant.cpp        rmw_count.cpp                          rmw_service.cpp                      types
get_publisher.cpp          rmw_get_gid_for_publisher.cpp          rmw_service_names_and_types.cpp      type_support_common.cpp
get_service.cpp            rmw_get_implementation_identifier.cpp  rmw_service_server_is_available.cpp  type_support_common.hpp
get_subscriber.cpp         rmw_guard_condition.cpp                rmw_subscription.cpp
identifier.cpp             rmw_init.cpp                           rmw_take.cpp

ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/src/types:
custom_waitset_info.hpp  guard_condition.hpp

ws/src/ros2/system_tests:
CONTRIBUTING.md  LICENSE  test_communication  test_rclcpp

ws/src/ros2/system_tests/test_communication:
CMakeLists.txt  msg  package.xml  srv  test

ws/src/ros2/system_tests/test_communication/msg:
BoundedArrayNested.msg      DynamicArrayNested.msg            Empty.msg               Primitives.msg             UInt32.msg
BoundedArrayPrimitives.msg  DynamicArrayPrimitives.msg        FieldsWithSameType.msg  StaticArrayNested.msg
Builtins.msg                DynamicArrayPrimitivesNested.msg  Nested.msg              StaticArrayPrimitives.msg

ws/src/ros2/system_tests/test_communication/srv:
Empty.srv  Primitives.srv

ws/src/ros2/system_tests/test_communication/test:
__init__.py           service_fixtures.hpp                     test_publisher_subscriber.cpp    test_secure_publisher_subscriber.py.in
message_fixtures.hpp  service_fixtures.py                      test_publisher_subscriber.py.in  test_secure_subscriber.cpp
message_fixtures.py   subscriber_py.py                         test_replier.cpp                 test_security_files
publisher_py.py       test_invalid_secure_node_creation_c.cpp  test_requester.cpp               test_subscriber.cpp
replier_py.py         test_messages_c.cpp                      test_requester_replier.py.in     test_subscription_valid_data.cpp
requester_py.py       test_publisher.cpp                       test_secure_publisher.cpp

ws/src/ros2/system_tests/test_communication/test/test_security_files:
1000.pem  1004.pem     ca.key.pem      index.txt           publisher               serial.old
1001.pem  1005.pem     ecdsaparam      index.txt.attr      publisher_invalid_cert  subscriber
1002.pem  ca.cert.pem  governance.p7s  index.txt.attr.old  publisher_missing_key   test_security_policies.yaml
1003.pem  ca_conf.cnf  governance.xml  index.txt.old       serial

ws/src/ros2/system_tests/test_communication/test/test_security_files/publisher:
ca.cert.pem  cert.pem  ecdsaparam  governance.p7s  key.pem  permissions.p7s  permissions.xml  req.pem  request.cnf

ws/src/ros2/system_tests/test_communication/test/test_security_files/publisher_invalid_cert:
ca.cert.pem  cert.pem  ecdsaparam  governance.p7s  key.pem  permissions.p7s  permissions.xml  req.pem  request.cnf

ws/src/ros2/system_tests/test_communication/test/test_security_files/publisher_missing_key:
ca.cert.pem  cert.pem  ecdsaparam  governance.p7s  permissions.p7s  permissions.xml  req.pem  request.cnf

ws/src/ros2/system_tests/test_communication/test/test_security_files/subscriber:
ca.cert.pem  cert.pem  ecdsaparam  governance.p7s  key.pem  permissions.p7s  permissions.xml  req.pem  request.cnf

ws/src/ros2/system_tests/test_rclcpp:
CMakeLists.txt  include  msg  package.xml  srv  test

ws/src/ros2/system_tests/test_rclcpp/include:
test_rclcpp

ws/src/ros2/system_tests/test_rclcpp/include/test_rclcpp:
utils.hpp

ws/src/ros2/system_tests/test_rclcpp/msg:
UInt32.msg

ws/src/ros2/system_tests/test_rclcpp/srv:
AddTwoInts.srv

ws/src/ros2/system_tests/test_rclcpp/test:
parameter_fixtures.hpp                        test_remote_parameters.cpp
pub_sub_fixtures.hpp                          test_repeated_publisher_subscriber.cpp
test_avoid_ros_namespace_conventions_qos.cpp  test_services_client.cpp
test_client_scope_client.cpp                  test_services_in_constructor.cpp
test_client_scope_consistency_client.cpp      test_services_server.cpp
test_client_scope_consistency_server.cpp      test_signal_handler_after_shutdown__expected_output.txt
test_client_scope_server.cpp                  test_signal_handler_after_shutdown__trigger_shutdown.txt
test_client_wait_for_service_shutdown.cpp     test_signal_handler_before_shutdown__expected_output.txt
test_executable_output.py.in                  test_signal_handler_before_shutdown__trigger_shutdown.txt
test_executor.cpp                             test_signal_handler.cpp
test_intra_process.cpp                        test_spin.cpp
test_local_parameters.cpp                     test_subscription.cpp
test_multiple_service_calls.cpp               test_timeout_subscriber.cpp
test_multithreaded.cpp                        test_timer.cpp
test_parameters_server.cpp                    test_two_executables.py.in
test_publisher.cpp

[wjwwood]

I think I've seen this in the past too, usually this happens on Windows though when a file is in use (Windows will prevent you from deleting a file if it is in use), but that's not usually an issue on Linux.

It might occur if files are still being added to the folder while trying to remove it. I think rm works like this:

• list all files
• remove each
• remove each directory as needed

But if you add a new file or directory to a directory between the first and third steps it might fail. It's possible this occurred because the abort did not kill all of the build processes? Maybe some other service was using the directory? I don't know. Without more information I don't know how else to debug it.

[nuclearsandwich]

Closing this until we have a resurgence of the issue or a way to reproduce the problem otherwise.

[mjcarroll]

We had (potentially) another instance of this failure, this time on MacOS:

https://ci.ros2.org/job/nightly_osx_release/865/

08:25:58 [nightly_osx_release] $ /bin/sh -xe /var/folders/b0/xvj8bp0n0zjf9r_bw5k2vqmw0000gn/T/jenkins268786503312576398.sh
08:25:58 + rm -rf ws workspace 'work space'
08:26:02 rm: ws/log/build_2018-05-21_00-48-13: Directory not empty
08:26:02 rm: ws/log: Directory not empty
08:26:03 rm: ws: Directory not empty

[sloretz]

Another possible instance of this https://ci.ros2.org/job/ci_osx/3705/

+ rm -rf ws workspace 'work space'
rm: ws/build: Directory not empty
rm: ws/log/test_2018-06-01_17-16-46/nav_msgs: Directory not empty
rm: ws/log/test_2018-06-01_17-16-46: Directory not empty
rm: ws/log: Directory not empty
rm: ws: Directory not empty

Instead of being aborted by a user, the previous job on mini2 died with a java stack trace in the console output https://ci.ros2.org/job/ci_osx/3701/console

[clalancette]

There was another instance on the nightly last night: https://ci.ros2.org/view/nightly/job/nightly_osx_release/889/

[nuclearsandwich]

Instances of this appear to have waned again. I believe the lack of cleanup is related to the connection drop and ssh-agent issues in the sense that a lost connection between the Jenkins instance and agent will lead to a bad state on the agent where the bad state consists of dangling workspace directories and running ssh-agent processes.

[dirk-thomas]

Closing as duplicate of #161.