Source code, data, presentation, and notes for my master’s thesis on consent dialogs in mobile apps on Android and iOS
Consent dialogs have become ubiquitous with seemingly every website and app pleading users to agree to their personal data being processed and their behaviour being tracked, often with the help of tens or even hundreds of third-party companies. They are an effort by website and app publishers to comply with data protection legislation like the GDPR, which imposes strict limits on how companies can process data. Previous research has established that companies often apply dark patterns to illegally nudge users into agreeing and that at the same time tracking is more common than ever with both websites and apps regularly automatically transmitting telemetry data.
But so far, there has been almost no research into consent dialogs on mobile. For my master’s thesis, I studied consent dialogs on Android and iOS in an automated and dynamic manner, analysing 4,388 popular apps from both platforms. I identified different types of consent elements in the apps and analysed their prevalence. I also identified dark patterns and violations by the apps based on a list of criteria for a legally compliant consent dialog that I have compiled. Finally, I measured the effect of the user’s choice in the consent dialog by comparing the traffic from before any interaction with the traffic after accepting and rejecting the dialog and analysing contacted trackers and transmitted data types. The results show that more than 90 % of consent dialogs implement at least one dark pattern and that a majority of apps transmits tracking data regardless of consent status.
- Run
brew install postgresql
to be able to installpsycopg2
later (ref). - Install
libimobiledevice
:brew install libimobiledevice ideviceinstaller
- Install
sshpass
:brew install esolitos/ipa/sshpass
- SSH into the iDevice manually once to add it to the list of known hosts.
npm install -g appium appium-doctor
# Run `appium-doctor --android` or `appium-doctor --ios` and resolve the issues.
git clone https://github.com/baltpeter/thesis-mobile-consent-dialogs
cd thesis-mobile-consent-dialogs
yarn
python3 -m venv venv
source venv/bin/activate
pip install mitmproxy frida-tools objection python-dotenv psycopg2-binary
cd src
cp .env.sample .env
# Edit the values accordingly.
nano .env
based on: https://appium.io/docs/en/drivers/ios-xcuitest-real-devices/ and https://makaka.org/unity-tutorials/test-ios-app-without-developer-account
- Create an Apple developer account (free is fine) and use that to log into XCode.
- Connect the iPhone via USB.
- Start XCode. In the menu bar, click Window and Devices and Simulators. Ensure the iPhone is available there.
- In a terminal, start
appium
. - Follow the steps at https://appium.io/docs/en/drivers/ios-xcuitest-real-devices/#full-manual-configuration (full manual configuration).
- Open Appium Inspector and create a session with the following desired capabilities to test (adapted to your configuration):
{
"appium:xcodeOrgId": "<org-id>",
"appium:xcodeSigningId": "Apple Development",
"appium:udid": "auto",
"platformName": "iOS",
"appium:app": "/path/to/app.ipa",
"appium:automationName": "XCUITest",
"appium:deviceName": "iPhone",
"appium:updatedWDABundleId": "tld.your-domain.WebDriverAgentRunner"
}
Replace appium:app
with the path to any IPA, appium:updatedWDABundleId
with the ID chosen in the previous step, and appium:xcodeOrgId
with the personal team ID. To find that, open the Keychain Access app. On the left, select login under Default Keychains, then choose My Certificates in the top bar. Doubleclick the correct certificate. The ID is listed under Organizational Unit.
6. Use the latter two values for running the analysis.
- Jailbreak using checkra1n (only works on iOS 14).
- Enable SSH server.
- Set the following settings:
- General
- Background App Refresh: off (to hopefully minimize background network traffic)
- Software Update
- Automatic Updates
- Install iOS Updates: off
- Download iOS Updates: off
- Automatic Updates
- Display & Brightness
- Auto-Lock: never
- Privacy
- Location Services
- Location Services: on
- Analytics & Improvements
- Share iPhone Analytics: off
- Apple Advertising
- Personalised Ads: on (default)
- Location Services
- App Store
- Automatic Downloads
- Apps: off
- App Updates: off
- Automatic Downloads
- General
- Install Activator.
- Setup mitmproxy: https://www.andyibanez.com/posts/intercepting-network-mitmproxy/#physical-ios-devices
- Install SSL Kill Switch 2 (https://steipete.com/posts/jailbreaking-for-ios-developers/#ssl-kill-switch)
- Install Debian Packager, Cydia Substrate, PreferenceLoader, PullToRespring, and Filza from Cydia.
- Download latest release: https://github.com/nabla-c0d3/ssl-kill-switch2/releases
- In Filza, go to
/private/var/mobile/Library/Mobile Documents/com~apple~CloudDocs/Downloads
and install. - Respring by opening Settings and pulling down.
- Enable in Settings under SSL Kill Switch 2.
- Install Frida.
- Uninstall all third-party apps that are not absolutely necessary.
- Turn on Bluetooth.
- Make sure to use an x86_64 emulator (Android 11).
- Uninstall unnecessary Google apps to avoid their background traffic:
adb shell 'pm uninstall --user 0 com.android.chrome' adb shell 'pm uninstall --user 0 com.google.android.apps.docs' adb shell 'pm uninstall --user 0 com.google.android.apps.maps' adb shell 'pm uninstall --user 0 com.google.android.apps.messaging' adb shell 'pm uninstall --user 0 com.google.android.apps.photos' adb shell 'pm uninstall --user 0 com.google.android.apps.pixelmigrate' adb shell 'pm uninstall --user 0 com.google.android.apps.wellbeing' adb shell 'pm uninstall --user 0 com.google.android.apps.youtube.music' adb shell 'pm uninstall --user 0 com.google.android.gm' adb shell 'pm uninstall --user 0 com.google.android.googlequicksearchbox' adb shell 'pm uninstall --user 0 com.google.android.videos' adb shell 'pm uninstall --user 0 com.google.android.youtube' adb shell 'pm uninstall --user 0 com.google.mainline.telemetry'
- Disable captive portal checks to further reduce background noise:
adb shell 'settings put global captive_portal_detection_enabled 0' adb shell 'settings put global captive_portal_server localhost' adb shell 'settings put global captive_portal_mode 0'
- Install the mitmproxy certificate as root CA (emu needs to be started with
-writable-system
):# Yields <hash>. openssl x509 -inform PEM -subject_hash_old -in ~/.mitmproxy/mitmproxy-ca-cert.pem | head -1 cp ~/.mitmproxy/mitmproxy-ca-cert.pem <hash>.0 adb root adb shell avbctl disable-verification adb disable-verity adb reboot adb root adb remount adb push <hash>.0 /sdcard/ adb shell 'mv /sdcard/<hash>.0 /system/etc/security/cacerts/' adb shell 'chmod 644 /system/etc/security/cacerts/<hash>.0' adb reboot
- Install Frida on the emulator (note that the version must match the one on the host):
wget https://github.com/frida/frida/releases/download/15.1.12/frida-server-15.1.12-android-x86_64.xz 7z x frida-server-15.1.12-android-x86_64.xz adb push frida-server-15.1.12-android-x86_64 /data/local/tmp/frida-server adb shell 'chmod 777 /data/local/tmp/frida-server' adb root adb shell 'nohup /data/local/tmp/frida-server >/dev/null 2>&1 &' # Should have `frida-server`. frida-ps -U | grep frida
This is the honey data that I set in my analysis. You should probably change it. Remember to also change query.ts accordingly.
- Contact:
JGKfozntbF TBFFZbBYea
, 0155 57543434,RYnlSPbEYh@bn.al
,https://q8phlLSJgq.bn.al
,N2AsWEMI5D 565, 859663 p0GdKDTbYV
- Messages:
9FBqD2CNIJ
(to|from) +49 155 75734343 - Calls from and to +49 155 75734343 (Android only)
- Calendar:
fWAs4GFbpN
, aturscf2178L
, 2022-08-14T08:56 (iOS only) - Reminder:
b5jHg3Eh1k
,HQBOdx4kx2
(scheduled for 2022-08-02T13:38) (iOS only) - Health details: Name
DkwIXobsJN t5TfTlezmn
, DOB 1973-05-15, female, height 146cm, weight 108.5kg (iOS only) - Home data: Rooms
bEZf1h06j1
,DX7BgPtH99
(basement); second homeg1bVNue3On
(iOS only) - Device name:
R2Gl5OLv20
- Note:
S0Ei7sFP9b
(iOS only) - Phone number: +49 155 85834346 (Android only)
- Location: Schreinerweg 6, 38126 Braunschweig; 52.235288, 10.564235
- Apple ID (iOS only)
- Clipboard:
LDDsvPqQdT
This code is licensed under the MIT license, see the LICENSE
file for details.
This repository is archived in the state it was in when I submitted my thesis. I will not be updating it any further. However, I am working together with @zner0L on a follow-up project. As a part of this project, we will also make sure to extract reusable libraries and tools from this research code and document them more thouroughly. Follow me and datarequests.org on Mastodon to stay up to date on that.