Adding file size to views #3539

ashleytqy · 2018-04-17T15:56:20Z

This PR addresses: #3521

Adds file size to files + notebooks, but not for directories (it's undefined for directories)
Displays it in the view, with sorting functionality
Filesize is humanized (originally in bytes)

@ckilcrease

Sort by ascending

…nto filesize

Carreau · 2018-04-18T00:24:36Z

Thanks,

I haven't touched this part of the code base in a while so i'll defer to someone else for complete review.

You added looking at filesize in each model when doing a Get request. Was that necessary to get that in directory listing ? Or is the server doing a os.stat on every file when listing a directory ? If is is there might be a perf bottle neck we need to be careful there.

You seem to have 16 commits and a collaborator (good job to both of you for handling Git). Do you wish to reduce your commit number (rebase), to make history cleaner.

From your screenshot I can see that some files are 2B and below, and actually fraction of a Bytes.... Are you sure this is right ? Is there not a factor of 1000 somewhere ? (feel free to use an external library, assuming it is small enough, the license is right... etc). )

I'll do further inline comments.

Carreau · 2018-04-18T00:26:21Z

notebook/services/api/api.yaml

@@ -814,6 +814,10 @@ definitions:
        type: string
        description: Last modified timestamp
        format: dateTime
+      size:
+        type: integer


Integer or None ? If size can't be found, probably say that in the description at least.

Carreau · 2018-04-18T00:27:37Z

notebook/static/tree/js/notebooklist.js

+        $("<span/>")
+            .addClass("file_size")
+            .addClass("pull-right")
+            .css("width", "65px")


You likely want to avoid inline css.

do you know where the css should go? custom.css seems like a user-specific file, and not quite right. from what i've seen, most of the CSS is defined like this:

notebook/notebook/static/tree/js/notebooklist.js

Lines 1250 to 1263 in 8034d71

var add_uploading_button = function (f, item) {

// change buttons, add a progress bar

var uploading_button = item.find('.upload_button').text("Uploading");

uploading_button.off('click'); // Prevent double upload

var progress_bar = $('<span/>')

.addClass('progress-bar')

.css('top', '0')

.css('left', '0')

.css('width', '0')

.css('height', '3px')

.css('border-radius', '0 0 0 0')

.css('display', 'inline-block')

.css('position', 'absolute');

Likely in notebook/static/tree/less/tree.less there are case when we do inline css, there might be a reason for progress-bar, maybe not, if not that's likely technical dept we need to fix at some point.

You'll likely need to recompile thee css with python setup.py css after modification of the less file.

Carreau · 2018-04-18T00:33:06Z

notebook/static/tree/js/notebooklist.js

@@ -835,6 +866,7 @@ define([
        // Add in the date that the file was last modified
        item.find(".item_modified").text(utils.format_datetime(model.last_modified));
        item.find(".item_modified").attr("title", moment(model.last_modified).format("YYYY-MM-DD HH:mm"));
+        item.find(".file_size").html(utils.format_filesize(model.size) || "&nbsp;");


the use of .html() should be a red flag. From a security standpoint it would not pass an audit. We had security issues before because of HTML usage.

If somewhere where to replace you API to return cannot find <size> for file <file> the you get scrip injection in your page, and your client is compromised.

If you've seen use of .html() in other place of the javascript where .text() should/could be used, please open an issue.

re: using .html() – we're using it so that   renders because .text() doesn't display an empty string. do you have any suggestions to how to display a   without using .html?

it's also used here, which i'm assuming the author also wanted to use the   character

notebook/notebook/static/tree/js/notebooklist.js

Line 763 in 8034d71

$('#counter-select-all').html(checked===0 ? ' ' : checked);

Good question, there are several way to approach that. A simple one would be to use .html() just for  :

if (cfoo){ $(...).text(foo); } else { $(...).html("&nbsp"); }

That reduce the use of html so it's one less risk.

Thinking about it more,   is also just an escape sequence for unicode character Non Breaking Space, which you can enter with your keyboard using some clever keyboard combinaison | | (nbsp), not like | |(space), check the source, on mac it's Alt Space. That's 1) annoying to type 2) annoying to see when reading code. That's not optimal. After thinking a few minutes more you can realize that hopefully most programming language allow you to have escape sequence, knowing that Non-Breaking space is unicode U+00A0 you can go with :

.text(checked===0 ? '\xA0' : checked);

If you want to even avoid using jQuery you can use the innerText attribute and assign to it.

With your gained knowledge you can even open an issue for $('#counter-select-all').html(checked===0 ? ' ' : checked); to be fixed.

As an exercise – if you like the fun of hacking – get in a shoes of an attacker and try to modify the server side to not return the size as an integer but a string, and attempt to have a script injection that display a message in the frontend.

Stepping back a bit from unicode discussions: do we actually need a non-breaking space at all? I think the layout should still work if the element has no text.

I fixed it with using \xA0 (thanks @Carreau !) . Without text, the span element wrapping it doesn't seem to adhere to the width it's given.

adding sort function add file size text

fixed typo

ashleytqy · 2018-04-18T04:07:09Z

@Carreau I tried to install and use pretty-bytes with bower but the library doesn't seem compatible with bower (it threw a module is not recognized error, so i think the library is better suited for an npm install which notebook doesn't use), so i copied over the code with credits if thats okay with you

takluyver · 2018-04-18T14:43:18Z

notebook/services/api/api.yaml

+      size:
+        type: integer
+        description: "The size of the file or directory."
+        format: bytes


I think this should just go in the description. Swagger specifies a fixed list of formats for things like dateTime. From the spec's point of view, this is just a number - I don't think it has a way of declaring a unit.

takluyver · 2018-04-18T14:46:45Z

notebook/services/contents/filemanager.py

@@ -183,6 +183,14 @@ def is_hidden(self, path):
        os_path = self._get_os_path(path=path)
        return is_hidden(os_path, self.root_dir)

+    def _get_file_size(self, path):
+        try:
+            size = os.path.getsize(path)


This will do a new stat on the file, which is best avoided for performance reasons. We already get a stat object in _base_model() - perhaps we can use the size from that, and remove it again for directories. It's a bit of an ugly trick, but for a big directory, it could make a difference.

@takluyver thank you -- I've updated so that it uses file size from st_size (retrieved from stat) in _base_model(), but let me know if this isn't what you meant

takluyver · 2018-04-18T14:50:20Z

notebook/static/base/js/utils.js

@@ -1038,7 +1038,34 @@ define([
            return (order == 1) ? 1 : -1;
        }
    };
+
+    // source: https://github.com/sindresorhus/pretty-bytes


I'm OK with this since it's one fairly small function, and it's unlikely to change. But we should include a copy of the license in the comment.

https://github.com/sindresorhus/pretty-bytes/blob/master/license

takluyver · 2018-04-18T14:54:38Z

Looking at the screenshots, I think there should be a bit more whitespace between the last modified time and the size, so it's clear that they're separate fields. Maybe right-align the sizes?

takluyver · 2018-04-18T20:11:14Z

Thanks, that was what I meant about using the existing stat info. :-)

I think this is looking good now, though I'm too tired to review it closely. But there's some problem with the JS tests on Travis - can you investigate?

Carreau · 2018-04-18T21:35:15Z

Travis seem happy, Appveyor was not but this seem like an unrelated error. I restarted he build.

Carreau

Good job !

takluyver · 2018-04-19T07:10:32Z

notebook/services/contents/filemanager.py

+        try:
+            # size of file 
+            size = info.st_size
+        except (ValueError, OSError):


I don't think there can be an error here - if lstat() returns, then we should have a valid stat object from which we can get st_size. But it shouldn't be a problem to be defensive in case I'm wrong.

takluyver · 2018-04-19T07:12:28Z

notebook/services/api/api.yaml

@@ -814,6 +814,9 @@ definitions:
        type: string
        description: Last modified timestamp
        format: dateTime
+      size:
+        type: integer
+        description: "The size of the file or notebook in bytes. If no size is provided, defaults to None."


Since this is describing a JSON API, it's probably better to refer to null, which is the JSON/Javascript name, rather than None, which is from Python. This isn't worth holding the PR up for, so I'll change it myself and then merge.

takluyver · 2018-04-19T07:40:01Z

Thanks!

ashleytqy and others added 14 commits April 13, 2018 21:41

adding file size to model

3650cc7

make file size optional pproperty

b864f5a

only get filesize for files and notebooks

3315dbd

starting out the sort function

388d636

add file size text

38acc78

revert chages

59b7c66

fix sort function

86c45a1

fix sorting function foreal

a299be7

empty function to format filesize

6475eaa

change column width, flip order

d6c7f2e

converting filesize to readable format

42c3c02

cleanup

8c96ad5

fixed filesize

d19f970

Merge branch 'filesize' of https://github.com/nyu-ossd-s18/notebook i…

a47a188

…nto filesize

Carreau reviewed Apr 18, 2018

View reviewed changes

ashleytqy force-pushed the filesize branch from 4c4622f to 2ef8962 Compare April 18, 2018 02:21

ckilcrease and others added 4 commits April 17, 2018 22:22

only get filesize for files and notebooks

ff8e1cc

adding sort function add file size text

change column width, flip order

abe05de

converting filesize to readable format

13933f3

fixed filesize

acc2056

fixed typo

ashleytqy force-pushed the filesize branch from 2ef8962 to acc2056 Compare April 18, 2018 02:23

use pretty-bytes

2e9ba4a

merged changes

8cc9734

takluyver reviewed Apr 18, 2018

View reviewed changes

remove inline css

06214ae

takluyver reviewed Apr 18, 2018

View reviewed changes

ashleytqy and others added 5 commits April 18, 2018 12:31

right align file size column

d35ac8b

add MIT licence

dd608ad

update api description

577cbe5

use text() not html()

dee58e0

get file size in base model

174e724

remove es6 syntax

192e3fe

Carreau approved these changes Apr 18, 2018

View reviewed changes

takluyver reviewed Apr 19, 2018

View reviewed changes

None -> null for JSON API

160754d

takluyver merged commit 5875235 into jupyter:master Apr 19, 2018

takluyver added this to the 5.5 milestone Apr 24, 2018

takluyver mentioned this pull request Apr 24, 2018

see file size in file browser #3521

Closed

github-actions bot added the status:resolved-locked label Apr 1, 2021

github-actions bot locked as resolved and limited conversation to collaborators Apr 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding file size to views #3539

Adding file size to views #3539

ashleytqy commented Apr 17, 2018 •

edited

Loading

Carreau commented Apr 18, 2018

Carreau Apr 18, 2018

Carreau Apr 18, 2018

ashleytqy Apr 18, 2018 •

edited

Loading

Carreau Apr 18, 2018

Carreau Apr 18, 2018

ashleytqy Apr 18, 2018 •

edited

Loading

Carreau Apr 18, 2018

takluyver Apr 18, 2018

ashleytqy Apr 18, 2018 •

edited

Loading

ashleytqy commented Apr 18, 2018 •

edited

Loading

takluyver Apr 18, 2018

takluyver Apr 18, 2018

ckilcrease Apr 18, 2018

takluyver Apr 18, 2018

takluyver commented Apr 18, 2018

takluyver commented Apr 18, 2018

Carreau commented Apr 18, 2018

Carreau left a comment

takluyver Apr 19, 2018

takluyver Apr 19, 2018

takluyver commented Apr 19, 2018

	var add_uploading_button = function (f, item) {
	// change buttons, add a progress bar
	var uploading_button = item.find('.upload_button').text("Uploading");
	uploading_button.off('click'); // Prevent double upload
	var progress_bar = $('<span/>')
	.addClass('progress-bar')
	.css('top', '0')
	.css('left', '0')
	.css('width', '0')
	.css('height', '3px')
	.css('border-radius', '0 0 0 0')
	.css('display', 'inline-block')
	.css('position', 'absolute');

Adding file size to views #3539

Adding file size to views #3539

Conversation

ashleytqy commented Apr 17, 2018 • edited Loading

Sort by ascending

Carreau commented Apr 18, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashleytqy Apr 18, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashleytqy Apr 18, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashleytqy Apr 18, 2018 • edited Loading

Choose a reason for hiding this comment

ashleytqy commented Apr 18, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

takluyver commented Apr 18, 2018

takluyver commented Apr 18, 2018

Carreau commented Apr 18, 2018

Carreau left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

takluyver commented Apr 19, 2018

ashleytqy commented Apr 17, 2018 •

edited

Loading

ashleytqy Apr 18, 2018 •

edited

Loading

ashleytqy Apr 18, 2018 •

edited

Loading

ashleytqy Apr 18, 2018 •

edited

Loading

ashleytqy commented Apr 18, 2018 •

edited

Loading