Difference between revisions of "Disk Quotas"

 
(51 intermediate revisions by 5 users not shown)
Line 1: Line 1:
On the '''''CS Ugrad Net''''', the Ugrad accounts may be subject to disk quotas, which are limits on the total amount of data you are permitted to store in your home directory (or, in some cases, other directories).  If you exceed a quota, you will be unable to create new files or add to existing files until you reduce the total mount of disk space you're using to go below your quota.
 
  
'''NOTE:'''  The '''''CS Grad/Research Net''''' does not currently have disk quotas enabled.  But, we ask that users on the CS Grad/Research Net be mindful that you are ''sharing disk space with much of CS''.
+
'''CS ''UGRAD'' Net Users:''' On the '''''CS Ugrad Net''''', the Ugrad accounts ''are subject to '''disk quotas''''', which are limits on the total amount of data you are permitted to store in your home directory (or, in some cases, other directories).  If you exceed a quota, you will be unable to create new files or add to existing files until you reduce the total amount of disk space you're using to go below your quota.
 +
 
 +
'''CS ''GRAD'' Net Users:'''  The '''''CS Grad/Research Net''''' does not currently have disk quotas enabled.  But, we ask that users on the CS Grad/Research Net be mindful that you are ''sharing disk space with much of CS''. If you need to store large amounts of data on the CS Grad Net, please see:  [[Storing Large Amounts Of Data On Our Systems]]
  
 
== How Quotas Work on the ''CS Ugrad Net'' ==
 
== How Quotas Work on the ''CS Ugrad Net'' ==
  
Every account that is subject to a quota has a ''soft limit'' and a ''hard limit''.  For most of our CS Ugrad accounts, those are ''8 GiB'' and ''8.5 GiB'', respectivelyYou cannot use more disk space than is given by your hard limit; once you hit that limit, all attempts to allocate more space (creating new files, making existing files larger, etc.) will fail.  
+
Every account that is subject to a quota has limit on the total amount of disk space used by that account's files.  For most of our CS Ugrad accounts, that limit is '''10 GiB'''.  Once you hit that limit, all attempts to allocate more space (creating new files, making existing files larger, etc.) will fail.  
  
If you exceed your soft limit, you will have a grace period of a week.  During that period of time you may continue to create files and add to existing files as normal (as long as you don't hit your hard limit).  If you remain above your soft limit for a full week, the soft limit begins to behave like a hard limit and you will be unable to use any additional space until you reduce your total disk usage below your soft limit.
+
''If you cannot allocate more space (because you've hit your quota limit), '''you might not be able to log in via the GUI on our Ugrad computer lab Linux systems'''.  If that happens, you will have to SSH in remotely and reduce your disk usage below your quota.''
  
''If you cannot allocate more space (you've hit your hard limit or you've exceeded your soft limit for more than a week), '''you might not be able to log in via the GUI on our Ugrad computer lab Linux systems'''.  If that happens, you will have to SSH in remotely and reduce your disk usage below your quota.''
+
==='''''{{red|IMPORTANT NOTE:}}  If You Are Using A ''Cloud Drive'' ('''Dropbox''', '''Google Drive''', etc.)===
  
The CS Department has a notification system in place that will email ''your CS Ugrad account'' if you exceed your disk quota.  This is a courtesy; you are ultimately responsible for keeping track of whether you're within the limits set for your account.
+
:If you have your CS Ugrad Linux account linked to a cloud drive such as '''Dropbox''' or '''Google Drive''', please note that those services may download local copies of your cloud files into your Ugrad account.  This is an extremely quick way to go over your CS Ugrad disk quota.  We recommend you not link cloud drives with your CS Linux account (unless your cloud drive is very small and well within your CS Ugrad disk quota.) We cannot accomodate users' cloud data within our provided home directory disk resources.
  
 
== How to Check Your Quota on the ''CS Ugrad Net'' ==
 
== How to Check Your Quota on the ''CS Ugrad Net'' ==
Line 21: Line 22:
 
If it doesn't print anything, you are not subject to any quotas.  If it does print something, the output will look like this:
 
If it doesn't print anything, you are not subject to any quotas.  If it does print something, the output will look like this:
  
  Disk quotas for user ''account'' (uid ''12345''):
+
  ''account'': 1.6GiB used out of 10.0GiB quota
      Filesystem  space  quota  limit  grace  files  quota  limit  grace
 
barley.cs.jhu.edu:/home
 
                  993776  4000000 4500000            9687      0      0
 
  
The numbers represent 1024-byte blocksIn the above example, the person is using 993776 blocks, which translates to roughly 970 MiB of data. The quota (aka soft limit) is 4000000 blocks, or just under 4 GiB.  The hard limit is 4500000 blocks, or a little under 4.5 GiB.
+
This should be fairly self-explanatoryThe person (with account name "''account''") has files totaling 1.6 gigabytes of data. The account's usage limit is 10 gigabytes.
  
 
== How to Get Yourself Under Quota on the ''CS Ugrad Net''==
 
== How to Get Yourself Under Quota on the ''CS Ugrad Net''==
Line 32: Line 30:
 
If you're over quota, the only way to get back under is to delete or shrink files until your total disk usage is reduced below your limit.
 
If you're over quota, the only way to get back under is to delete or shrink files until your total disk usage is reduced below your limit.
  
You can use the <code>du</code> command line program to see how much space individual files or directories are using.  One of the simplest ways to invoke it is to run the following command from your home directory:
+
You can use the <code>du</code> command line program (short for "d"isk "u"sage) to see how much space individual files or directories are using.  One of the simplest ways to invoke it is to run the following command from your home directory:
  
 
  du -h | sort -h
 
  du -h | sort -h
  
This will show you every directory in your home directory (and every subdirectory of them and every subdirectory of those and so on) and how much space it and all of its contents occupies, with the largest directories shown last.  The directories containing the most data are probably the best candidates to start cleaning out.
+
The above command will show you every directory in your home directory (and every subdirectory of them and every subdirectory of those and so on) and how much space it and all of its contents occupies, with the largest directories shown last.  The directories containing the most data are probably the best candidates to start cleaning out.
  
 
Another useful command is the following:
 
Another useful command is the following:
Line 44: Line 42:
 
This version of running the du command lists all directories ''and all files'' in the current directory and their sizes.  (The ''previous'' version of running the du command shows all directories, not just the ones directly in the current directory, ''and'' it doesn't list files directly; they simply get added to the size tally for the directory they're in.)
 
This version of running the du command lists all directories ''and all files'' in the current directory and their sizes.  (The ''previous'' version of running the du command shows all directories, not just the ones directly in the current directory, ''and'' it doesn't list files directly; they simply get added to the size tally for the directory they're in.)
  
===NOTE: If You Are Using A ''Cloud Drive'' ('''Dropbox''', '''Google Drive''', etc.)==
+
:'''NOTE:''' With either of the two above commands, it might be helpful to pipe them to ''less'' or ''more'', or something similar, should your output be long enough to scroll by.  For  example...
 +
 
 +
:<pre>du -h | sort -h | less</pre>
 +
 
 +
Another use of du can provide some additional detail of what files are taking up the most space:
  
:If you have your CS Ugrad Linux account linked to a cloud drive such as '''Dropbox''' or '''Google Drive''', please note that those services may download local copies of your cloud files into your Ugrad account. This is an extremely quick way to go over your CS Ugrad disk quota.  We recommend you not link cloud drives with your Linux account (unless your cloud drive is very small and well within your CS Ugrad disk quota.)  We cannot accomodate users' cloud data within our provided home directory disk resources.
+
  du -ha |sort -h |tail
  
  
 
=== If You Have .nfs Files ===
 
=== If You Have .nfs Files ===
  
In some cases, you might see a file with a name starting with "<code>.nfs</code>" followed by a long string of numbers. This happens when a file is deleted while a running program is still using it.  Our systems simply rename the file while the program runs.  As soon as the program exits, the <code>.nfs</code> file will be removed.
+
If, from the above list of commands, you run:
 +
 
 +
du -sh * .??* | sort -h
 +
 
 +
and you notice a file with a name starting with "<code>.nfs</code>" followed by a long string of numbers, you will want to remove it, as it can be a very large (and unwanted) file. And it's not a file you can use the <code>rm</code> command with (unlike other with other files.)
 +
 
 +
The .nfs file is created when some other file ''is deleted'' while a running program ''is still using'' that file.  Our systems simply renames that other file (that you deleted) to <code>.nfs</code> (followed by a long string of numbers) while the program is still running.  As soon as the program exits, the <code>.nfs</code> file will be removed.  But, if the .nfs file remains in your directory, as mentioned, you cannot simply remove it with a <code>rm</code> command, as you would do with regular files.  The .nfs file remains because there is some process/program open that is keeping the .nfs file in place.  If you tried to directly remove the .nfs file that some process has open, there would be a replacement .nfs file created.  So, you need to stop the program that is running that is keeping the .nfs file open.
  
 
If you need to figure out what program is using the file, you can use the <code>lsof</code> program, like this:
 
If you need to figure out what program is using the file, you can use the <code>lsof</code> program, like this:
Line 64: Line 72:
 
The most useful information is at the beginning, giving the name and process ID of the running program.
 
The most useful information is at the beginning, giving the name and process ID of the running program.
  
''This will only work on the system running the program.''  If, for example, you started a program on our <code>ugradx</code> system, you would only see useful information from <code>lsof</code> if you also ran it on <code>ugradx</code>.  If you ran it on, say, <code>ugradz</code>, it would still see the <code>.nfs</code> file, but it wouldn't be able to find what program was using it.
+
''This will only work on the system running the program.''  If, for example, you started a program on our <code>ugradx</code> system, you would only see useful information from <code>lsof</code> if you also ran it on <code>ugradx</code>.  If you ran it on, say, <code>ugradz</code>, it would still see the <code>.nfs</code> file (because all the ugrad systems see the same home directory folders), but it wouldn't be able to find what program was using it.
  
 
=== If You Can't Find any Large Files ===
 
=== If You Can't Find any Large Files ===
  
Sometimes the <code>quota</code> program will say that you're using a lot of disk space, but <code>du</code> doesn't show anything large.
+
Sometimes the <code>quota</code> program will say that you're using a lot of disk space, but the <code>du</code> command doesn't show anything large.
  
 
One common cause for this is because of a quirk in Unix filesystem behavior.  If a you delete a file while it's in use by a running program, the file will disappear from the directory, ''but it will still exist, invisibly using disk space'' until the running program exits.  Even worse, if the program writes data to this deleted file, the extra data gets counted against your quota, even though you can't see the file.  (Usually when you delete a file that's in use you'll get a .nfs file, [[#If You Have .nfs Files|as described above]], but sometimes the file will just disappear entirely.)
 
One common cause for this is because of a quirk in Unix filesystem behavior.  If a you delete a file while it's in use by a running program, the file will disappear from the directory, ''but it will still exist, invisibly using disk space'' until the running program exits.  Even worse, if the program writes data to this deleted file, the extra data gets counted against your quota, even though you can't see the file.  (Usually when you delete a file that's in use you'll get a .nfs file, [[#If You Have .nfs Files|as described above]], but sometimes the file will just disappear entirely.)
Line 93: Line 101:
 
If the system insists that you're over quota but you can't figure out why, you can always email us at [mailto:support@cs.jhu.edu support@cs.jhu.edu] for assistance.
 
If the system insists that you're over quota but you can't figure out why, you can always email us at [mailto:support@cs.jhu.edu support@cs.jhu.edu] for assistance.
  
==NOTE:  If You Are Using A ''Cloud Drive'' ('''Dropbox''', '''Google Drive''', etc.)==
+
== A Note on File Size and Disk Usage ==
  
If you have your CS Ugrad Linux account linked to a cloud drive such as '''Dropbox''' or '''Google Drive''', please note that those services may download local copies of your cloud files into your Ugrad accountThis is an extremely quick way to go over your CS Ugrad disk quotaWe recommend you not link cloud drives with your Linux account (unless your cloud drive is very small and well within your CS Ugrad disk quota.)  We cannot accomodate users' cloud data within our provided home directory disk resources.
+
On our systems, files are compressed before they're stored on diskConsequently, the disk space used by a file (what you see if you run <kbd>du -sh <var>file</var></kbd>) is likely to be smaller than the amount of data stored in the file (which is what you see if you run <kbd>ls -lh <var>file</var></kbd>).  Since our quotas are based on the actual amount of disk space used, the sizes shown by <code>du</code> are most relevant hereIf you rely on just the sizes shown by <code>ls</code>, you might be misled about how much your quota is being affected by particular files.
  
 
== If You Need a Larger Quota on the ''CS Ugrad Net''==
 
== If You Need a Larger Quota on the ''CS Ugrad Net''==
Line 109: Line 117:
 
* How long you need the increased quota.  After that, your quota will be reduced to its normal level, and you will need to be below that level to write files to your directory.
 
* How long you need the increased quota.  After that, your quota will be reduced to its normal level, and you will need to be below that level to write files to your directory.
  
''Quota increases are not guaranteed. Allow at least one full business day for your quota increase request to be processed.''
+
''Quota increases are not guaranteed. Allow '''at least''' one full business day for your quota increase request to be processed.''
  
 
[[Category:Accounts]]
 
[[Category:Accounts]]
 
[[Category:Linux Clients]]
 
[[Category:Linux Clients]]
[[Category:Troubleshooting]]
 

Latest revision as of 15:10, 8 November 2024

CS UGRAD Net Users: On the CS Ugrad Net, the Ugrad accounts are subject to disk quotas, which are limits on the total amount of data you are permitted to store in your home directory (or, in some cases, other directories). If you exceed a quota, you will be unable to create new files or add to existing files until you reduce the total amount of disk space you're using to go below your quota.

CS GRAD Net Users: The CS Grad/Research Net does not currently have disk quotas enabled. But, we ask that users on the CS Grad/Research Net be mindful that you are sharing disk space with much of CS. If you need to store large amounts of data on the CS Grad Net, please see: Storing Large Amounts Of Data On Our Systems

How Quotas Work on the CS Ugrad Net

Every account that is subject to a quota has limit on the total amount of disk space used by that account's files. For most of our CS Ugrad accounts, that limit is 10 GiB. Once you hit that limit, all attempts to allocate more space (creating new files, making existing files larger, etc.) will fail.

If you cannot allocate more space (because you've hit your quota limit), you might not be able to log in via the GUI on our Ugrad computer lab Linux systems. If that happens, you will have to SSH in remotely and reduce your disk usage below your quota.

IMPORTANT NOTE: If You Are Using A Cloud Drive ('Dropbox, Google Drive, etc.)

If you have your CS Ugrad Linux account linked to a cloud drive such as Dropbox or Google Drive, please note that those services may download local copies of your cloud files into your Ugrad account. This is an extremely quick way to go over your CS Ugrad disk quota. We recommend you not link cloud drives with your CS Linux account (unless your cloud drive is very small and well within your CS Ugrad disk quota.) We cannot accomodate users' cloud data within our provided home directory disk resources.

How to Check Your Quota on the CS Ugrad Net

Log in to one of the Linux Clients on the CS Undergrad Net and run the following command from the command line:

quota

If it doesn't print anything, you are not subject to any quotas. If it does print something, the output will look like this:

account: 1.6GiB used out of 10.0GiB quota

This should be fairly self-explanatory. The person (with account name "account") has files totaling 1.6 gigabytes of data. The account's usage limit is 10 gigabytes.

How to Get Yourself Under Quota on the CS Ugrad Net

If you're over quota, the only way to get back under is to delete or shrink files until your total disk usage is reduced below your limit.

You can use the du command line program (short for "d"isk "u"sage) to see how much space individual files or directories are using. One of the simplest ways to invoke it is to run the following command from your home directory:

du -h | sort -h

The above command will show you every directory in your home directory (and every subdirectory of them and every subdirectory of those and so on) and how much space it and all of its contents occupies, with the largest directories shown last. The directories containing the most data are probably the best candidates to start cleaning out.

Another useful command is the following:

du -sh * .??* | sort -h

This version of running the du command lists all directories and all files in the current directory and their sizes. (The previous version of running the du command shows all directories, not just the ones directly in the current directory, and it doesn't list files directly; they simply get added to the size tally for the directory they're in.)

NOTE: With either of the two above commands, it might be helpful to pipe them to less or more, or something similar, should your output be long enough to scroll by. For example...
du -h | sort -h | less

Another use of du can provide some additional detail of what files are taking up the most space:

du -ha |sort -h |tail


If You Have .nfs Files

If, from the above list of commands, you run:

du -sh * .??* | sort -h

and you notice a file with a name starting with ".nfs" followed by a long string of numbers, you will want to remove it, as it can be a very large (and unwanted) file. And it's not a file you can use the rm command with (unlike other with other files.)

The .nfs file is created when some other file is deleted while a running program is still using that file. Our systems simply renames that other file (that you deleted) to .nfs (followed by a long string of numbers) while the program is still running. As soon as the program exits, the .nfs file will be removed. But, if the .nfs file remains in your directory, as mentioned, you cannot simply remove it with a rm command, as you would do with regular files. The .nfs file remains because there is some process/program open that is keeping the .nfs file in place. If you tried to directly remove the .nfs file that some process has open, there would be a replacement .nfs file created. So, you need to stop the program that is running that is keeping the .nfs file open.

If you need to figure out what program is using the file, you can use the lsof program, like this:

lsof .nfs000000003114807500000002

(Obviously, use the actual name of the file in your directory.) Its output will look something like this:

COMMAND  PID USER    FD   TYPE DEVICE  SIZE/OFF      NODE NAME
pv      9671 example  1w   REG   0,45 167772160 823427189 .nfs000000003114807500000002

The most useful information is at the beginning, giving the name and process ID of the running program.

This will only work on the system running the program. If, for example, you started a program on our ugradx system, you would only see useful information from lsof if you also ran it on ugradx. If you ran it on, say, ugradz, it would still see the .nfs file (because all the ugrad systems see the same home directory folders), but it wouldn't be able to find what program was using it.

If You Can't Find any Large Files

Sometimes the quota program will say that you're using a lot of disk space, but the du command doesn't show anything large.

One common cause for this is because of a quirk in Unix filesystem behavior. If a you delete a file while it's in use by a running program, the file will disappear from the directory, but it will still exist, invisibly using disk space until the running program exits. Even worse, if the program writes data to this deleted file, the extra data gets counted against your quota, even though you can't see the file. (Usually when you delete a file that's in use you'll get a .nfs file, as described above, but sometimes the file will just disappear entirely.)

You can use the lsof program to find files that have deen deleted but are still in use, like this:

lsof +aL1 /home

If you have any files in this state, it'll print something like this:

COMMAND   PID USER    FD   TYPE DEVICE  SIZE/OFF NLINK      NODE NAME
pv      16081 example  1w   REG   0,45 159907840     0 823427190 /home/example/inuse (barley.cs.jhu.edu:/home)

That will show the name of the program and the location where the file used to be before it was deleted. If you don't have any other way to stop the program, you can usually terminate it with the kill command, like this:

kill 16081

For the number, use the process ID (PID) from lsof's output.

This will only work if you run lsof on the same system where the other program is running. You might need to log into several systems to check things if you've used more than one for your recent work.

If You Still Can't Reduce Your Disk Usage

If the system insists that you're over quota but you can't figure out why, you can always email us at support@cs.jhu.edu for assistance.

A Note on File Size and Disk Usage

On our systems, files are compressed before they're stored on disk. Consequently, the disk space used by a file (what you see if you run du -sh file) is likely to be smaller than the amount of data stored in the file (which is what you see if you run ls -lh file). Since our quotas are based on the actual amount of disk space used, the sizes shown by du are most relevant here. If you rely on just the sizes shown by ls, you might be misled about how much your quota is being affected by particular files.

If You Need a Larger Quota on the CS Ugrad Net

If your class projects or research require more disk space than your quota will allow, please contact support@cs.jhu.edu and detail the reasons your current quota is insufficient. (Note: Please do not request a larger disk quota to accommodate cloud drives such as Dropbox and Google Drive.) Please include the following information when requesting more disk space:

  • Your CS username
  • CS student status (ugrad? grad? concurrent?)
  • Your JHED ID
  • How much additional space you require.
  • Name and section of class you're taking that's requiring the extra space.
  • Name of the professor teaching the class.
  • How long you need the increased quota. After that, your quota will be reduced to its normal level, and you will need to be below that level to write files to your directory.

Quota increases are not guaranteed. Allow at least one full business day for your quota increase request to be processed.