Disk Quotas

(Redirected from Login Problems Due To Disk Quotas)

INTRO


CS GRAD Net Users: The CS Grad/Research Net does not currently have disk quotas enabled. But, we ask that users on the CS Grad/Research Net be mindful that you are sharing disk space with much of CS. If you need to store large amounts of data on the CS Grad Net, please see: Storing Large Amounts Of Data On Our Systems


CS UGRAD Net Users: On the CS Ugrad Net, the Ugrad accounts are subject to disk quotas, which are limits on the total amount of data you are permitted to store in your home directory (or, in some cases, other directories). If you exceed a quota, you will be unable to create new files or add to existing files until you reduce the total amount of disk space you're using to go below your quota.

How Quotas Work on the CS Ugrad Net

Every account that is subject to a quota has limit on the total amount of disk space used by that account's files. For most of our CS Ugrad accounts, that limit is 10 GiB. Once you hit that limit, all attempts to allocate more space (creating new files, making existing files larger, etc.) will fail.

If you cannot allocate more space (because you've hit your quota limit), you might not be able to log in via the GUI on our Ugrad computer lab Linux systems. If that happens, you will have to SSH in remotely and reduce your disk usage below your quota.

IMPORTANT NOTE: If You Are Using A Cloud Drive ('Dropbox, Google Drive, etc.)

If you have your CS Ugrad Linux account linked to a cloud drive such as Dropbox or Google Drive, please note that those services may download local copies of your cloud files into your Ugrad account. This is an extremely quick way to go over your CS Ugrad disk quota. We recommend you not link cloud drives with your CS Linux account (unless your cloud drive is very small and well within your CS Ugrad disk quota.) We cannot accomodate users' cloud data within our provided home directory disk resources.

How to Check Your Quota on the CS Ugrad Net

Log in to one of the Linux Clients on the CS Undergrad Net and run the following command from the command line: 

quota

If it doesn't print anything, you are not subject to any quotas. If it does print something, the output will look like this: 

account: 1.6GiB used out of 10.0GiB quota

This should be fairly self-explanatory. The person (with account name "account") has files totaling 1.6 gigabytes of data. The account's usage limit is 10 gigabytes.

How to Get Yourself Under Quota on the CS Ugrad Net

If you're over quota, the only way to get back under is to delete or shrink files until your total disk usage is reduced below your limit.

You can use the du command line program to see how much space individual files or directories are using. One of the simplest ways to invoke it is to run the following command from your home directory: 

du -h | sort -h

The above command will show you every directory in your home directory (and every subdirectory of them and every subdirectory of those and so on) and how much space it and all of its contents occupies, with the largest directories shown last. The directories containing the most data are probably the best candidates to start cleaning out.

Another useful command is the following: 

du -sh * .??* | sort -h

This version of running the du command lists all directories and all files in the current directory and their sizes. (The previous version of running the du command shows all directories, not just the ones directly in the current directory, and it doesn't list files directly; they simply get added to the size tally for the directory they're in.)

NOTE: With either of the two above commands, it might be helpful to pipe them to less or more, or something similar, should your output be long enough to scroll by. For example...
du -h | sort -h | less


If You Have .nfs Files

If, from the above list of commands, you run: 

du -sh * .??* | sort -h

and you notice a file with a name starting with ".nfs" followed by a long string of numbers, you will want to remove it, as it can be a very large (and unwanted) file. And it's not a file you can use the rm command with (unlike other with other files.)

The .nfs file is created when some other file is deleted while a running program is still using that file. Our systems simply renames that other file (that you deleted) to .nfs (followed by a long string of numbers) while the program is still running. As soon as the program exits, the .nfs file will be removed. But, if the .nfs file remains in your directory, as mentioned, you cannot simply remove it with a rm command, as you would do with regular files. The .nfs file remains because there is some process/program open that is keeping the .nfs file in place. If you tried to directly remove the .nfs file that some process has open, there would be a replacement .nfs file created. So, you need to stop the program that is running that is keeping the .nfs file open.

If you need to figure out what program is using the file, you can use the lsof program, like this: 

lsof .nfs000000003114807500000002

(Obviously, use the actual name of the file in your directory.) Its output will look something like this: 

COMMAND  PID USER    FD   TYPE DEVICE  SIZE/OFF      NODE NAME
pv      9671 example  1w   REG   0,45 167772160 823427189 .nfs000000003114807500000002

The most useful information is at the beginning, giving the name and process ID of the running program.

This will only work on the system running the program. If, for example, you started a program on our ugradx system, you would only see useful information from lsof if you also ran it on ugradx. If you ran it on, say, ugradz, it would still see the .nfs file (because all the ugrad systems see the same home directory folders), but it wouldn't be able to find what program was using it.

If You Can't Find any Large Files

Sometimes the quota program will say that you're using a lot of disk space, but the du command doesn't show anything large.

One common cause for this is because of a quirk in Unix filesystem behavior. If a you delete a file while it's in use by a running program, the file will disappear from the directory, but it will still exist, invisibly using disk space until the running program exits. Even worse, if the program writes data to this deleted file, the extra data gets counted against your quota, even though you can't see the file. (Usually when you delete a file that's in use you'll get a .nfs file, as described above, but sometimes the file will just disappear entirely.)

You can use the lsof program to find files that have deen deleted but are still in use, like this: 

lsof +aL1 /home

If you have any files in this state, it'll print something like this: 

COMMAND   PID USER    FD   TYPE DEVICE  SIZE/OFF NLINK      NODE NAME
pv      16081 example  1w   REG   0,45 159907840     0 823427190 /home/example/inuse (barley.cs.jhu.edu:/home)

That will show the name of the program and the location where the file used to be before it was deleted. If you don't have any other way to stop the program, you can usually terminate it with the kill command, like this: 

kill 16081

For the number, use the process ID (PID) from lsof's output.

This will only work if you run lsof on the same system where the other program is running. You might need to log into several systems to check things if you've used more than one for your recent work.

If You Still Can't Reduce Your Disk Usage

If the system insists that you're over quota but you can't figure out why, you can always email us at support@cs.jhu.edu for assistance.

If You Need a Larger Quota on the CS Ugrad Net

If your class projects or research require more disk space than your quota will allow, please contact support@cs.jhu.edu and detail the reasons your current quota is insufficient. (Note: Please do not request a larger disk quota to accommodate cloud drives such as Dropbox and Google Drive.) Please include the following information when requesting more disk space:

  • Your CS username
  • CS student status (ugrad? grad? concurrent?)
  • Your JHED ID
  • How much additional space you require.
  • Name and section of class you're taking that's requiring the extra space.
  • Name of the professor teaching the class.
  • How long you need the increased quota. After that, your quota will be reduced to its normal level, and you will need to be below that level to write files to your directory.

Quota increases are not guaranteed. Allow at least one full business day for your quota increase request to be processed.