[TUHS] Quotas - did anyone ever use them?

[Pete Wright]

On 5/30/19 6:49 AM, David wrote:
> I think it was BSD 4.1 that added quotas to the disk system, and I was just wondering if anyone ever used them, in academia or industry. As a user and an admin I never used this and, while I thought it was interesting, just figured that the users would sort it out amongst themselves. Which they mostly did.
>
> So, anyone ever use this feature?

Lots of interesting insights/stories on this thread so figured i'd throw 
my hat in the ring and share a business anecdote...

For quite a while i worked in the special effects/animation industry 
where fortunately (for me) unix has a long and interesting history. One 
secret abut the VFX world is that it's tremendously expensive with 
relatively little financial upside.  in my experience it is the studios 
who get most of the residual income from a blockbuster feature.  also a 
crew for a AAA feature requires lots of human power, computers and 
storage.  my shop frequently had 3-5 features in full production mode at 
the same time so we were redlining all of our systems 24/7.

So aside from the cost of maintaining a large renderfarm, unix/linux 3d 
workstations, editing bays etc we also had an enormous NetApp footprint 
to support all these systems.  Now artists love creating lots and lots 
of high resolution images, and if they had their way there were be 
unlimited storage so they'd never have to archive a shot to tape in the 
event they need to reference it later.  But obviously that's not 
reasonable from a financial perspective.

Our solution was to make heavy use of storage quotas in our environment, 
and then leverage quotas to provide per-department billing.  An 
individual user was given something like 1GB by default (here they had 
their mailbox, source-code, scripts etc), then the show they were booked 
on was then given an allocation of say 1TB of storage.  The show would 
then carve out this allocation in a per-shot basis.  This allowed us as 
an organization to actually keep pretty detailed records on our costs 
and unfortunately isn't something I've seen replicated well at lots of 
startups flush with cash these days.

I was briefly on the team responsible for managing these quotas for a 
show and it was seriously an around the clock operation to keep our 
disks from filling up.  One of the tricks was to figure out how much 
space a rendered sequence of images would consume, factor in the 
time-to-render a frame and attempt to line up your backup jobs to free 
up enough space so the render nodes could write out the images to NFS.

-pete

-- 

Pete Wright
pete at nomadlogic.org
@nomadlogicLA