Re: Guest Post: More on Debt Ceilings

1

I think this one is essentially a political question: it's constitutional if you can make it stick. Which is really saying nothing.

The interpretation Epps puts on it sounds fine to me, but God knows what the Supremes might or might not do. Line six, Section 9, Article 1 sayeth that 'No Money shall be drawn from the Treasury, but in Consequence of Appropriations made by Law;" so I am not clear that even with Amendment 14 on, that Obama can spend money, as Epps suggests. At the tightest Trasury should be able to service, er, should be required to service the debt, debt ceiling or no, and they can fight out the constitutionality in court. But I am pretty not clear that that helps much.

I AM sure that the right can come up with plausible arguments that whatever Obama does in the respect he has to do whatever Congress wants, and if that goes to the courts the courts may go along with them. But if so that's purely an exercise of political power.

(Ron Paul has argued that the Federal Reserve cannot, in fact print money. The theory for this is that the Constitutional convention eliminated the words 'and emits bills' from line two, Section 8 Article 1. It's a very silly argument, but back in 1913 I think the bankers heeded this, which is why the Federal Reserve is officially owned by the banks, thus getting around such a limitation.)

max
['Welcome to the non-shooting civil war.']


Posted by: max | Link to this comment | 04-29-11 1:55 PM
horizontal rule
2

I remember fondly the "every regulation is a 'taking' " days of the Gingrich '90s.


Posted by: Flippanter | Link to this comment | 04-29-11 2:00 PM
horizontal rule
3

If I were litigating this, I'd make a statutory argument, that a budget requiring borrowing over the debt limit abrogates prior inconsistent law -- an earlier Congress can't bind a later one.


Posted by: LizardBreath | Link to this comment | 04-29-11 2:00 PM
horizontal rule
4

(I wouldn't call that argument a slamdunk -- I could see a judge going either way on it. But it's good enough to be worth making.)


Posted by: LizardBreath | Link to this comment | 04-29-11 2:07 PM
horizontal rule
5

[A]n earlier Congress can't bind a later one.

I don't mean to challenge this argument, with which I am as familiar as the next shyster, but, further to the "it's a political question" point in 1, I wonder how, if at all, it can be used to persuade the common clay of the new West simple people of the land voters.


Posted by: Flippanter | Link to this comment | 04-29-11 2:07 PM
horizontal rule
6

As is clear from all of the many thousands of words I've written here over the years, I've got no idea of how you convince voters, generally, of anything.

If I were in the same room with an individual I was trying to convince of this, I'd run them through the form of the legal argument: there's a firm principle that if the same legislature passes two inconsistent laws, the later one controls. Then I'd point out that we just had a huge political fight over the budget, that was all about the debt -- we've got an agreement on how much the US is going to tax, and how much money it's going to spend, and that agreement is later than the debt-limit statute, and was arrived at with the full knowledge that it was impossible to tax and spend at the agreed-upon rates while complying with the debt limit statute.

Therefore, Congress abrogated the debt-limit statute by agreeing upon an incompatible budget.

But that's not a 'selling it to voters' persuasive pitch, that's the same argument I'd make to convince anyone I actually knew, which means it'd probably be completely useless.


Posted by: LizardBreath | Link to this comment | 04-29-11 2:15 PM
horizontal rule
7

OT: I presume that we're all too democratic to discuss The Wedding in public, but I'll say, at least, that (i) Prince Philip looks pretty good for 90 and (ii) the Brits really know how to put on the full gold-braided, scarlet-coated, Knight of the Bath military dog.


Posted by: Flippanter | Link to this comment | 04-29-11 2:15 PM
horizontal rule
8

I just got off the phone with my dad, whose Anglophilia is showing. He had the same comment about Philip and the wedding generally. And thinks the new Duchess is just adorable.

I worry that these are early signs of dementia, but he's always had an unfortunate soft spot for the trappings of aristocracy.


Posted by: LizardBreath | Link to this comment | 04-29-11 2:17 PM
horizontal rule
9

Brad DeLong ...links to Bruce Bartlett, which does little more than confirm the OP, and send it a speculative SCOTUS

Crooked Timber has been seriously discussing MMT the last few days. I like MMT, but I know far less than these guys.

I do not like debt (prefer printing), for a lot of reasons, and it is one of the differences I have with Krugman, and a big problem I have with DeLong, who I think is still committed to New Keynesian trickle-down.

But the legality of printing is considerably different than the legality of Geithner giving more money to the vampire squids. IOW, I don't know that you can fund the parks this way. But remember, the US is still 60% funded with ongoing revenues.


Posted by: bob mcmanus | Link to this comment | 04-29-11 2:43 PM
horizontal rule
10

there's a firm principle that if the same legislature passes two inconsistent laws, the later one controls.

How does that square with TVA v. Hill?


Posted by: Minivet | Link to this comment | 04-29-11 2:46 PM
horizontal rule
11

I'd distinguish it on the basis that the incompatibility is fundamental to the nature of the budget and of the debt limit -- it's not an implicit overruling in the TVA v. Hill sense, but a much more direct conflict. Appropriations to build a dam don't abrogate the Endangered Species Act, because the purpose of building a dam isn't killing snail darters. On the other hand, budgeting to spend money that you don't have unless you borrow it, is much more explicitly a commitment to borrow that money.

But as I said in 4, it's not a slamdunk -- a court could easily refuse to distinguish TVA v. Hill.


Posted by: LizardBreath | Link to this comment | 04-29-11 2:55 PM
horizontal rule
12

But the lawyers can plead and and play. I have two three facts.

1) The Bush years should have taught us that there is no law Obama cannot break while finding plausible arguments to defend his actions.

2) SCOTUS and Congress have no armies.

3) There is a lot of ruin endurance in a nation.


Posted by: bob mcmanus | Link to this comment | 04-29-11 2:59 PM
horizontal rule
13

ll

May I hijack this thread? I am obsessed right now with questions about SANs. The project I mentioned here is getting underway (and I even managed to get a photographer at a better rate), but now the institution is bitching about the increased volume of data storage we'll be consuming. I've been asked to assume the costs of purchasing said storage, which didn't seem to me like a big deal, until I learned that they were proposing to buy something like this. Not that it's a bad product, but we already have a SAN, and I can't imagine we've reached capacity for it. I don't know the exact model, but it was purchased probably about 18 months ago, is also a HP Lefthand, and was bought with an initial capacity of 5.4 TB. Right now the amount of visible space available on it (used and unused) seems to be 9 TB. I'm a child about these things but I think (from what I've heard from our IT guy) that they are doing either RAID level 5 or 6 on it, so I'm not sure how that affects the total amount of available storage there.

So: given this sketchy information, is it possible that my rather modest needs (pushing through the currently established 9 TB level of total network space up to a level of maybe 14 TB) would cause us to actually need an entirely new SAN? Or are they trying to snow me because they want a bigger SAN for other reasons? Given the people involved, I highly suspect the latter. Any information would be appreciated. Thanks.

l>


Posted by: JL | Link to this comment | 04-29-11 4:39 PM
horizontal rule
14

The space you can see already factors in the Raid level in use. If it says you have 1tb of free space, that's what you have, even if it's actually two physical 1tb drives running mirrored.

I don't know anything about the kit you are using but it might be the case that you can't physically add, say, the 7 or 8 pjysical drives you might need to get your desired 6 or 7 TB increase. It's also possible, and i know nothing about your setup here, that some of any potential storage expansion may already be earmarked for use. Also you always round the numbers up becausr everyone gets it wrong or lies. "oh we'll just need a couple of TB for a month or two" often means "we'll need 12, indefinitely, and we've no money to pay for any labour or proper management. In fact, we'll expect you to absorb that cost."

Somewhat Beck's style now but can prob discuss this tomorrow if you like.

Fwiw, i manage the backend on projects like this and academic and quasi-academic staff are always trying to bilk us on storage costs. Properly managed storage is non-cheap.


Posted by: nattarGcM ttaM | Link to this comment | 04-29-11 5:10 PM
horizontal rule
15

Thanks, ttaM. That's helpful. If you have any additional thoughts in the morning, I'd like to hear them.

Obviously it's hard to know what the possibilities are without knowing the exact configuration we have going. I hope to get that information next week. I actually basically trust our IT guy, but he really is an innocent bystander in this. It's his boss who is insisting (without talking to IT) that we need a new SAN for this project. And since said boss has the ethics of a used car salesman, and I know he has plans that require increasing our storage potential, I'm very suspicious. But if it is what it is, I'll pay it--I'll grouse a bit about the fact that no other project in the institution is being asked to carry this weight, and I'll look for a discount based upon the fact that I'll be buying far more storage than I'll actually use (and which will be eagerly hoovered up by others), but I am committed to carrying my own costs in doing this project.

Fwiw myself, I'm somewhere between front- and backend in this case. Responsible for some of the systems, but not the actual hardware. And definitely more on the service side of things.


Posted by: JL | Link to this comment | 04-29-11 5:35 PM
horizontal rule
16

My role is similar. I manage the software inrastructure for projects I work on and only have input rather than final say on hardware. However, for reasons of internal politics i have, at times, had to do end runs around some central IT teams and have both bought and managed storage in the past. Largely, tbh, for medium term rather than long term use. Long term storage is largely out of the reach of any individuak project team or manager. I do work specifically on academic computing projects based on the production, storage and delivery of images, though, so I might have some experience that's relevant to what you are doing.


Posted by: nattarGcM ttaM | Link to this comment | 04-29-11 5:45 PM
horizontal rule
17

Friday night, so I have to go with mom to Bingo at the church. I'll see you guys (the best!) later tonight, okay? Don't do anything I wouldn't do, homeys!


Posted by: Pauly Shore | Link to this comment | 04-29-11 5:54 PM
horizontal rule
18

I do work specifically on academic computing projects based on the production, storage and delivery of images, though, so I might have some experience that's relevant to what you are doing.

You definitely do. One part of my office (the part involved in this project) is charged with doing the same, in this case for a mid-sized art museum. Where we may differ is in the utter dysfunction of my institution's management structure. Get me out of here.


Posted by: JL | Link to this comment | 04-29-11 6:11 PM
horizontal rule
19

I bought two 4TB server drives and fucked it big time. To be fair to me, there was absolutely no reason to think I would know what I was doing. Suddenly, we needed 8TB of storage and I was the only one who was not obviously incapable. I was subtly incapable, so I made them find somebody who knew what they were doing to buy an 8TB server that we are now supposed to back-up on to the two 4TB drives that don't work because I'm stupid or they are junk. At least opinions differ on the last point.


Posted by: Moby Hick | Link to this comment | 04-29-11 6:17 PM
horizontal rule
20

Properly managed storage is non-cheap

Yeah I think we figured our SGI fiber channel storage is running us about $6,000/TB right about now if you calculate in all the peripheral costs.


Posted by: CJB | Link to this comment | 04-29-11 6:32 PM
horizontal rule
21

No wonder ours isn't working.


Posted by: Moby Hick | Link to this comment | 04-29-11 6:42 PM
horizontal rule
22

re: 20

Yeah, we have very similar figures used for internal costing. Rounded up from there a bit to cover contingencies. Those numbers are worked out by people other than me; I'm the middle-man most of the time. However, people running academic projects sometimes either fail to budget for storage at all, or assume it costs what it costs them at PC World.

"But I can buy a 1TB drive for £60."

"Dude, you want to host 20TB of data, forever. It isn't going to cost you 500 quid."


Posted by: nattarGcM ttaM | Link to this comment | 04-29-11 6:52 PM
horizontal rule
23

Properly managed storage is non-cheap.

But it's a bargain compared to improperly managed storage.


Posted by: JL | Link to this comment | 04-30-11 2:49 AM
horizontal rule
24

To complain some more, here are the details on SANs in the same line as ours. The model we have isn't listed, as I don't think it's made anymore, but it was purchased about 18 months ago with a starting capacity of 5.4 TB. I know storage capacity grows all the time, but when I see that a currently available SAN starting at 7.2 TB can be expanded to over 100 TB, I have trouble accepting that our slightly older model can't rise above a total of 9 TB of usable space to, say, 15 TB. Maybe I'm wrong, but I'd like to know the details. Not that I'll get anywhere without knowing exactly what we have. Instead I just get to torment myself and bore you all weekend about it.


Posted by: JL | Link to this comment | 04-30-11 3:18 AM
horizontal rule
25

JL, I certainly cannot vouch for the motives of the various actors, but the model you link in 13 is described, The StorageWorks P4300 G2 7.2TB SAS Starter SAN Solution adds storage capacity to the HP StorageWorks P4300 G2 Series Storage Systems. and if you go to the systems' description linked in 24 (various P4000 series systems) you see, Use storage clustering capability and add incremental P4000 G2 Storage Systems into the SAN to expand storage capacity. So I'd guess that they may be looking to incrementally beef up the one they have with the new "SAN". Although the descriptions are a bit deceptive to me in that I believe it can also be used stand-alone (you might need to buy software in that case). But if you do grow those base systems you do it by adding the kind of thing you link in 13.

So it may be a "legitimate" (you mention 9 TB "used and unused" on the current system) recommendation for expansion in that your needs put it over the top (by their calculations--as ttaM says everyone sandbags, often for good reason). However, you definitely should have the, "Is this adding to the existing system or standing on its own and why does my incremental storage need request have to buy the whole thing in either case" discussion. And then you can point out that for the exact same price they can get the 16 TB P4300 MDL SAS expansion rather than just the 9 TB (disks are lower performing ... and possibly fail more, but unless the stuff is accessed a lot or needs to be written to disk very quickly, probably adequate for your needs). Then they tell you to shut up. And then you mumble something about imaginary friends on the Internet. And then they say, "What are you mumbling about?", and then you say, "Like you'd understand anyway", and then you win.


Posted by: JP Stormcrow | Link to this comment | 04-30-11 7:08 AM
horizontal rule
26

Also I hate dealing with fucking storage (actually it's not the storage itself, it's the backup, OR and DR). Yesterday I had the pleasure of running a half-day "design input session" with about 20 folks for a large worldwide system whose primary use is storage. Turns out everybody wanted a lot more for a lot less money. Who'd a thunk it? You'll get nothing a certain amount and like it. (Actually, building in easy extensibility is a key goal. Help me Amazon (or the like), you're my only hope. But then again, nice outage, dudes.)


Posted by: JP Stormcrow | Link to this comment | 04-30-11 7:11 AM
horizontal rule
27

Just doing a quick read over the specs for the P4000 systems it looks like the way you expand storage is by adding whole new nodes not just adding disk trays which is why you would need to purchase another P4300.


Posted by: CJB | Link to this comment | 04-30-11 7:13 AM
horizontal rule
28

Pwned by JP who also gave more info. Need to preview before I post.

I do agree that there should be someway to spread the cost of the storage out between users. It still isn't going to be cheap, but I don't know why you would have to pay the whole increment.


Posted by: CJB | Link to this comment | 04-30-11 7:18 AM
horizontal rule
29

28: But snappy and succinct is good. Less characters=less storage=more freedom for everybody!


Posted by: JP Stormcrow | Link to this comment | 04-30-11 7:22 AM
horizontal rule
30

Next time I get asked for 8TB, I'm going to buy five and a half million 3 1/2 inch disks and demand a huge closet.


Posted by: Moby Hick | Link to this comment | 04-30-11 8:44 AM
horizontal rule
31

Or 8 cars worth of floppies.


Posted by: JP Stormcrow | Link to this comment | 04-30-11 8:51 AM
horizontal rule
32

Thanks for the comments, everyone. Obviously I'm in a bit over my head. I can say that, while the various actors involved are definitely not to be trusted, I can anticipate that my project's needs for storage will be larger than any other single department or project within the institution. It's a museum, we're the collection, and we're talking about a lot of high-res images. But I also have strong reason to suspect that other departments have been promised a free ride on storage that my project pays for. And while we are the single biggest user, we're still only a fraction of the whole institution's storage needs--and it looks like what they're planning will be the new platform for everyone.

So yeah, I'm willing to pay some, even a large chunk of the costs. But it is more than a little infuriating that our head of operations is trying to evade paying for what will be an overall institutional upgrade (with me as admittedly the biggest beneficiary) while he's spending his time using the regular IT capital budget to buy magic beans.


Posted by: JL | Link to this comment | 04-30-11 9:02 AM
horizontal rule
33

Looking over the numbers, I probably only need about 3 TB in the next couple of years, and don't need to acquire anything until some time next year. On the other hand, I think I may have misread the total space on our network--it looks like it might be less than 9 TB total. Whatever. To take a different tack, since I don't have confidence in the people planning, let me ask all of you: if you were looking to get good reliable digital storage appropriate for an institution to host long-term assets on, something that would offer say 3-4 usable space TB to start and could expand, what would you look at and what would you expect to pay? Keeping in mind that the purchasing institution is also a struggling non-profit.


Posted by: JL | Link to this comment | 04-30-11 9:48 AM
horizontal rule
34

3TB isn't a lot of images, btw. What sort of image resolution [and camera equipment] are you working with?


Posted by: nattarGcM ttaM | Link to this comment | 04-30-11 9:55 AM
horizontal rule
35

I'm not really the one to ask, but we use some sort of Canon, recently purchased. We save DNG, TIFF, and JPEG files. The TIFFs are 300 dpi, with the intent of being printed at 8" x 10", if I remember correctly. But I could be a bit off on that. Of course one can go larger, but images at this scale (if I've represented what we do accurately) suit almost all of our needs. We can get an overall image suitable for a catalog, and we very rarely need detail shots. There are some instances where we know we will, and we'll take separate details, and for some larger paintings and similar items, we'll take multiple shots to get a variety of details. But that's fairly rare.


Posted by: JL | Link to this comment | 04-30-11 11:57 AM
horizontal rule
36

if you were looking to get good reliable digital storage appropriate for an institution to host long-term assets on, something that would offer say 3-4 usable space TB to start and could expand, what would you look at and what would you expect to pay?

There are a lot of variables that would affect the cost. I think I could probably design systems that varied from 5000 to 200000 dollars up front that might meet those requirements. A lot depends on how much you value the data and how available you need it to be. Keep in mind there are also ongoing costs to systems like these as well.


Posted by: CJB | Link to this comment | 04-30-11 2:11 PM
horizontal rule
37

33

This may be a dumb question, but have you considered using something like this ?


Posted by: James B. Shearer | Link to this comment | 04-30-11 2:15 PM
horizontal rule
38

This may be a dumb question, but have you considered using something like this ?

I think the data transfer costs would be problematic for image data assuming they are doing anything with images other than just storing them.


Posted by: CJB | Link to this comment | 04-30-11 2:40 PM
horizontal rule
39

There are a lot of variables that would affect the cost.

Of course. We value the data very highly. Aside from all the usual reasons an organization might value its data of all kinds, the material I'm specifically producing consists of images of our collection. These represent a large amount of work that would be difficult and risky to reproduce if we needed to do so. Losing the images would also be a blow to our ability to document the history of the objects and their condition over time. This is a museum; we're in it for the long haul, and that applies to our data as well.

But obviously, being a museum, and a neglected mid-sized one at that, we have limited resources. If it were all coming out of my budget, I'd really prefer not to spend more than $20,000, soup to nuts. $15,000 would be even better.

This may be a dumb question, but have you considered using something like this ?

I've looked a little bit at it, but to be honest, I'm new to the business of evaluating storage. I had thought that, given that we purchased a SAN last year, we were going to be set for the near future at the very least. It was only in the past week that I got a sense of how much of a bite they were hoping to take from my budget and I started taking a greater interest in the topic. One problem is that I don't really have (as ttaM said was true of him above as well) a final say on what we do. That's IT's area, and they have a preferred way of working (and possibly a superior with an agenda.) If I can document that a solution is better or as good and cheaper, I can probably swing things my way, but we're unlikely to go with anything very innovative or different from current practice.


Posted by: JL | Link to this comment | 04-30-11 2:48 PM
horizontal rule
40

Most of the data, to tell the truth, just sits there. We work with the DNG files when they are created to check them and create the TIFFs, but after that they are simply archival masters--we really hope we never have to access them again. The TIFFs typically only get used for publication purposes. For most of them that means never, though there is a core group of several hundred that see repeated use, and a smaller group that gets regular use (by which I mean several times a year.) The JPEGs are our everyday access files and get used a lot, but they're not a storage concern in any significant sense.


Posted by: JL | Link to this comment | 04-30-11 2:52 PM
horizontal rule
41

If I can document that a solution is better or as good and cheaper, I can probably swing things my way, but we're unlikely to go with anything very innovative or different from current practice.

My guess is that you won't be able to find anything better or cheaper. There is already a decent sunk cost when you choose a system like this in the form of things like maintenance contracts and software licensing costs. What they are trying to get you to buy is an expansion to an already in place system not an entirely new system so some of those costs are going to be less than if you acquired an entirely new system.

Obviously I don't know all the details, but I think your best bet is going to try and come to some arrangement where you only eat some of the cost and not all of it.


Posted by: CJB | Link to this comment | 04-30-11 3:09 PM
horizontal rule
42

40

Most of the data, to tell the truth, just sits there. We work with the DNG files when they are created to check them and create the TIFFs, but after that they are simply archival masters--we really hope we never have to access them again. The TIFFs typically only get used for publication purposes. ...

There are various TIFF compression schemes that may be useful in this situation.


Posted by: James B. Shearer | Link to this comment | 04-30-11 3:17 PM
horizontal rule
43

Thanks, CJB. I think you are right. And I would appreciate their investment more if I didn't hear yesterday that the person to whom IT reports has been developing a plan for SAN-to-SAN replication as part of a larger initiative. From what I hear, his whole scheme gets very pricey, and what better way to offload some of the costs than to insist that our relatively newly purchased SAN is not able to expand to cover storage needs that were known at the time it was acquired, and so someone else's budget needs to carry the load of a new one? But yeah, if I can manage to only eat some of the costs, I'll be . . . not happy, but ok with it.


Posted by: JL | Link to this comment | 04-30-11 3:19 PM
horizontal rule
44

There are various TIFF compression schemes that may be useful in this situation.

Thanks, I'll look into it. But we tend to get very fussy about this stuff, given the nature of the material, best practice standards for the field, etc. But yeah, if I could say that my storage needs are less than what was anticipated, it would take the pressure off me to shoulder the costs.


Posted by: JL | Link to this comment | 04-30-11 3:21 PM
horizontal rule
45

insist that our relatively newly purchased SAN is not able to expand to cover storage needs that were known at the time it was acquired, and so someone else's budget needs to carry the load of a new one?

Now we get into semantics. I would say that the system that they purchased is in fact expandable to meet the known needs. The way you expand it is by adding more storage nodes. The SAN is the whole system not the individual pieces.


Posted by: CJB | Link to this comment | 04-30-11 3:51 PM
horizontal rule
46

Now we get into semantics

And where my head starts to swim a little. Part of the problem is I have so little details as yet as to what they are planning (I'm working on getting the information.) But please correct me if I'm wrong: part of the appeal of this kind of set-up is that, once you've made the initial outlay to put it in place, you can expand it at comparatively lower cost, right? Not infinitely, but at least for some amount of storage.

I'm especially curious because I see that I made a stupid error when stating the amount of storage we currently have active, whether used or unused: it's not 9 TB, it's 3 TB. I had arrived at the figure of 9 TB by looking at the three shared drives that are running on the SAN that each indicated 3 TB and adding it up. But of course a closer look revealed that the amount marked as used and unused was the same for all three. Which says to IT illiterate me that it's because that figure is for the whole SAN (or rather, the portion that we can use.) And about 3 TB seems to be the right amount for a 5.4 TB SAN with RAID 6 assuming a 25% holdback after RAID, which is what I believe we have. So it looks like we've not expanded the system beyond its original footprint as of yet--and I thought being able to do so at a lower cost was part of the point of having such a system.

I must acknowledge at this point what is obvious: I don't know much about this stuff and am very paranoid about the people making decisions here. But nevertheless, it seems odd.


Posted by: JL | Link to this comment | 04-30-11 4:17 PM
horizontal rule
47

JL, we currently use colour-corrected uncompressed TIFFS as our archival format, RAW files are thrown away. However, some institutions have moved to jpeg2000 [lossless] and some are even moving to jpeg2000 [lossy]. The latter is often supported by what seems to me to be spurious reasoning but it is a route some institutions have gone down. There was a big jpg2000 conference last year with lots of institutions presenting papers, the work is online [I can email you details if you want].

300dpi at 8x10 is smaller than the file sizes we use as we are working with medium-format backs rather than standard dSLRs [and have worked with large format scanning backs and other more exotic devices in the past]. I have some images I've worked with [composited multiple scans of A3 glass plates, for example, and scans of maps] that are huge, but generally 100MB a shot is a rough rule of thumb for our content. Yours'll be less than half that, I expect? Although you are storing derivatives and masters.

I don't know what current standards are for best practice for museum objects. With books/manuscripts it's generally 600dpi at 100% of the size of the original although that isn't always achievable, and in some cases more is desirable.


Posted by: nattarGcM ttaM | Link to this comment | 04-30-11 5:11 PM
horizontal rule
48

I think the huge advantage of Amazon S3 is that it gives you offsite backup capabilities managed by people who know what they are doing. There are so many ways to screw up backup and disaster recovery on your own hardware; better to outsource it to an infrastructure with redundant backups baked into the system.


Posted by: Spike | Link to this comment | 04-30-11 6:13 PM
horizontal rule
49

47: We used to keep color-corrected uncompressed TIFFs as the archival format, but became convinced it made sense to keep the raw file as a DNG as well. There are a lot of different sources on best practices for museums, but a more-or-less consensus opinion can be found be looking at the pdfs here. At any rate, by keeping both the TIFFs and the DNGs, we can't go too far wrong. Unless storage is an issue, of course . . .

It's very true that the standards for books and manuscripts is more exacting than for fine art photography, for a variety of reasons. Just up the street from us is a major research library doing digitization projects that make us look like primitives. Last time I stopped by, they told me they were getting requests (and accommodating them) for 900 dpi.

JPEG2000 makes me laugh just on general principle, but feel free to email the info. I think our TIFFs average around 60MB, but the new camera is doing something more on the order of 80MB, if I remember correctly.


Posted by: JL | Link to this comment | 04-30-11 7:28 PM
horizontal rule
50

re: 49

I don't think any of the Canons can produce much more than about 1/3 of the output of the higher-end PhaseOne and Leaf backs. Although raw pixels don't translate directly into filesizes, I suppose.

We decided not to keep the raw files as we had, at one time, at least 5 different raw formats in use and once you get into storing proprietary formats you also need to store copies of the software to read them as there's no way to be sure that when someone comes along in 2025, say, that their copy of 'Photoshop 21' will be able to read the raw. In fact, I'm pretty sure we were producing files only 10 years ago that you couldn't read with any standard current software. Dicomed went bust in the late 90s, for example, and I'm not sure [never checked] how much software support there is for the original files.

My colleagues here are working on this, so they are building archives of software and hardware along with their dark archive. They find themselves trying to find ways to resurrect, say, some politician's personal letters as stored on 8" Wang floppies.

They blog about it here:
http://anonym.to/?http://futurearchives.blogspot.com/

We don't want/need to bother with that.

Curious why jpeg2000 makes you laugh?


Posted by: nattarGcM ttaM | Link to this comment | 05- 1-11 2:02 AM
horizontal rule
51

I don't think any of the Canons can produce much more than about 1/3 of the output of the higher-end PhaseOne and Leaf backs.

I think you're right. I just checked a TIFF from our latest camera that I had with me and it clocks in at just under 40MB, less than I thought. I think I was under the impression that our previous camera did files that size or a bit larger and extrapolated out from what I had been told was the percentage increase we were doing now.

The beauty of DNG is that it isn't tied to a particular camera or associated proprietary software. So you have the raw file with less of the concerns about being tethered to a particular platform. Not that there are no concerns, but having DNGs along with TIFFs means you have both sides covered: a raw file, with all the information it has, along with the TIFF, for actual use and as a hedge against DNG not making it as a file format. But we decided that DNG's future was sufficiently promising to make it worthwhile to have it along with the TIFFs.

Your colleagues work looks very cool. One disaster we have, currently not being addessed at all, involves our conservation records from the 1980's and '90's. Piles of floppies, getting older every day.

Jpeg2000 makes me laugh because of the name--anything with "2000" in it is inherently funny. Especially since it seems to be one of those initiatives that was introduced with a lot of promise and fanfare but never really taken off, at least in the museum context. I've heard far many people talk about it than actually say that they use it.


Posted by: JL | Link to this comment | 05- 1-11 4:47 AM
horizontal rule
52

Yeah, however most/all of our camera gear doesn't produce DNG files, they produce raw 'TIFF' file* or .mos files. So in our case there's no incentive to keep the raw files. Converting legacy formats is always a PITA -- digital or analogue. We've been doing a load of conversion of nitrate negs recently [because they have the additional problem of blowing up] and while 99.99% of the content of those will never be accessed again there's no practical way of making this assessment. It's a case of digitizing the lot and then freezing the negs in deep storage.

Ditto glass plate negs, although those are actually really quite stable and robust. I expect they'll outlive a lot of successor formats.

Our 'born digital' image archive [the stuff I work with] only dates back to the late 90s, so I don't have the same conversion issues as my colleagues linked above.

* a format that includes a thumbnail version of the standard .tiff file information AND raw information but which isn't actually a totally standard tiff [not the cleverest choice of file extension, imho].


Posted by: nattarGcM ttaM | Link to this comment | 05- 1-11 6:00 AM
horizontal rule
53

re: jpeg2000s, yeah, however there's been an upsurge in its use recently and it's getting used quite a bit in the presentation of digital collections. Slice and dice the image, have a bunch of layers of resolution and bob's yer uncle. I think there's also been a move to use it more as image creation methods speed up.

For example, someone using a scanning back, say, was producing a fraction of the output of a one-shot camera, and that in turn is a fraction of the output of kit doing bulk digitization of microfilm and printed copy, so storage starts to be a real issue. We've not moved to it, yet, but I've been doing preparatory investigation, partly for those reasons.


Posted by: nattarGcM ttaM | Link to this comment | 05- 1-11 6:03 AM
horizontal rule
54

I may have to give jpeg2000 another look. On the one hand, I'm reluctant to move of the standard we're on, or add too much to it (it's in accordance with widely accepted practices for museum, though god knows others execute it beter), but if there is real value, it might be worth it.

We're in a similar boat as your colleagues regarding conversion of historic photography, though we've hardly begun to address the issue. Down the hall from my office is an archive of over 100 years of collection photographs, prints and negatives, from glass plates through every other process you can think of. As daunting as it is, it also helps reassure me. It's a reminder that photographing the collection is a bit like painting a bridge--by the time you finish, it's time to start doing it again.


Posted by: JL | Link to this comment | 05- 1-11 6:18 AM
horizontal rule
55

A big factor is whether it's primarily for archival purposes, or whether you need a large amount of fast concurrent access. If it's the former, something like a Backblaze Pod may be cheap and appropriate.

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

http://www.cloudstoragesystems.com/


Posted by: bbass | Link to this comment | 05- 1-11 8:41 AM
horizontal rule
56

re: 55

For a while I was using a little 4-port eSATA backplane + drives + PSU box that we bought for a few hundred quid, populated with decent off-the-shelf drives. Not as archival storage, just as working storage running off the back of one of our image-processing servers. That's since proved problematic and we are looking again at solutions not that dissimilar to the Backblaze -- that's a very useful link, thanks!


Posted by: nattarGcM ttaM | Link to this comment | 05- 1-11 9:24 AM
horizontal rule
57

The Backblaze idea is very cool, thanks. I can't imagine I could get it approved at our institution, unfortunately. IT wouldn't want to do it and everyone else would consider it too frightening, I'm afraid. We'll see how it goes.


Posted by: JL | Link to this comment | 05- 1-11 10:24 AM
horizontal rule
58

OK, I finally clicked on the second link in 55, and holy crap. That is so cool. I still doubt I could get anyone at my place to bite, but what an awesome thing.


Posted by: JL | Link to this comment | 05- 1-11 6:35 PM
horizontal rule
59

It's very true that the standards for books and manuscripts is more exacting than for fine art photography, for a variety of reasons.

As someone whose only exposure to any of this is reading books and having occasionally looked at fine art, I find this fascinating. I mean, I sort of get, intellectually, why you'd want to see every discoloration on the goat-skin manuscript, or what-have-you. It's just ... a different approach to books than, say, an illegal, crappy (27mb/450 pages), but entirely readable scan from library.ru.


Posted by: x.trapnel | Link to this comment | 05- 1-11 7:20 PM
horizontal rule
60

Capezio dance gucci handbags sale are a common choice for ballet and jazz styles of dance. Savvy marketers launch events for creating environmental awareness and distribute the promotional tote gucci outlet store to the participants of the event. The reason for the price difference in suits is by the fabric gucci sale handbags used to make. While the airline should make one available to you upon request, sometimes http://www.authenticguccioutletonlinestorex.info/#87294 don't, or can't, or perhaps you don't want to call attention to http://www.authenticguccioutletonlinestorex.info/#73795 in the first place.


Posted by: joochyr | Link to this comment | 09- 1-12 6:51 AM
horizontal rule