Recently I am developing a feature where I need to protect a file from all users including the root. Most of the kernels do this by restricting the permissions stuff and/or attaching some special flags with the files notifying that this file should not be touched. While shaving this morning, I had a weird idea.
I want to protect a file, so restricting the permissions to root is an obvious thing to do here. But still root can modify this file. If I want to block root user as well, what if I had an another internal root like user? This would also be a root user, say a root cousin which is invisible to outside world. So in this case, root can read the file but can not modify it. I know this breaks the standard UNIX legacy we have that there is only one powerful god and that is root. What if we shatter this? There would be more than one god, so would there be a clash of titans? No, the rule is one god does not interfere with other god. The gods are read-only while interfering with another god. In this case, it gives an impression that root user is not able to modify certain file. Since internally this file is owned by root cousin, root is not able to change/modify it. So the change of behavior is quite noticeable. I don't know if this would be acceptable.
Since this root cousin is invisible to end user, he/she can not inherit root cousin privileges. And a root user can not change/modify the files that need to be protected.
This is a very simple idea and might have occurred to a lot of people. The moment I struck it, I felt like noting it down somewhere and thats why I am posting it here. I still need to figure out how to implement this. Will update about it soon.
Few basics on how to implement this. This root cousin needs a dedicated uid and gid at least for UNIX. This uid and gid can not be used by end user. Kernel uses this identifiers for its own protection. So there is no way an end user can make a file owned by root cousin. Only the kernel is able to use these dedicated uid and gid in order to protect some files even from root. So it is quite obvious that we should not use these root cousin privileges everywhere. Rather, its use should be kept to minimal. For files, we don't want even root to change/modify, we should make them own by root cousin. While displaying the file properties (ls command), should we display the dedicated uid and gid for root cousin or should we show root uid and gid instead? If we show root uid and gid, the end user will still have the illusion that these files are owned by root but they can not be changed/modified. Great!! This goes well with the UNIX methodology. Lots of questions are popping in my head. Will update again as I have more answers.
Showing posts with label Storage. Show all posts
Showing posts with label Storage. Show all posts
Thursday, July 17, 2008
Saturday, May 31, 2008
Deduplication anyone?
De duplication is one of the hot topics in storage world. With tons of vendors offering de dup products and biggies like Netapp offering integrated de dup solutions with their NAS products, the competition is fierce. How does du duplication help when actually the consumer is trying to keep redundant data in order to facilitate disaster recovery?
Effectively de dup is doing opposite of what RAID, replication, snap shots do. This is not exactly how it sounds. The de dup essentially tries to take a whole different approach in order to save disk space on a file system. The granularity of de dup could be a file or a file system block. If done at file level, it would de dup less data since very few times are two files are entirely identical. But blocks could be identical very often and it would definitely save more space. We would discuss block based de dup here.
- De dup calculates a kind of identity signature for each block on the file system and stores it with a data base. Now the blocks containing the same data will generate same signature and can be detected to be a duplicate of an existing block. A cryptographic hash algorithm like MD5 or SHA1 can be used to generate this signature.
- How to store these signatures, that is the layout of database storing these signatures is highly platform dependent. The main requirement from this database is to give a list of blocks generating same signature (having same data), something like a hash bucket storing all elements generating same hash value.
- Another important requirement is that de dup should work while the file system is online. Putting the file system off line is not an option. Hence if write comes on a block for which signature has been stored with the data base needs to regenerate the signature in order to keep up with the latest data. This definitely needs a trap in the IO path but it should be such that it should have a minimal impact on the IO performance.
- Please keep in mind that de dup will only de dup the data blocks and not the meta data. Meta data is duplicated on purpose and should not be touched.
- So for the very first time de dup is started, it will generate signatures for all the data blocks in the file system. Once this pass is finished, we have all the information in the data base. Traversing this data base will give us a list of blocks bearing same data.
- For these blocks, only one copy can be kept while other blocks will be freed and the meta data of freed blocks will point to the one copy.
- One side effect of de duplication is that the next time a write comes on some block, we need to know if this block is sharing data with some other data. If it is, we need to do a copy-on-write here. Basically we allocate new block, write data on the new block and update meta data of the block. This way, the writes might have to bear a read penalty.
- For file systems which have copy-on-write in IO path like WAFL and ZFS, this would be not a problem. Other file systems would have to bear this penalty.
- Getting the data base in core is another problem. Either it needs to be implemented as cache otherwise it will occupy a lot of space. This would be very implementation specific.
Any more thoughts?
Update : Curtis Preston explains this in a very simple manner. Have a look at this - http://www.backupcentral.com/content/view/175/47/
Effectively de dup is doing opposite of what RAID, replication, snap shots do. This is not exactly how it sounds. The de dup essentially tries to take a whole different approach in order to save disk space on a file system. The granularity of de dup could be a file or a file system block. If done at file level, it would de dup less data since very few times are two files are entirely identical. But blocks could be identical very often and it would definitely save more space. We would discuss block based de dup here.
- De dup calculates a kind of identity signature for each block on the file system and stores it with a data base. Now the blocks containing the same data will generate same signature and can be detected to be a duplicate of an existing block. A cryptographic hash algorithm like MD5 or SHA1 can be used to generate this signature.
- How to store these signatures, that is the layout of database storing these signatures is highly platform dependent. The main requirement from this database is to give a list of blocks generating same signature (having same data), something like a hash bucket storing all elements generating same hash value.
- Another important requirement is that de dup should work while the file system is online. Putting the file system off line is not an option. Hence if write comes on a block for which signature has been stored with the data base needs to regenerate the signature in order to keep up with the latest data. This definitely needs a trap in the IO path but it should be such that it should have a minimal impact on the IO performance.
- Please keep in mind that de dup will only de dup the data blocks and not the meta data. Meta data is duplicated on purpose and should not be touched.
- So for the very first time de dup is started, it will generate signatures for all the data blocks in the file system. Once this pass is finished, we have all the information in the data base. Traversing this data base will give us a list of blocks bearing same data.
- For these blocks, only one copy can be kept while other blocks will be freed and the meta data of freed blocks will point to the one copy.
- One side effect of de duplication is that the next time a write comes on some block, we need to know if this block is sharing data with some other data. If it is, we need to do a copy-on-write here. Basically we allocate new block, write data on the new block and update meta data of the block. This way, the writes might have to bear a read penalty.
- For file systems which have copy-on-write in IO path like WAFL and ZFS, this would be not a problem. Other file systems would have to bear this penalty.
- Getting the data base in core is another problem. Either it needs to be implemented as cache otherwise it will occupy a lot of space. This would be very implementation specific.
Any more thoughts?
Update : Curtis Preston explains this in a very simple manner. Have a look at this - http://www.backupcentral.com/content/view/175/47/
Saturday, May 26, 2007
Storage Upcoming!!!
With ideas like thin provisioning taking shape, storage market is bumping with new ideas. And now comes pNFS - parallel NFS which enables multiple NFS servers to contribute to become a single server and share a clustered file system.
NAS though cheap has a fundamental limit in scaling and performance. Here the pNFS chips in. It can reuse your commodity servers to form a cluster out of them and can really scale linearly. It almost works like a multi path file system where a regular NFS server is replaced by a NFS4.1 Meta Data server(MDS). The MDS is moved out of band and now clients can directly talk to the array behind.
The best part is - this effort has been standardized through IETF and on its way to become a part of NFS4.1 draft. So storage vendors know which direction to drive their efforts to. As of now only Panasas provides a pNFS solution and chances are others will follow. It would be really interesting to see next few months on the market :)
Sun is likely to come up with the next implementation. For some good tutorials about pNFS, watch this space .... http://opensolaris.org/os/project/nfsv41/pnfsdemos/basics
An html version of the NFS 4.1 draft is available here
http://www.nfsv4-editor.org/draft-10/draft-ietf-nfsv4-minorversion1-10.html
NAS though cheap has a fundamental limit in scaling and performance. Here the pNFS chips in. It can reuse your commodity servers to form a cluster out of them and can really scale linearly. It almost works like a multi path file system where a regular NFS server is replaced by a NFS4.1 Meta Data server(MDS). The MDS is moved out of band and now clients can directly talk to the array behind.
The best part is - this effort has been standardized through IETF and on its way to become a part of NFS4.1 draft. So storage vendors know which direction to drive their efforts to. As of now only Panasas provides a pNFS solution and chances are others will follow. It would be really interesting to see next few months on the market :)
Sun is likely to come up with the next implementation. For some good tutorials about pNFS, watch this space .... http://opensolaris.org/os/project/nfsv41/pnfsdemos/basics
An html version of the NFS 4.1 draft is available here
http://www.nfsv4-editor.org/draft-10/draft-ietf-nfsv4-minorversion1-10.html
Tuesday, May 8, 2007
System layering
Recently Andrew Morton, a lead Linux kernel developer called ZFS as layering violation. And you can see a fantastic reply from Jeff Bonwick, the lead of ZFS project here -
http://blogs.sun.com/bonwick/entry/rampant_layering_violation
IMHO, even if ZFS violates the conventional layering stack, it will thrive as long as it is performing better than other file systems. ZFS is the future of file systems and has introduced many path breaking features which no other file systems can claim as of now. Few mentions would be detecting silent data corruption, numerous snapshots, scalable infrastructure, in short end to end data integrity which is what matters at the end of the day.
I might sound like anoother ZFS fan but the truth is out there. ext2/ext3 apart from doing what they are supposed to do are prone to silent corruptions and are doing basically what FFS was doing few years ago. Since ZFS guys were brave enough to throw away years old conventions, they have been able to aggregate the control to only one entity, the file system which can manage from raid groups to backup to clones.
I personally believe Linux is the best thing happened with the computers in last few years. But ZFS took over almost everyone in File Systems development. I think both OpenSolaris and Linux would thrive and Linux might adapt the new features offered by ZFS.
http://blogs.sun.com/bonwick/entry/rampant_layering_violation
IMHO, even if ZFS violates the conventional layering stack, it will thrive as long as it is performing better than other file systems. ZFS is the future of file systems and has introduced many path breaking features which no other file systems can claim as of now. Few mentions would be detecting silent data corruption, numerous snapshots, scalable infrastructure, in short end to end data integrity which is what matters at the end of the day.
I might sound like anoother ZFS fan but the truth is out there. ext2/ext3 apart from doing what they are supposed to do are prone to silent corruptions and are doing basically what FFS was doing few years ago. Since ZFS guys were brave enough to throw away years old conventions, they have been able to aggregate the control to only one entity, the file system which can manage from raid groups to backup to clones.
I personally believe Linux is the best thing happened with the computers in last few years. But ZFS took over almost everyone in File Systems development. I think both OpenSolaris and Linux would thrive and Linux might adapt the new features offered by ZFS.
Subscribe to:
Posts (Atom)