Storage: Difference between revisions
(Created page with "= New IFS Storage Information = Your IFS storage is now available. I've given the thom-users group rights on it. Your managed workstations have been updated to connect it w...") |
No edit summary |
||
| (7 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
= Backed-up storage = |
|||
| ⚫ | |||
There are a number of places you can store files off your workstation. Note that <code>/scratch</code> on your workstation and <code>cerebro:/sharedscratch</code> are *not* backed up. |
|||
Your IFS storage is now available. I've given the thom-users group rights on it. |
|||
<code>~/thom-fs-common</code>, <code>~/thom-fs-nethome</code> and <code>~/theory-fs-common</code> are mounted to your workstation. |
|||
| ⚫ | |||
<code>cerebro:/filestore</code> contains 37Tb backed-up storage. |
|||
| ⚫ | |||
Avoid copying large amounts of 'useless' data (e.g. scratch files which cannot be read by anything). It is also sensible to have a README file to index what is where. |
|||
| ⚫ | |||
| ⚫ | |||
sudo mount -t cifs -o 'domain=AD,username=cen1001,uid=1000,gid=1000,file_mode=0750,dir_mode=0750' //rfs.ch.private.cam.ac.uk/thom wherever |
|||
| ⚫ | |||
| ⚫ | |||
| ⚫ | |||
| ⚫ | |||
| ⚫ | |||
| ⚫ | |||
sudo mount -t cifs -o 'domain=AD,username=cen1001,uid=1000,gid=1000,file_mode=0750,dir_mode=0750' //rfs.ch.private.cam.ac.uk/thom |
|||
| ⚫ | |||
| ⚫ | |||
| ⚫ | |||
\\rfs.ch.private.cam.ac.uk\thom |
|||
If <code>~/ifs-thom</code> is not showing up, it could be that you've not logged in with your password (e.g. just remotely logging in with ssh key authentication). |
|||
In that case you can force a log in with |
|||
ssh -oPubkeyauthentication=no localhost |
|||
You can immediately log out again and the ifs connections will persist. |
|||
= Bandwidth Limit for Easter Term 2019 = |
= Bandwidth Limit for Easter Term 2019 = |
||
Chemistry Department had to delay the upgrade of their link to CUDN to 10Gbps because the new UIS switches turned out not to support the Department's current resilient connection arrangement. This however has been arranged for after Easter Term 2019. Consequently, the Department's uplink still runs at the same speed as every desktop's individual connection, and so it's very easy for even one person uploading lots of data to the IFS to saturate Chemistry's outward link. If there is a need for bulk upload data before the upgrade of the link, please use <code>rsync</code> with the <code>--bwlimit</code> option or some |
|||
One important thing I need to warn you about. We had to delay the |
|||
upgrade of our link to CUDN to 10Gbps because the new UIS switches |
|||
turned out not to support our current resilient connection arrangement, |
|||
and it will now happen after exam term. Consequently our uplink still |
|||
runs at the same speed as every desktop's individual connection, and so |
|||
it's very easy for even one person uploading lots of data to the IFS to |
|||
saturate Chemistry's outward link. I found this out the hard way when |
|||
transferring the COs' share. If you have to bulk upload data before |
|||
upgrade the link, please use rsync with the --bwlimit option or some |
|||
other program that can limit its bandwidth. 300Mbps should be safe. |
other program that can limit its bandwidth. 300Mbps should be safe. |
||
An example command for <code>rsync</code> is |
|||
| ⚫ | |||
rsync -a --bwlimit=300000 dir1 ~/ifs-thom/''crsid''/calc |
|||
which syncs in archive mode (<code>-a</code>) with a bandwidth limit of 300000 kilobit per second (equal to 300 Mbps), and which places <code>dir1</code> in <code>~/ifs-thom/''crsid''/calc</code> (provided that <code>''crsid''/calc</code> has been created). |
|||
| ⚫ | |||
Latest revision as of 18:02, 16 June 2020
Backed-up storage
There are a number of places you can store files off your workstation. Note that /scratch on your workstation and cerebro:/sharedscratch are *not* backed up.
~/thom-fs-common, ~/thom-fs-nethome and ~/theory-fs-common are mounted to your workstation.
cerebro:/filestore contains 37Tb backed-up storage.
Avoid copying large amounts of 'useless' data (e.g. scratch files which cannot be read by anything). It is also sensible to have a README file to index what is where.
IFS Storage
Managed workstations in the group have been updated to connect to the new IFS storage. It will appear as ~/ifs-thom. If this is not shown, please log out and log in again.
For unmanaged machines, one of the following should work:
- Linux CLI (wherever), substituting the appropriate values:
sudo mount -t cifs -o 'domain=AD,username=cen1001,uid=1000,gid=1000,file_mode=0750,dir_mode=0750' //rfs.ch.private.cam.ac.uk/thom
- Mac Finder and some Linux file managers will accept:
smb://rfs.ch.private.cam.ac.uk/thom
- Windows Explorer:
\\rfs.ch.private.cam.ac.uk\thom
If ~/ifs-thom is not showing up, it could be that you've not logged in with your password (e.g. just remotely logging in with ssh key authentication).
In that case you can force a log in with
ssh -oPubkeyauthentication=no localhost
You can immediately log out again and the ifs connections will persist.
Bandwidth Limit for Easter Term 2019
Chemistry Department had to delay the upgrade of their link to CUDN to 10Gbps because the new UIS switches turned out not to support the Department's current resilient connection arrangement. This however has been arranged for after Easter Term 2019. Consequently, the Department's uplink still runs at the same speed as every desktop's individual connection, and so it's very easy for even one person uploading lots of data to the IFS to saturate Chemistry's outward link. If there is a need for bulk upload data before the upgrade of the link, please use rsync with the --bwlimit option or some
other program that can limit its bandwidth. 300Mbps should be safe.
An example command for rsync is
rsync -a --bwlimit=300000 dir1 ~/ifs-thom/crsid/calc
which syncs in archive mode (-a) with a bandwidth limit of 300000 kilobit per second (equal to 300 Mbps), and which places dir1 in ~/ifs-thom/crsid/calc (provided that crsid/calc has been created).
Once the Department gets the outbound link up to 10Gbps this will not be an issue.